[ad_1]
Adam Mastroianni has an excellent post on strong bond vs weak bond problems in science. He writes:
Weak link problems are problems where the overall quality depends on how good the worst things is. Weak link problems are fixed by strengthening the weaker links or removing them altogether.
Food safety is a weak link problem, banking or computer security is a weak link problem, many production processes are weak link problems, also called O-ring problems.
(But) some issues are strong link issues – overall quality depends on how good the better things are, and bad things hardly matter… Venture capital is a strong-link problem: it’s okay to invest in a bunch of startups that fail as long as one of them hits a billion.
….Here’s the crazy thing: most people treat science like it’s a weak union problem.
Peer review of publications and grant proposals, for example, is a massive weak-link intervention. We invest ~15,000 collective years of effort each year trying to prevent bad research from being published. We force scientists to spend large amounts of time completing grant applications, most of which will be unsuccessful, because we want to make sure we’re not wasting our money.
These policies, like all forms of surveillance, are potentially great solutions to weak link problems because they can kill the worst investigations. But they are lousy solutions to strong link problems because they can kill the better research, too. Reviewers are less likely to give papers and grants the green light if they are novel, risky, or interdisciplinary. When you’re trying to solve a strong link problem, it’s like swallowing a big chunk of kryptonite.
In Maximum Progress, Max Tabarrok has some good diagrams illustrating the problem:
If you have a weak view of science, you would think that peer review works this way. The relationship between quality and final impact is linear, or maybe even tilted a bit. Moving resources from projects of low input quality to average projects is at least as important for ultimate impact as moving resources from average projects to high quality projects.
In a strong link science model, filtering out the lower half of the quality distribution is less important for the final impact (because the impact of the research is highly nonlinear).
Although peer review has the same perfect filter in the quality distribution, it does not translate into large changes in the impact distribution. Many resources continue to be given to projects with very low impact. Although the average input quality increases by the same amount as in the weak link model, the average final impact hardly changes. Since peer review has significant costs, the slightly higher average impact might not make up for the losses in total output compared to no peer review.
This is a simplified model, but many of the simplifying assumptions are favorable for peer review. For example, peer review is modeled here as a filter at the lower end of the quality distribution… But if peer review also removes some projects at the higher end, its increase in average scientific research impact will be would be silenced or even reversed.
[ad_2]
