Strong and weak link issues and the value of peer review
Adam Mastroianni has an excellent post on strong bond vs weak bond problems in science. He writes:
Weak link problems are problems where the overall quality depends on how good the worst things is. Weak link problems are fixed by strengthening the weaker links or removing them altogether.
Food safety is a weak link problem, banking or computer security is a weak link problem, many production processes are weak link problems, also called O-ring problems.
(But) some issues are strong link issues – overall quality depends on how good the better things are, and bad things hardly matter… Venture capital is a strong-link problem: it’s okay to invest in a bunch of startups that fail as long as one of them hits a billion.
….Here’s the crazy thing: most people treat science like it’s a weak union problem.
Peer review of publications and grant proposals, for example, is a massive weak-link intervention. We invest ~15,000 collective years of effort each year trying to prevent bad research from being published. We force scientists to spend large amounts of time completing grant applications, most of which will be unsuccessful, because we want to make sure we’re not wasting our money.
These policies, like all forms of surveillance, are potentially great solutions to weak link problems because they can kill the worst investigations. But they are lousy solutions to strong link problems because they can kill the better research, too. Reviewers are less likely to give papers and grants the green light if they are novel, risky, or interdisciplinary. When you’re trying to solve a strong link problem, it’s like swallowing a big chunk of kryptonite.
In Maximum Progress, Max Tabarrok has some good diagrams illustrating the problem:
If you have a weak view of science, you would think that peer review works this way. The relationship between quality and final impact is linear, or maybe even tilted a bit. Moving resources from projects of low input quality to average projects is at least as important for ultimate impact as moving resources from average projects to high quality projects.