Peer review worsens precision but improves recall

I think Adam Mastroianni has done a fine job critiquing peer review as a barrier to entry and as a service to paper-readers, but that's not how it really functions in practice, for better or worse. Peer review is a system by which random, anonymous paper-writers are able to force other researchers to read their work. In other words, peer review is a service to paper-writers. It is especially a service to paper-writers lacking prestigious affiliations and notable previous work — authors whose papers would go unread if they just posted on arXiv. So in this sense, peer review is a path to entry, not a barrier to entry.

This is why being a reviewer is so painful: it imposes a uniform prior for paper quality, even though we know that some researchers do much better work than others, and we would prefer to focus on reading their work. This is why even peer-review is not a guarantee of quality: the process ignores said priors, and also reduces the incentive for developing a reputation as someone who only submits quality work.

At the same time, blinded paper reviewing is probably the only way to identify good work coming from new and unknown researchers. In other words, there is a precision-recall tradeoff to peer review. If reviewers were not forced to read papers they otherwise would not, precision would increase but recall would decrease. Yitang Zhang’s recent groundbreaking work on prime number gaps is a striking example of this. Because he hadn’t published anything since 2001, had no prior background in number theory, and held a lowly lecturer position, his paper received attention only because the reviewers at Annals of Mathematics were forced to read it.

Furthermore, as a prominent researcher with Twitter clout, it is no suprise that Mastroianni had no difficulty getting 110+ comments in reply to his Substack post about bias in human imagination. But if a grad student at Podunk State University wrote the exact same thing, it would quite easily go unread. This means that graduate students will, rather than spending time to fulfill the demands of peer reviewers, instead spend time trying to obtain Twitter clout. The easiest way to obtain Twitter clout is to write hot takes and engage in herding behavior on politicized topics. Thus, removing peer review will hurt recall further, by making it harder for boring, non-hot-taking researchers to disseminate their findings.