Getty
Peer review is often the key obstacle between obtaining some data and publishing it in the scientific literature. As such, it is often essential to keep questionable results out of the scientific literature. But for a large number of scientists with solid but unexciting results, it can be an obstacle that raises frustrations to thermonuclear levels. Therefore, it is not surprising that many scientists privately wish for certain reviewers to end up engaging in activities that cannot be mentioned in a largely family-oriented publication like Ars.
What was a surprise was seeing a peer-reviewed post making this wish public. Very public. As in the document title “Dear Critic 2: Go F ‘Yourself” audience levels.
Naturally, we read the newspaper and contacted its author, David Peterson, from the state of Iowa, to find out the details of the study. The key detail is that the title is somewhat misleading: it is actually Reviewer 3 Who is the heartless bastard who keeps trying to torpedo the careers of other academics? Otherwise, well, keep reading.
We have to ask: why?
Peterson presented his case by looking at a particular reviewer in his article, in the helpful section titled “Why Reviewer 2?”
The main motivation for this article is that the community at large has decided that Reviewer 2 is a monster. A Google search for “Reviewer 2” produces the Facebook interdisciplinary group “Reviewer 2 must be stopped!” (which has more than 9,000 members), a blog post titled “How Not to Be Reviewer # 2” and countless images that combine almost every imaginable visual meme. In academia, it’s fair to say Reviewer 2 is the last man in the bag. He is Pennywise the Clown, combined with the chupacabra, wrapped in the Witch Blair.
Put another way, Peterson wrote: “Reviewer 2 despises other people’s work, is lazy, belligerent, and smug.”
But that doesn’t get to the broader problem: why look at this problem at all? Peterson said it’s more or less because he had the data anyway. He was editor of Political Behavior magazine for four years, and Peterson had been analyzing the results of his peer review as part of a process looking for systemic biases in results based on things like people’s race or gender. that they tried to post there. “So I had all the data, right? I had collected everything for this other project,” he told Ars. “And then I realized, honestly, after a beer or two, that I could try to prove this. It’s pretty straightforward, you know? It’s a very, very direct statistical test.”
Do that two statistical tests. In the first, Peterson verified whether there was any systematic difference in the classifications of articles according to the number of reviewer. That turned out absolutely nothing. But Peterson was not done. “There is this kind of second possibility: that when academics … when they get mad at the reviews, actually, it’s the negative outlier that we hate, right? And then maybe I could try to capture that idea that the Reviewer 2 is the number of the reviewer will probably matter as it is a lower average category than the average of the other reviewers. “
He did the statistics to check if a reviewer frequently rated the documents very differently than their other peers. “I developed an original measure of ‘being Reviewer 2’,” Peterson wrote, before going on to say that “Reviewer 2’s real problem is that it is an outlier and that can only be seen when the manuscript is strong enough to be positive reviews from other reviewers. This is when Reviewer 2 crushes their hopes. “
We could have told you that
Surprisingly, this turned out somewhat. When asked if this surprised him, Peterson’s response was “Oh, God, yes.” But the surprise did not end with the fact that there was no result at all; extended to the fact that outliers was not Reviewer 2.
It was the reviewer 3.
Those of you biologists will be nodding wisely when (confirmed through Dr. Beth Mole) that field has always blamed Reviewer 3. In fact, there is a full Downfall meme about Reviewer 3.
Caution: many fake NSFW subtitles.
We asked Peterson about this, and he speculated that biologists might be a bit sharper when it comes to choosing the nefarious reviewer. “I think biologists were right,” he told Ars. “I think biologists might be a little bit better at this than [political scientists] They are. Honestly, that’s fun for me. And I’m not sure why different disciplines would choose different numbers to make the devil. “
He suggests it might have something to do with how reviewers are chosen. Reviewers of Political Behavior end up in the dreaded 2 slot largely by self-selection. Knowing that many potential reviewers would say no for various reasons, Peterson said he would send requests to more people than he needed. Anyone who said yes would simply receive a reviewer number based on the order in which they responded. Other magazines might handle that differently.
What stood out for Peterson was the fact that, at least among political scientists, Reviewer 3 is the problem, yet the community has managed to pass the blame on to someone else. “Not only is Reviewer 3 the bad actor, but Reviewer 3 is cunning enough to blame Reviewer 2,” he told Ars. “What tickled me endlessly, frankly.”
How is this published?
In the document, Peterson omits normal academic language to evaluate this: “It appears that this is the last idiotic move.” A language like that, references to the chupacabra, and the title itself are quite unusual in academic literature. But Peterson published it without abusing the fact that he was an editor. Part of this is due to the fact that, in essence, this is a quantitative analysis of human behavior, the kind of study that many magazines handle.
Still, that did not facilitate publication. “This was not the first newspaper I sent it to,” he admitted. Apparently part of the problem was that some of its reviewers had managed to remain oblivious to the concept of a Hell Critic. “I kept getting reviewers who had never heard of a Reviewer 2 idea,” he said. “So the basic idea that there is an idiot out there was totally foreign to them, so they didn’t understand why anyone would think it was an interesting question. Which surprised me. But yeah.”
Finally, he had a conversation with the people who would serve as editors in the magazine where it was published. “I have known the editors of the Social Science Quarterly for a long time and had a conversation with them before I sent it out, to make sure they would recognize it for what it was,” Peterson said.
Still, it wasn’t necessarily easy for them to translate that into getting the role accepted. “I think the editors were careful in their selection of reviewers,” said Peterson.
The other drawback he had with editing is the title, which combines obscenity with blaming the wrong reviewer, the latter of which was almost changed to Reviewer 3 by an editor. “When I sent it to other magazines. The title was ‘Is Reviewer 2 really Reviewer 2?’ And that’s probably a better title, but I like this one better. “
Social Sciences Quarterly, 2020. DOI: 10.1111 / ssqu.12824 (About DOIs).
