Session Report: “Peer Review: History and Issues”

By Daniel Ucko

Report on session A14 at the APS March meeting in Baltimore, MD, on March 14, 2016.

For the second year running, the Forum on the History of Physics organized a session on Peer Review at the American Physical Society March Meeting. The session followed a similar event at the March Meeting in San Antonio, TX, in 2015. The title of this session was “Peer Review: History and Issues”, and featured four speakers, all talking about journal publications from different perspectives.

The first speaker was Jamie Hutchins, Publishing Director of the Institute of Physics Publishing (IOPP), who presented results from an extensive survey of 6000 physicists performed in October 2015. The survey was aimed at elucidating the respondents’ means of sharing scientific results and information. While the means of sharing information varies by discipline, there are some common themes. For instance, one finding was that journals still represent the “gold standard” for disseminating scientific results, even though the arXiv is also strong. Perceived journal quality is a strong determinant for where researchers will publish data, as well as these journals’ ability for dissemination of results. Surprisingly, the popularity of reference managers such as Mendeley, ResearchGate, or Sci Val, was lower than expected, which is in conflict with the reported number of articles actually deposited on these sites.

The survey also broke down data by seniority of researchers, who have different priorities for journal choices. Researchers later in their careers were more likely to make decisions based on previous experiences with a journal, including as a reviewer, value personal relationships with editors and editorial boards, and the absence of page or color charges. By contrast, researchers earlier in their careers were more often guided by their supervisors or collaborators as to journal choice, but also were favorably influenced by the existence of open access and supplemental material options.

The next talk was by Alex Csiszar of Harvard University, who focused on the origins of referee systems in 19th-Century Britain. The actual history of peer review is complex, for not only is the invention of referee systems more recent than is often believed, but their imagined functions have changed over time. Csiszar locates these changes in periods of broader political change, when the role of science in society was being renegotiated.

It is in the early 19th Century that scientific societies began to consult systematically experts to make decisions about what to publish. The Royal Society of London’s first attempt to set up a system of referee reports began with something that resembled open peer review, in which recognized authorities would publish their reports themselves as a form of publicity. But the system shifted toward anonymous specialists, in part because of the culture of criticism then dominant in England. Just as it is today, the legitimacy of anonymous judgment was the subject of lively debate throughout the period.

Next to talk was Melinda Baldwin, also of Harvard University, whose talk concerned refereeing for the National Institutes of Health (NIH) and the National Science Foundation (NSF). Peer review for grant organizations such as these exists in an atmosphere of tension between research autonomy and a governmental desire for oversight. Dr. Baldwin noted that early in NIH’s history, grant proposals were decided on by the directors, and there was no requirement to have them reviewed externally. By contrast, the NSF did consult external referees, but the reports obtained were not sent to the applicants, and directors still had significant say in which proposals received funding.

Dr. Baldwin presented a number of instances in which the procedures of grant review in both NIH and NSF had come under political scrutiny, and the timing of these instances echoed Dr. Csiszar’s contention that changes to peer review coincide with periods of historical upheaval. In the 1960’s, following controversy over the exact spending of NIH grants, the NIH introduced stricter internal accounting rules.

In 1975, following hearings on its peer review process, it was decided that NSF should provide verbatim referee reports to grant applicants and give more weight to referee opinions in the decision-making process. As this was unfolding, NIH also quietly updated their own process, possibly anticipating similar congressional challenges to their procedures. It is clear that peer review in this area changed drastically during the 20th Century, in an attempt to find a compromise between scientific accountability and government oversight.

Last to speak was Daniel Ucko, a doctoral student in philosophy at Stony Brook University and Associate Editor of Physical Review Letters of the American Physical Society, who presented an attempt to localize trust in the procedures of peer review. Peer review, Ucko claims, mimics the usual definition of the scientific method, with the paper under review as a hypothesis evaluated by the experiment of peer review. It represents a reach towards objectivity by attempting to eliminate subjectivity, through various strategies, include single- or double-blind review.

Trust as a concept is most often studied when it becomes conspicuous by its absence, when a breakdown of trust occurs. Since science as well as peer review is built on empiricism and skepticism, one could say that science is built on distrust. However, as in all dealings between humans, some level of trust is required. It is Dr. Ucko’s contention that what trust does exist in peer review as well as in science is built on a foundation of distrust. If all actors share the same level of epistemological skepticism, then a system of scrutiny emerges that allows for trusting to happen in spite of the skepticism that empirical science requires.

The session ended with a lively Q&A, with questions being asked of all the participants. It is clear that peer review continues to be a fascinating and controversial topic that draws a wide audience at this meeting.

---

The articles in this issue represent the views of their authors and are not necessarily those of the Forum or APS.