Peer Review in Scientific Publishing: _{What it is and how it works}

History and development of peer review

The origin of peer review is closely tied to the development of scientific institutions and scientific journals. When the Royal Society of London (see our Scientific Institutions and Societies module) began publishing its journal Philosophical Transactions in 1665 (Figure 2), the review process was solely the responsibility of the editor (at the time, Henry Oldenburg). The editor sometimes enlisted other scientists to provide opinions as necessary, but it was at his sole discretion whether works were reviewed by others or not. In 1752, the Society itself took over the editorial responsibilities for the journal and instituted a review policy wherein each manuscript was sent to a small group of experts in the field before being published (Spier, 2002). This practice was the beginning of the peer review process as we know it today.

The process spread sporadically through the sciences in the nineteenth century, but became a standard for scientific review in the mid-twentieth century. Some have linked the spread of peer review in the twentieth century to the popularization of Xerox® photocopying machines, which facilitated the replication and distribution of manuscripts to reviewers (Spier, 2002). Whatever the motivation, both Science and the Journal of the American Medical Association implemented consistent peer review practices in the 1940s, and other scientific journals followed suit.

Today, more than 5,000 different scientific journals exist, and each of them receives many manuscripts from scientists who feel their work should be published. The journal Science alone receives as many as 200 manuscripts per week. To evaluate the appropriateness of each manuscript for publication, journal editors identify two or three peer reviewers to read and comment on each submitted manuscript.

Peer reviewers are knowledgeable scientists who are not directly involved with the research being evaluated but who are familiar with the field of study or the research methods used. The reviewers may not know the scientists who wrote the paper or grant proposal, they may be colleagues, or they may even be scientific competitors with the authors. So, in an attempt to remove bias from the review process, most manuscripts are independently considered by multiple reviewers, and it is left to the journal editor to make the final decision regarding publication based on the reviewers' comments.

Reviewers consider the relevance of the work to the journal or funding program being targeted, the validity of the methods and interpretations, the originality of the finding, and the clarity of the writing. Reviewers then provide feedback, generally in written form, on the work they have read. And these comments are commonly sent back to the authors so that the authors can use the reviewers' comments to refine and improve the text of their manuscript or their research. Reviewers can often choose whether they wish to remain anonymous to the authors or not. Journal editors and grant program directors rely on the reviewers' feedback to guide their decisions – they may choose to accept a work as is, they may ask the author(s) for revisions, or they may reject the work based on the peer review comments.

Peer review in practice

On January 22, 2007, Conrad Mauclair and colleagues submitted a manuscript entitled "Quantifying the effect of humic matter on the suppression of mercury emissions from artificial soil surfaces" for consideration to the journal Applied Geochemistry. The manuscript was sent out by the editor to two peer reviewers, who were given one to two months to complete the review. The reviewers sent comments to the editor, and after considering the reviewers' comments, the editor chose to accept the manuscript with revisions, and responded as such to the authors approximately five months after their initial submission. Excerpts from the letter the editor wrote to the authors detailing this decision are printed below.

27 May 2007
Dear Authors:

I have received two reviews of your manuscript entitled "Quantifying the effect of humic matter on the suppression of mercury emissions from artificial soil surfaces" submitted for publication in Applied Geochemistry. In addition I have read your paper and have some additional comments that are below. All reviewers including myself agree the paper after revisions is acceptable for publication.

I have attached both reviewers' comments to this email. Both reviewers raise some important issues that need to be clearly addressed in your revised paper. I agree with their concerns and below have added a few others that need to be addressed.

Sincerely,
Editor for Special Issue of Applied Geochemistry

Additional detailed comments from the Editor:

The mass balance needs to be considered [as detailed by reviewer 2]. My guess is your flow rate is producing an artificially high flux. The way to deal with this would be to use the actual concentration difference between the inlet and outlet instead of the flux to calculate the amount lost. Plot the difference between the outlet and inlet concentrations rather than flux.

The reviewers' comments were attached to the editor's letter with the names of the reviewers removed. In the case of journal manuscripts accepted with revision, the authors have the opportunity to read and respond to the reviewers' comments and make changes to the article in question. In the case of grant submissions, scientists read the reviewers comments and use these to strengthen their submission the next time they apply for a grant. For the article by Mauclair and colleagues, the reviewers had a number of recommendations for improving the article, as the excerpts provided below detail.

Excerpts of comments from Reviewer 1:

I have received two reviews of your manuscript entitled "Quantifying the effect of humic matter on the suppression of mercury emissions from artificial soil surfaces" submitted for publication in Applied Geochemistry. In addition I have read your paper and have some additional comments that are below. All reviewers including myself agree the paper after revisions is acceptable for publication.

The "suppression of mercury emission" [in the title] is an interpretation of the experimental observations, rather than an unequivocal conclusion. It might be better to use a [more] conservative title like "Quantifying the Effect of Humic Matter on Mercury Emissions from Artificial Soil Surfaces". I'd think the reader might come up with some different interpretations other than "suppression".

Would [additional experiments with] controls of humic matter plus Hg(II) salt only (without any sand) offer any more [information]?

Excerpts of comments from Reviewer 2:

In response to direct comments requested by the editor:

• Originality: This paper systematically tests the combined impact of humic matter content and light in synthetic soils. Very few studies have reported similar work.

• Importance: This work's main conclusion is that organic matter content alters Hg emissions from soils. This conclusion is of significant interest to mercury biogeochemists and may promote related field-based research, and help in the interpretation of current sets.

• Manner of presentation: The paper is short, clear and to the point. More discussion of possible mechanisms and more details on related field studies (where fluxes and organic matter have been correlated) could be added.

• Quality of figures and tables: I do not think that the authors have reached an optimal design for the graphical presentation of their data. Figure 1, 2 & 3 could be easily combined, which would help the reader to compare the results taken at different intervals for the same experiments. In fact, these graphs could even be transformed in time series line graphs (instead of histograms). I am not sure of the most attractive final design, but the present design can be improved.

• Serious flaws or can [the paper] be improved by condensation or deletion of information: I have not found any serious flaws. I can say that I am not totally at ease with a study that reports only results from synthetic soils. It would have been nice to complement this data with some «real» soils. But I think that such a systematic, laboratory study is useful and pertinent.

• Does the title and abstract correspond to the content of the manuscript: Yes

• Would you be willing to re-review this paper after submission with revisions: Not necessary

Specific comments regarding the manuscript:

1. Can the authors comment on the realism of their approach? What are the limits of using synthetic soils and mixtures of inorganic Hg + humic acids? The fact that they tried different kinds of humic matter is comforting, but I would have like to see more info on potential limitations.

2. Page 9. Please clarify the design for the long term monitoring section. For instance, were the lights on for 14 days in the "light treatment"? Was this continuous flux sufficient to decrease the pool of Hg in the sample? The following back [of] the envelope calculations left me worried by the results presented here:

3. If we take an average flux of 2000 ng/m2/h for the light + sand treatment (see figures 1, 2 and 3), then we get over 14 days [and] 44 µg lost by evasion, whereas only 25 µg were added!!

4. I suspect that the lamps were only ON during the readings, once every week, but this should be more obvious. More info on the impact of the flux on the mass balance of the samples should be added. If the lights were turned OFF between weekly readings, how long were they ON for the readings?

Responses to specific comments raised by the Editor:

• While turnover rate is a significant issue, our work represents the relative comparison of samples that were all measured at a constant turnover rate, thus the effect of chamber turnover rate on our conclusions is negated. A discussion of this has been added to the manuscript. In an effort to guide future research, we have added mention of more recent personal communication regarding chamber turnover rate, to our knowledge new data regarding turnover rate is not published. A suggestion was made to report the difference between chamber inlet and outlet Hg concentrations rather than fluxes. As described above, the mass balance of Hg in the samples is not problematic. Further, because all Co-Ci differences are multiplied by a constant turnover rate in the flux equation, this would simply have the effect of changing the magnitude of the numbers (and graph axes) reported, not the relative difference between numbers - which is the basis of all conclusions of the work. Also, because the majority of researchers report results as fluxes, we feel that reporting our results as concentration differences would make this work inaccessible to mercury researchers. Our methodology and flux measurements are all based on peer-reviewed, published literature (Lindberg et al., 2002) and follow standard protocols. We believe discussing the limitations of the method is therefore sufficient in this context.

Once comments are received regarding a manuscript, it is up to the authors to address those comments, or in cases where they disagree with a reviewer's comment, provide an explanation as to why they have not addressed the comment. In the above case, the author's addressed the majority of the reviewers' comments and sent a letter back to the editor on June 10, 2007, detailing the changes made to the article and discussing why some changes were not made:

July 10,2007
Dear Editors:

Enclosed is our revised manuscript. We have addressed all of the comments returned to us in the reviews of our paper. In addition, at the suggestion of reviewer 1, we have conducted additional experiments with 100% humic acid and have added the results of this experiment to our paper to assure that we have adequately addressed the experimental design comments. A detailed list of all individual changes is included below.

All of the listed authors have read the revisions and agree with their conclusions.

Sincerely,
Authors

Detailed list of manuscript revisions.

Responses to comments raised by Reviewer 1:

• As directed, we have revised the title of the manuscript to "Quantifying the Effect of Humic Matter on the Emission of Mercury from Artificial Soil Surfaces."

• The reviewer raises an interesting question regarding the use of Hg-humic controls (without sand), these controls were not examined at the time of our study. However, to satisfy this question we have since conducted additional experiments with a 100% humic sample using 1g humic and HgCl2 sample (no sand). The results from this sample were consistent with those presented for our 5% humic sample, confirming that the effect we saw was due to humics, and not the interaction of humics with the sand. We have added this data to the paper and to Figure 1.

Responses to specific comments raised by Reviewer 2:

1. We have condensed the presentation of data in the Figures as suggested so that only one pair of graphs is now used (new Figure 1) instead of the three pairs that were used in the previous version of the manuscript (former Figures 1, 2, and 3). We have also edited Figures 2 and 3 (formerly Figures 4 and 5) as recommended.

2. Regarding the manner in which samples were stored between measurements, we had detailed this in the version of the manuscript submitted for review, our Methods section states "All samples were stored in the dark at constant temperature (~23°C) between measurements and monitored in both dark and light for mercury flux at regular intervals." We have tried to emphasize this statement in the results section of the rewritten manuscript and we have added a statement that all flux measurements were taken over a 1.5-2 hr sampling period.

3. The mass balance calculated by the reviewer overestimates Hg loss from the samples as he/she assumes that the samples were exposed to light continuously (see point #2 above). We conservatively estimate that the maximum Hg loss from the sample exhibiting the highest emission rate (sand-Hg only) was 30% of the mercury added. Humic-containing samples showed much lower Hg losses.

4. As the reviewer states, the samples were not under light continuously and this has been clarified as per the two points above.

Responses to specific comments raised by Reviewer 2:

• While turnover rate is a significant issue, our work represents the relative comparison of samples that were all measured at a constant turnover rate, thus the effect of chamber turnover rate on our conclusions is negated. A discussion of this has been added to the manuscript. In an effort to guide future research, we have added mention of more recent personal communication regarding chamber turnover rate, to our knowledge new data regarding turnover rate is not published. A suggestion was made to report the difference between chamber inlet and outlet Hg concentrations rather than fluxes. As described above, the mass balance of Hg in the samples is not problematic. Further, because all Co-Ci differences are multiplied by a constant turnover rate in the flux equation, this would simply have the effect of changing the magnitude of the numbers (and graph axes) reported, not the relative difference between numbers - which is the basis of all conclusions of the work. Also, because the majority of researchers report results as fluxes, we feel that reporting our results as concentration differences would make this work inaccessible to mercury researchers. Our methodology and flux measurements are all based on peer-reviewed, published literature (Lindberg et al., 2002) and follow standard protocols. We believe discussing the limitations of the method is therefore sufficient in this context.

The comments from Reviewer 2 highlight the fact that peer review helps the scientific publishing system to assure that manuscripts meet certain minimal standards. Reviewer 2 commented on the originality of the submission, the perceived importance of the work in the field of science, the manner of presentation of the writing in the text, the quality of the figures and tables and data analysis in general, whether he/she found any serious flaws in the work, and specifically the appropriateness of the manuscript title and abstract since these are the parts of the paper that will be cataloged by literature databases (see our Utilizing the Scientific Literature module) and thus widely read.

In addition, reviewers may recommend that authors clarify the text or add certain references that they had not previously considered; they might suggest changes because they feel that the authors' interpretations are not supported by their data; they may recommend additional research to clarify questionable points in the study; or they may recommend that a manuscript be rejected completely because of questions about the research methods, data collection, or interpretation.

Similarly, grant proposal reviewers may make specific recommendations for improving a study and recommend that the authors resubmit their proposal in another grant cycle after it has been improved. Grant reviewers might also recommend that more background research be conducted before the authors submit their proposal again, that another scientist with a different expertise be included on the research team, or that the scope of the research be broadened (or narrowed).

Mauclair and colleagues made the majority of the changes requested by the reviewers to their manuscript. They revised the title, added additional explanation to the text, and even conducted additional experiments to satisfy a question raised by Reviewer 1. In the reviews, Reviewer 2 calculated a mass balance of mercury in the experiments and concluded that the samples lost more mercury than was added to the system; the editor then followed this comment with a suggestion as to why this might have occurred and suggested reporting the data in a different manner. The authors address this comment by correcting an erroneous assumption of the reviewer, and explain why they have chosen not to report the data in a different manner as suggested by the editor. Thus, while they did not follow the suggestion made in the review, they provide a detailed explanation as to why.

Once a revised manuscript is resubmitted, the editor reads the authors' response, and if satisfied that the authors adequately addressed all of the issues raised, moves the manuscript forward in the journal's publishing cycle. In some cases, where major revisions are required, the editor may redistribute the manuscript to the peer reviewers a second time before accepting it as complete. The Mauclair manuscript was accepted for publication in August 2007, was first published on the journal's website in January 2008, and was finally published in the March 2008 issue of the printed version of the journal (Mauclair et al., 2008).

Implications of peer review

It is worth noting the lengthy time frame involved in publishing scientific articles. From the initial submission to the final printing, the described article took 14 months, which does not even include the time spent doing the initial work that led to the publication. The sluggishness of the peer review process is often criticized, but it reflects the understanding that published work enters the scientific literature permanently, as work that can be built upon by other scientists, and thus should be carefully considered.

Additionally, journals and funding agencies vary in their selectivity and research focus. Consequently, scientists choose where to submit their manuscript based on the perceived impact of the research, the likelihood of acceptance, and the size of the audience they wish to reach. In turn, reviewers consider the appropriateness of the research to the journal's audience. For example, while the journal Applied Geochemistry focuses on research articles that discuss chemical transformations and process that take place in the environment, the journal Cell publishes articles focused on biological process related to cell function. The article by Mauclair and colleagues was published in Applied Geochemistry; however, it would likely have been rejected by Cell.

As part of the scientific process, reviewers are expected to keep the information in a manuscript confidential until it is published, but it is rare that the work comes as a complete surprise to the entire scientific community. This is because peer review is integrated into almost every step of science, including requests for public funding for research. Funding decisions are made by a committee of peer review scientists who debate each proposal's likelihood of success, the validity of its approach, and the importance of the question being asked. The research methods and ideas published by Mauclair and colleagues in 2008 were reviewed as part of a grant submitted to the Research Foundation of the City University of New York, which was funded in 2005 (Carpi & Frei, 2005). Once funded, the research begins, and preliminary data may be presented at scientific meetings. This allows the findings to be described and debated with colleagues prior to publication. Data that were eventually used in the manuscript were presented to the scientific community in August 2006 at a large international conference on the pollutant mercury (Mauclair et al., 2006), and thus the final publication was anticipated by some scientists in the field.

Peer-reviewed publications and funded research proposals carry significance for the individual scientist beyond simply doing science. In many cases, hiring, promotion, and award decisions are made on the basis of the number and quality of peer-reviewed publications or grants authored by an individual. Scientists also benefit in many ways from serving as peer reviewers – being asked to review a manuscript or proposal is an acknowledgement of one's expertise in an area. All scientists both receive reviews from their peers and review the work of others, and this process comes with a cost. A recent report by the Research Information Network estimated the cost of volunteered time provided by scientists for peer review at $3.7 billion (Research Information Network, 2008). So why do scientists volunteer so much time to this process? Because it is one of the obligations of their profession and one factor that helps build the community of science (see our Scientists and the Scientific Community module).

One of several mechanisms of validating science

Peer review is just one of several mechanisms embedded within the process of science that help validate the work of scientists. While it helps to validate journal and grant submissions, it is not a fool-proof filter that assures quality in scientific publishing, especially when the authors of a study are engaged in fraud or deception (see our Scientific Ethics module). For example, between 2000 and 2003, Jan Hendrick Schon and colleagues published over 25 papers on superconductivity, all of which passed through the peer review process. After several of the papers were published, Professor Lydia Sohn and Professor Paul McEuen noticed that different experiments carried out under very different conditions and published in different papers displayed the same background error (Figure 4). When confronted with the problem, Schon first claimed that a graph had been mistakenly reproduced in several papers. Shortly thereafter, Bell Labs, the research institute where Schon worked (see our Scientific Institutions and Societies module), conducted an investigation; they found numerous instances of misconduct and fraud and consequently fired Schon. At least sixteen of Schon's papers have since been declared to be false, and the journal Science has withdrawn eight of his papers (Bao et al., 2002).

While cases of scientific misconduct can be embarrassing because of the publicity they receive, they highlight the self-correcting nature of science. A key aspect of science is that research results must be reproducible and well-documented. Instances of scientific misconduct that have gotten through the peer review system are often quickly exposed when other scientists scrutinize the data and attempt to reproduce the results. To keep this system working, scientific articles include detailed descriptions of research protocols that enable others to reproduce experiments, and they include tabular or graphical presentations of data so that they can be scrutinized by the community at large (see our Understanding Scientific Journals and Articles module). One of the first pieces of evidence that raised suspicion over Schon's work was the fact that other scientists had trouble reproducing his experiments with similar results. As occurred with most of Schon's publications, scientific research articles can be retracted if they are found to be in error (whether or not that error is a result of misconduct), thus removing them from the literature of science.

The truth is that retractions are rare, and retractions due to scientific misconduct are even rarer. In an analysis of the scientific literature cataloged in the MEDLINE database maintained by the National Library of Medicine, Sara Nath and colleagues identified 395 articles that were retracted in the two decades from 1982 to 2002 out of over 8.5 million citations listed in the MEDLINE database for that same period (Nath et al., 2006). Of those articles retracted, only 27% were found to be cases of scientific misconduct, 62% were identified as unintentional errors, and 11% of retractions could not be categorized. Nath and colleagues also found that an additional 2,772 errata were published during this same period, which are simple corrections of small mistakes in published manuscripts.

Consequences of peer review

One of the consequences of the peer review system is that it can influence the dissemination and progress of scientific research. It is the peer reviewers that make recommendations as to what research is published in what journals. And it is the peer reviewers that influence the types of research studies that receive funding. This is generally a positive effect as it opens the process to the scientific community at large. But bias among reviewers can negatively impact this process. For example, some researchers have suggested that peer reviewers can be biased in favor of research that reports positive effects (i.e. that drug x has an effect) over research reports that report a no effect (i.e. drug y has no significant effect) (Callaham et al., 1998). Thus, published studies showing positive effects far outnumber ones showing none. Another complication that affects the peer review process is that in a closed review system, where reviewers are kept anonymous from the authors, it is possible for reviewers to pass unnecessarily harsh judgment – or unworthy praise – on a manuscript or application for funding, simply because they have personal differences or friendships with authors.

The widespread use of electronic publishing has prompted a recent re-evaluation of the peer review process as a whole. While scientists still largely agree on the value of peer review, they are sometimes discouraged by the length of time involved from submission of a manuscript, through review, revision, and resubmission, which may take a year or more – as in the case of the article described here. As a result, some authors have suggested that the time-consuming and closed pre-publication peer review process be abandoned entirely in favor of open access, online publishing that allows for constant reviews and updates. A number of scientific publishing media are adapting to the changing nature of publishing. For example, the Public Library of Science (PLoS) project publishes a number of journals in the PLoS family that make their formal peer reviews available to the public and then they further provide a mechanism for additional public comment and review of published articles on their website. Even the journal Nature, which has been published for over 130 years, has recently experimented with open access peer review for submitted manuscripts.

Publications without peer review

There are publications in the sciences that are not peer-reviewed. For example, many journals, including Science and Nature, publish news and commentary sections in which they provide weekly updates on major scientific events or issues. A number of journals also publish "letters." Letters to journals include commentary on previously printed articles, but they may also report new, preliminary and intriguing scientific results that have not yet been tested and replicated enough to pass full peer review.

Scientists also write material specifically for non-peer reviewed publication. For example, the evolutionary biologist Steven J. Gould was a prodigious writer and became well known for his books and magazine articles on topics ranging from evolution to baseball. The theoretical physicist Stephen Hawking is well known for his books aimed at explaining cosmology, which include such popular titles like A Brief History of Time and The Universe in a Nutshell. And the astronomer Carl Sagan not only wrote numerous popular magazine articles, but also authored the best-selling book Contact, which was turned into a blockbuster film. The articles and books that scientists write for sources other than peer-reviewed journals have an important, but very different, purpose than the peer-reviewed literature. These pieces are often directed at explaining science in more common language to non-scientists and thus serve a crucial role in describing the impact of science to the general public. As such, they are generally, but not always, based on the peer-reviewed literature that forms the basis of our scientific knowledge.