how long does paper essay review take

Stack Exchange Network

Stack Exchange network consists of 183 Q&A communities including Stack Overflow , the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

How long typically are paper reviews? Is there such a thing as “too long”?

When you review papers submitted for publication, is there an “optimal” length for reviews? In my experience as an author and referee, I have seen a large range of review lengths (for reference, a paper in my field is typically between 3 and 8 printed pages):

• Zero length: for some of the papers I have authored, the reviewer just clicked the “accept as is” or “reject” checkmark on the review sheet, without adding any comment (at least, not any comment visible to me). It's not so common, but it has happened.
• Short length: a lot of time, I received reviews who consisted of a single paragraph.
• Medium: one full page, maybe two pages.

Although I have never received any such long reviews, I have myself written on a few occasions reviews that exceed two pages, including once or twice a four or five-page review. These were cases where the paper was good, but could be (in my view) much improved and some of the aspects/consequences had escaped the authors' consideration.

I am wondering how useful reviews of various length are to editors . I often consider that “unmotivated” reviews are useless, as they do not give any real insight about the paper to the editor. For example, if the editor gets two conflicting non motivated reviews, how is she to decide?

On the other hand, although I write some from time to time, I have never received long reviews, so… is this something frowned upon?

• peer-review

• 8 I'm not an editor but as an author, I certainly prefer a long review full of detailed suggestions to a one-line "your paper does not meet quality standards, reject". Sometimes a reviewer ends up contributing more to a paper, than one of the "minor" co-authors. –  gerrit Commented Oct 30, 2013 at 10:38
• I know one of the journal reviews went on for a year and finally the author had to push the reviewers a lot to get the final results. This is the longest I have seen. –  user2915398 Commented Oct 30, 2013 at 12:24
• 2 @user2915398 I was thinking in term of length of the written review, not time-wise –  F'x Commented Oct 30, 2013 at 12:50
• 1 This is awesome! I have been wondering about that as well. And may I suggest the answerer to also mention if you are speaking as an editor or not ? I value all inputs and tips but would also love to know some norm from the people in the industry. –  Penguin_Knight Commented Oct 30, 2013 at 13:08

4 Answers 4

Reviews can be of quite varying length but obviously the extremes indicate some problems.

A review consisting of "Accept as is" would be highly suspicious in my mind (as an editor). It usually means the reviewer has not done any work, essentially no manuscript is that close to perfect (although it may of course happen). A review of "Reject" without additional comment is equally pointless (I am then assuming the journal has some form of quality check before accepting for review). An absence of comments is just a big warning sign since there is no perspective on why the MS is either perfect or perfectly worthless.

Considering the length of a review, it is governed by two factors: the quality of the manuscript and the personality of the reviewer. To some extent longer reviews indicate more questions to be resolved. At the same time some reviewers may be more nit-picking than others so that also influences the length. Based on my experience as an editor, I would say, as a rule of thumb, that at least a page of (single spaced) comments would be a basis for a descent review for a normal manuscript (15-20 pages double spaced excluding references, tables, figures) in the field experiment/observation based science where I work. A review of more than three or four pages of (single spaced) comments would be unusual and probably involve comments down to spelling issues. "A decent review" involves providing clear and constructive comments that will allow the editor to value the manuscript and the author to improve the manuscript.

So I would not say that a long review would necessarily be frowned upon, it clearly depends on how constructive it is. If someone spends a lot of effort improving language and grammar (which does not necessarily constitute the expectations from a review) that could be very useful. Normally such comments may be made as revisions in a file rather than a written report. So length is not a major issue, constructiveness is.

• 2 As an editor, how do you handle zero length reviews? If the review just says "accept" or "reject", do you grudgingly follow the reviewer's advice, or do you look for another reviewer? –  Nate Eldredge Commented Oct 30, 2013 at 13:36
• 7 Most likely add another reviewer. The least one can do as a reviewer is to justify why one makes the recommendation. It does not necessarily have to be a long justification, but should provide a perspective on the paper and why it should be accepted/rejected. –  Peter Jansson Commented Oct 30, 2013 at 13:45

Speaking from the point of view of an editor: One of the best reviews I ever got was longer than the paper. The author, a young researcher, had proved three theorems, one of which I recognized as a known result. So I asked the original discoverer of that known result to referee the paper. In my cover letter, I mentioned that I recognized one of the theorems as his, and asked whether the other two theorems had enough novelty for a publication. It turned out that the other two theorems weren't new either. The referee could easily have just given citations for those two theorems and recommended rejection. Instead, he gave me (or, really, gave the author) a long, clear explanation of the state of the art in that subject, and he suggested some open problems that the author could try working on.

• 6 I at least once wrote a review longer than the manuscript itself. The paper pushed some buttons of mine, and I couldn't stop writing. (Similarities with answers of mine here at Academia.SE are coincidental.) I think I recommended rejection and explained in considerable detail the non-trivial error the authors had fallen into. The editor appears to have appreciated this review; he still has me review regularly and introduces me as "the guy that writes reviews longer than the original paper." –  Stephan Kolassa Commented Dec 2, 2015 at 20:00

An "accept as is" option is useful after resubmissions; it signifies that no more work needs to be done. However, it is unusual to see that happen in an article on the first round of submissions. (I've had that happen precisely once in my career.)

Otherwise, I would say that the more detailed a review can be, and the more precise the suggestions for improving the paper are, the better it will be.

One to two pages is typically the norm; however, I have submitted a few three- to four-page reviews when I thought an article was already quite good, but could be better.

On the other hand, if a paper is already of relatively poor quality, I will explain the methodological or other significant flaws, but skip over an analysis of minute points; (it's simply not worth the time to rearrange the furniture when the roof is going to collapse any minute.)

As an author, reviewer, and key reader of a respected engineering journal, I can offer some perspective. The shortest review I received was one I solicited from a highly-respected professor at a prestigious university. His review was basically “This manuscript is not written well enough to be reviewed.”

The longest review I have received as an author was about five bulleted comments some of which were optional revisions and some minor but necessary clarifications; the shortest was one minor comment approving the manuscript.

As a reviewer I have on several occasions completely rewritten a non-English language author’s manuscript as a gratis professional service. To my surprise, I received thank you letters from the professional society publications chair and the editor thanking me for my ‘laudatory’ service. I took that to indicate my effort was unusual.

My shortest key reader review summary was to a VERY famous author who after a 22 page derivation, which he summarized as ‘simple’. With feigned seriousness, I ‘required him to remove the word ‘simple’ since he was on this uncustmary occasion communicating with mortals.

You must log in to answer this question.

Not the answer you're looking for browse other questions tagged peer-review editors ..

• Featured on Meta
• Join Stack Overflow’s CEO and me for the first Stack IRL Community Event in...
• User activation: Learnings and opportunities

Hot Network Questions

• How do exchange mail enabled security groups handle emails to multiple groups with overlapping members?
• Can a 20A circuit mix 15A and 20A receptacles, when a 20A is intended for occassional space heater use?
• How is AC and DC defined?
• Which law(s) bans medical exams without a prescription?
• How to interpret odds ratio for variables that range from 0 to 1
• Why is Germany looking to import workers from Kenya, specifically?
• Does it ever make sense to have a one-to-one obligatory relationship in a relational database?
• Horror short film about a guy trying to test a VR game with spiders in a house. He wakes up and realizes the game hasn't started
• Play the Final Fantasy Prelude
• How to avoid repeated input while using newcommand?
• Date not working correctly
• Is it possible to make sand from bones ? Would it have the same properties as regular sand?
• What early 60s puppet show similar to fireball XL5 used the phrase "Meson Power?"
• Establishing Chirality For a 4D Person?
• What is all this —?
• Numerical integration of ODEs: Why does higher accuracy and precision not lead to convergence?
• Confused about the uniform distribution of p-values under the null hypothesis
• Change of variable, u = y(x)
• “…[it] became a ______ for me.” Why is "gift" the right answer?
• Rav Moshe Feinstein's advice on doing teshuvah
• My team is not responsive to group messages and other group initiatives. What should be the appropriate solution?
• Returning to the US for 2 weeks after a short stay around 6 months prior with an ESTA but a poor entry interview - worried about visiting again
• What's "jam" mean in "The room reeled and he jammed his head down" (as well as the sentence itself)?
• Removing undermount sink

Our systems are now restored following recent technical disruption, and we’re working hard to catch up on publishing. We apologise for the inconvenience caused. Find out more: https://www.cambridge.org/universitypress/about-us/news-and-blogs/cambridge-university-press-publishing-update-following-technical-disruption

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > Journals
• > BJPsych Advances
• > Volume 27 Issue 4
• > Peer reviewing made easier: your questions answered

Article contents

Learning objectives, what is peer reviewing, why do peer reviews, who should do it, how to become a reviewer, what to do if invited to review, how long does it take, how do you approach a peer review, what do you write in a peer review, how to write a review when the paper is seriously substandard, controversies in peer review, where else can i find help as a reviewer, conclusions, peer reviewing made easier: your questions answered.

Published online by Cambridge University Press:  26 August 2020

• Supplementary materials

Peer reviewing is a hugely important part of the scientific process that ensures published articles are of sufficient quality to deserve dissemination to the wider scientific community. Building on a previous article published in this journal, this article addresses topics that potential or practising peer reviewers may find useful. These include what peer reviewing is, why do peer reviews, how to become a reviewer, what to write in a review and where to find more information. It includes a template for writing a review, and lists various websites and guidelines that can help ease the entire process depending on what type of article is being reviewed. Peer reviewing can be enormously rewarding and help clinicians diversify their scope of work while also benefiting the scientific community by contributing to the quality control of published work.

After reading this article you will be able to:

• demonstrate a greater understanding of what the peer reviewing process entails

• write a comprehensive review for a peer-reviewed medical journal

• know where to go to for further information.

Peer review has been a formal part of scientific communication since the first scientific journals appeared more than 300 years ago. Philosophical Transactions , a journal launched in 1665, is thought to be the first journal to formalise the peer review process; it is also the first journal to be made freely available online ( https://royalsocietypublishing.org/toc/rstl/1/1 ).

More recently, attempts have been made to establish peer reviewing guidelines. In 1978, a few editors met in Vancouver to establish guidelines for articles being submitted to their journals. This small group was named the Vancouver Group and continues to publish helpful advice and guidance in this area under the name of International Committee of Medical Journal Editors ( http://www.icmje.org ). This article uses some guidance from there as well as imparting knowledge and sharing learning from our own experience that should be useful for clinicians, academics, undergraduates and postgraduates in psychiatry.

Peer reviewing for medical journals is a richly rewarding experience benefiting the reviewer, author and journal. There appears to be a shortage of articles in psychiatry journals for those who have questions about the process and the details of peer review. This article aims to answer those questions and hopefully enthuse more people into becoming peer reviewers. It should be read in conjunction with our earlier article (Halder Reference Halder, Ramsay and Tyrer 2011 ), which discussed ‘the nuts and bolts’ of how to do it. Since 2011, N.H. has adapted and changed the style of peer reviewing based on experience and feedback, and this is shared in the present article.

A peer-reviewed journal is one that publishes articles that have been checked by people who are ideally experts in the field and is available to the wider scientific community. Peer review is the critical assessment by unbiased and independent experts of manuscripts submitted to journals. It can be seen as an important extension of the scientific process and it aims to validate academic work.

Peer reviewers are people deemed suitably qualified to act as independent assessors of such articles. They will have a duty to the authors to try to improve the standard of the article by constructive feedback. It can be seen as a form of ‘quality control’ for the journal. Indeed, 82% in a survey agreed with the statement that ‘without peer review there is no control in scientific communication’ (Ware Reference Ware 2016 ).

The reviewer will also have a duty to the journal to help the editor decide whether or not an article should be accepted (albeit with some amendments). Ultimately it is the editor who makes the decision to accept, revise or reject a paper.

The key stages of the peer review process are outlined in Box 1 .

BOX 1 What is the process of peer reviewing?

1 Editor receives a manuscript submission

2 Editor can choose to reject straightaway

3 If the manuscript has potential, editor will involve usually two peer reviewers for comments (picked from a database or suggested reviewers that the authors have provided)

4 If there is a wide discrepancy in scores, an additional reviewer will be involved

5 Editor makes final decision

The new decade has brought with it people in growing numbers who appear to be sceptical or even dismissive of the notion of evidence-based information, whether that is reflected in growing numbers of so-called antivaxxers (Hussain Reference Hussain, Ali and Ahmed 2018 ) or climate-change deniers (Gross Reference Gross 2018 ) to name but two. It has therefore never felt more important a time to push back against this narrative and one part of doing that is either to engage in research or to act as a peer reviewer to ensure high quality publications.

There are a number of reasons to engage in the peer review process ( Box 2 ). For one, it enhances critical appraisal skills. This helps to avoid losing the great skill that, for many, peaks at the time of their professional membership examinations. Some consultants mention a desire to engage in something different from their day-to-day routine. Not only does peer reviewing break up the clinical work, but it also adds a new skill set to consultants’ repertoire. It is something that can then be added to a CV and subsequently help in future job interviews. From a research perspective, reviewers benefit from reading about new research at the earliest stage and a chance to shape the final product. Simply having more knowledge about how and what reviewers look for increases the chances of having their own work published in future, as they will know what to look out for. As a general rule, it is desirable for peer reviewers to maintain an interest in a few specific areas of research where they have special knowledge.

BOX 2 Reasons for doing peer review

• Enhances critical appraisal skills

• Adds new skill set

• Being part of the scientific research community

• Good for CV

• Breaks up clinical work

• Helps with writing your own work for publication

• Enjoyable

• Contributes to the scientific process

• Enhances reputation/career

Arguably most importantly, peer reviewing can be quite enjoyable. Part of the enjoyment could be the internal satisfaction of being part of an important process of quality control and making a contribution to the scientific process, ultimately benefiting their medical colleagues.

An added bonus, particularly for trainees and new consultants wishing to enhance their CV, is that many journals are now registered with Publons ( publons.com ), a free website that provides a service to track and verify the work of peer reviewers.

The most popular reasons for participating in peer review reported in one survey (Ware Reference Ware 2016 ) related to social factors (playing a part as a member of the community was endorsed by 93%, and reciprocating others’ reviewing work by 75%) and intrinsic factors (enjoy helping improve the paper: 83%; enjoy seeing work ahead of publication: 72%). Instrumental or self-interested reasons were much less cited (e.g. to increase the chance of future acceptances: 16%; to increase the chance of a place on the editorial board: 24%; to enhance reputation or further career: 42%).

The answer to this is simple. Anyone can become a peer reviewer. All consultants can make a contribution, given the experience they have already gained in reaching this level in their career. However, trainees and even undergraduates can also make meaningful contributions. If anyone has completed a masters or PhD, it is likely that they will know more about the subject matter than many others. They can put themselves forward to be a peer reviewer for that subject. Trainees can certainly comment on a wide variety of issues, including education- and training-related subjects, having been ‘experts by experience’. A growing number of medical student journals are peer reviewed, when appropriate, by other medical students. This will help home in on skills from a very early stage, which will continue to improve with experience and time.

You can contact the editor or the administrator of a journal in the first instance. All journals have email addresses or contact details on their website or inside the journal itself. As a minimum, the details that you should include in this first contact are your qualifications and experience, followed by your areas of expertise and interests (with details why you consider this to be so). If you have any previous publications these should be listed. You should also provide your CV. If accepted, your name and details will be placed on a database so that the editor can pick you if a submission that matches your expertise or interest comes in.

There are many hundreds of peer-reviewed medical journals available and it may be difficult to choose which one or ones to be a peer reviewer for. There are different ways to choose. For example, you can simply choose journals that you enjoy reading. Others choose by looking at affiliated bodies (e.g. only choosing journals that belong to the Royal College of Psychiatrists or another body). Others choose journals that tend to take submissions from a particular group, such as medical students or trainees. Some search by looking at the impact factor, which is considered to be a proxy measure for the quality of a journal (although not all peer-reviewed journals have an impact factor). Or you can simply list your areas of expertise and search on the internet for the journals that most closely match those areas. Readers are sometimes surprised at how accurately their interests are matched with a particular journal. As an anecdote, N.H., currently working with people with brain injuries, searched for journals on neurology, neurosurgery and psychiatry, as his work involves liaising with these groups of people. The first search item located was the Journal of Neurology, Neurosurgery and Psychiatry .

In some cases, you may be asked by a journal to review a paper after they found your details on a database such as PubMed or via a recommendation. (If, however, you have not reviewed a paper for a long time, you may be removed from the database of reviewers at the discretion of the editor.)

The Committee on Publication Ethics (COPE) has published helpful guidelines on things to look for before agreeing to take on a review (Committee on Publication Ethics 2020 ). These include checking that you have the necessary expertise to assess the manuscript. Note that if you would like to do the review but do not quite feel experienced enough, some journals would accept a review co-signed by your supervisor. Other important issues include declaring any conflicts or competing interests and only accepting an invitation if you can meet the deadline or have agreed an extension with the journal. Unfortunately, and sadly increasingly, you also need to check that the journal is legitimate. There has been an increase in so-called ‘predatory journals’, where certain open access journals exist for revenue rather than scholarly activity and often send out spam emails to solicit contributions with little concern for quality (Kisely Reference Kisely 2019 ).

It is important to note that you do not have to ‘accept’ all requests for peer reviewing. Clinicians go through phases of being busier at certain times and there is no pressure to accept all requests for reviews that you receive. When this arises, you may be given the option to nominate or suggest other suitable reviewers for the paper. This is much appreciated by editors, who often find it difficult to get reviewers for highly specialised subjects.

Of course, this is subjective and dependent on a number of variables. With more practice the time taken can decrease significantly. When N.H. first started peer reviewing as trainee editor of the Psychiatric Bulletin , the process could take a weekend of dipping in and out. Now it can take an average of about 2 h. This is in keeping with findings from a study in the British Journal of Psychiatry (Walsh Reference Walsh, Rooney and Appleby 2000 ), which reported that the mean time taken to complete an open review (where reviewers’ names were published) was 2.05 h and 1.65 h for masked (‘blind’) reviews (where reviewers’ names were not known to the authors). Footnote † However, another survey (Ware Reference Ware 2016 ) found the median time per article spent reviewing was 5 h, with a reduction in the mean time needed per review with increasing age from the under-36s (9.6 h) to the over-65s (5.8 h).

You must resist the temptation of skim reading to save time. Authors have invested a significant amount of time from their busy schedule to complete the work that sits in front of you. It is crucial that you take enough time to give the authors the respect they deserve. You can, however, save time by only producing one review for both the authors and the editor, as these versions can be the same, with some very minor adjustments.

Everyone may have their own way of approaching a review, and if it works for them then the advice would be to stick with this. Here is a suggested method. If you are not familiar with the journal, read the its aims and scope and instructions for authors to get an idea of what it is looking for. Once the full article is obtained, read it through once completely without breaks. Resist critiquing on the first read through, tempting though this may be. After short break, a second read through could be accompanied by written notes or questions that arise as you read the article. N.H. tends to print out the paper and write in the margins or between the lines. You may then write a first draft of the review (one way of setting this out is given in the next section). It is helpful then to leave it overnight or even longer. This allows you to organise your thoughts or formulate new ideas to help improve the article. Some say that if you go back to a review after a period it can sometimes appear that you are reading it with ‘fresh eyes’, generating fresh ideas. These can be incorporated into a revised draft. Finally, read through your review thoroughly to make sure it makes sense and does not itself have typos or grammatical errors. It is embarrassing to be critical of the authors’ poor grammar when your own writing is error-laden. For the review that the authors will see, you need to make sure that it addresses the main objective of improving the paper in a way that is encouraging and constructive. For the review for the editor, you have to ask whether this review helps the editor to make the decision about publishing the paper in their journal. Once you have completed and submitted your review you should destroy or delete any copies of the original article.

Although a template was given in our earlier article ( Reference Halder, Ramsay and Tyrer Halder 2011 ), N.H. has changed his approach to laying out a review, in light of experience and feedback. The original article advocates for a summary, followed by major points and then minor points. It is now felt more helpful for all parties to lay out the review in the order of how the authors have written their paper, using the subheadings that they use. This makes it easier to follow and appears to be logical. A summary is still useful to both editors and authors, but any recommendations about whether to reject or accept a paper should only be sent to the editor and not authors. Box 3 outlines a revised template for writing a review, and we discuss each element in more details in the rest of this section.

BOX 3 Template guide for structuring reviews

• The article title

• A summary of the review

• Recommendation (to be included in review for editor only)

Next, address the following areas:

• Title: Does this succinctly and accurately describe the main theme of the article?

• Abstract: Does this summarise the main points from the text?

• Introduction: Does the paper demonstrate an adequate understanding of the relevant literature and theory in the field?

• Method: Can this be replicated on the basis of the available information?

• Results: Are they accurately described using the correct empirical tools?

• Discussion: Do the conclusions follow clearly from the arguments/points made?

• Limitations: Does the paper recognise its limitations?

• References: Are they relevant, up to date and correctly cited? Has any important work been omitted?

Finally, give a statement under each of the following headings:

• Originality

• Readability

• Topicality

• Likely appeal

• Generalisability/implications

In practice this means that (after the brief summary) you should start with the title. Does it accurately reflect the work conducted? Some journals specifically state that they do not want the title in the form of a question. Some journals have guidelines on the word count and format of the title, but that can be left to the copy editors. The reviewers can suggest amendments to the title or even alternatives as they see fit. Some journals will provide a specific template for reviewers.

The next section is usually the abstract. Together with the title, these make up the first thing readers will read when they are searching for journal articles. If the article is not open access (meaning freely available on any platform in full) then the title and abstract could be the only text that is visible without having to pay, subscribe or have special authorisation to the database in which the article is archived. The abstract should therefore not be overlooked or only given passing remarks. The abstract should contain the key points and findings from the main text succinctly. The key question to answer is: Does this accurately reflect the main text? Previous work has highlighted this can be a major problem (Pitkin Reference Pitkin, Branagan and Burmeister 1999 ).

The rest of the review should follow the subheadings as given in the article being reviewed; for the majority of scientific papers this will follow the IMRAD headings (Introduction, Method, Results and Discussion).

Introduction

A reviewer after reading the Introduction section should be able to answer the question: Why was this paper written? The authors should provide a brief context or background for the study (rather than a whole history on the topic). Any purpose, aim, objective or hypothesis should, ideally, be included here. Sometimes authors include data or conclusions from the work being reported, but these should appear later in the relevant section. Comments should be made about the theoretical underpinnings to the work carried out. Does the paper demonstrate an adequate understanding of the relevant literature and theory in the field? Is this referenced appropriately? Is the theory clearly and succinctly presented within the submission? Is any significant work ignored? Ideally, as the reviewer you should already be an expert in the field and so should know the relevant papers and omissions. If these are not at your fingertips, it is worthwhile doing a literature search of your own to check.

Regarding the Method section, the questions to ask include: are the aims of the paper clearly stated? Is the methodology described sufficiently so that it could be replicated? Are the correct statistics used? If it is a primary research paper, are there power calculations? Authors should clarify why any variables or cut-off points were used. If ethical approval was necessary, there should be information contained here. You should comment on any omission of this. Likewise, concerns of misconduct should be sent to the editor immediately.

Results should be presented clearly. You can note if you feel there is a better way of conveying the results that the authors have overlooked. Tables and figures are most often included in this section and the editor expects the peer reviewers to comment on these. Specifically: Do the tables capture information concisely and display it efficiently? Do they provide enough information? Including data in tables rather than text frequently makes it possible to reduce the length of the text. Comment if data in tables or figures are repeated in the text. As a reviewer you may be confident enough to check the statistics yourself, but if in doubt you can ask the journal statistician to comment. If there is no in-house journal statistician, you need to be open and honest about your remit and say this in the review. How adequate is the analysis, including qualitative and/or quantitative analysis?

The Discussion section should adequately tie together the other elements of the paper. It should answer the question: What do the results mean? Do the conclusions reached follow clearly from the arguments/points made? Have the authors emphasised any new and important aspects of the study? As a peer reviewer it is important to check that authors have not repeated in detail data or other material given in other sections. The conclusions should be linked with the aims of the study, and any unqualified statements and conclusions not adequately supported by the data should not be there. A paragraph on limitations of the study is useful here. As a peer reviewer you need to ask: Does the paper recognise its limitations and consider these well?

With references, you would have already mentioned any omissions. It is more helpful to provide details and specifics of the references you think are important instead of using airy language such as ‘the authors should be aware of the work by Bloggs et al carried out in the 1990s’. Include some comments about whether you think the references provided are relevant, up to date and correctly cited. This includes checking that authors have adhered to the journal style. Copy editors and typesetters are invaluable here in checking the article is in keeping with the journal requirements, but many are paid per page or per 1000 words so it can become costly to fix references and other areas that are not to style. References to papers accepted but not yet published should be highlighted; authors should obtain written permission to cite such papers, as well as verification that they have been accepted for publication.

Other areas for comment

After commenting on the sections above, it is useful to write a paragraph on the following areas.

Originality

Original ideas make for interesting reads, and medical journals, like newspapers, need readers to survive.

Readability

This would include whether there is a logical flow throughout. Each section should flow from the one preceding it. The aims described early on should be addressed in full. Some journals have guidance on how the paper should read. For example, some favour short, succinct sentences and ask authors to avoid the passive sentence structure. You can include comments on the English style and grammar. It is also acceptable for reviewers to simply comment that the spelling and grammar are poor, rather than going through with a fine-tooth comb to correct it themselves, and there is ongoing debate whether this task falls within the remit of reviewers at all (see ‘Controversies in peer review’ below).

This links in with originality in the sense that readers are more likely to read articles on topics that are trending or being actively discussed in scientific or wider circles for whatever reason.

Have there been any other studies that can confirm the authors’ findings? If the study is the first of its kind, there should be a mention of having to replicate it to ascertain that the results have not been due to chance alone.

Likely appeal

This relates to the appeal for that particular journal and the appeal to its readership. Many papers are rejected because the question being asked is not interesting enough or does not add enough to the body of scientific evidence.

Generalisability or implications for practice, research and/or society

Does this paper inform practice for the reader and contribute to the field of theory, research and/or practice?

Are clear implications for the reader demonstrated? Are the implications consistent with the findings and conclusions of the paper?

Throughout, the reviewer should always have in mind what can be done to improve the paper, and this question alone should guide all the writing.

Because the peer review process includes confidential comments to the editor that will never be seen by the authors (unless they go so far as to request information via the Freedom of Information Act) the frank opinion of a very poor paper should be made primarily to the editor. The comments to the authors should be robust, accurate and relevant, but not rude. The first paper one of us (P.T.) sent many years ago to the British Journal of Psychiatry received an angry response, ‘Why on earth is this registrar carrying out research at his age when he should be carrying out his clinical duties?’ This now would not be tolerated.

A very poor paper will naturally tend to receive a shorter report than a better one.

Journals that do not operate a peer-review process tend to be considered of not high quality because of the risk of editorial bias and, at worst, fraud. A survey revealed that, in addition to detecting the best manuscript for the journal, growing numbers of people believe that peer reviewers should also be looking to detect fraud (i.e. the results given are falsified) and plagiarism, although acknowledging difficulties in actually doing this (Ware Reference Ware 2016 ). Indeed, one study found that, out of 400 consecutively submitted manuscripts to a major specialty medical journal, 17% of submissions contained unacceptable levels of plagiarised material, with 82% of plagiarised manuscripts submitted from countries where English was not an official language (Higgins Reference Higgins, Lin and Evans 2016 ). However, to detect fraud, reviewers would potentially need to check and verify the raw data of a study, which could be impractical and discourage prospective reviewers from participating in the peer review process (Glonti Reference Glonti, Cauchi and Cobo 2019 ). Some may argue that peer review should not involve ‘policing’ other people's work, especially when there is sophisticated software available, such as Turnitin ( turinitin.com ), that can be tasked to do this. The Royal College of Psychiatrists has recently formed a Research Integrity Group and this will be focused on, among other things, the role of reviewers in spotting fraudulent, plagiarised or poor research.

There is an ongoing debate whether peer reviewers should engage in copy-editing. Although the majority of articles examined stated that copy-editing does not fall within the duty of peer reviewers, several articles specifically mentioned that reviewers should offer grammatical and linguistic improvements (Glonti Reference Glonti, Cauchi and Cobo 2019 ). Many journals have their own copy-editing team, who are specifically trained to identify and address such aspects of the manuscript. Reviewers may argue that their time should be focused on the improvement of scientific content rather than the linguistic fine-tuning.

There are guidelines freely available online for almost all types of research ( Box 4 ). These can be consulted to help inform your review, and they are especially useful if you are struggling to benchmark what has been written against a standard. The International Committee of Medical Journal Editors (ICMJE) provides guidelines for biomedical journals in general ( http://www.icmje.org ).

BOX 4 Guidelines for particular studies that may help reviewers

• Observational studies: STROBE

• Systematic reviews: PRISMA

• Study protocols: SPIRIT

• Diagnostic/prognostic studies: STARD

• Case reports: CARE

• Clinical practice guidelines: AGREE

• Qualitative research: SRQR

• Animal pre-clinical studies: ARRIVE

• Quality improvement studies: SQUIRE

• Economic evaluations: CHEERS

Links to the above guidelines can be found at https://www.equator-network.org

COPE produces guidelines on the peer reviewing process from an ethical standpoint, including what to do if you suspect fraudulent practices ( https://publicationethics.org/peerreview ).

The Royal College of Psychiatrists runs annual or biannual 1-day research methods courses. There are hands-on and practical, facilitated by experienced reviewers. These are advertised on the College website and in the newsletter.

We hope that this article, alongside our original article (Halder Reference Halder, Ramsay and Tyrer 2011 ), has enthused a new generation of reviewers and continues to be a source of guidance for existing reviewers. Peer reviewers play a crucial role in the editorial process, and yet their role and tasks tend to be poorly defined (Glonti Reference Glonti, Cauchi and Cobo 2019 ). We hope this article has helped answer some of the key questions involved in this process. N.H. is happy to be contacted if needed.

Acknowledgement

We thank the three reviewers who helped with their comments on how to improve this article.

Author contributions

N.H. led on the writing, editing and responding to the reviewers’ comments. P.T. contributed to the writing, editing of the drafts and approving the final version. P.C. contributed to the writing, editing of the drafts and approving the final version.

Declaration of interest

P.C. is Editor of BJPsych Advances . She has had no involvement in the editorial process or peer reviewing of this article.

ICMJE forms are in the supplementary material, available online at https://doi.org/10.1192/bja.2020.62 .

Select the single best option for each question stem

a qualitative research

b systematic reviews

c case reports

d diagnostic studies

e quality improvement studies.

a provide a recommendation to the editor

b give constructive feedback to authors

c inform authors if their paper is to be rejected

d help the editor decide on publication

e help to improve the quality of the paper.

a information on any obvious omission of relevant studies

b comments on the methodology

c the time taken to conduct the review

d a constructive critique of the different sections of the article

e suggestions on areas for improvement.

a peer reviewers can reject the offer of a review if there is a conflict of interest

b peer reviewing can involve more than one person

c peer reviewers are usually picked from a database or suggested reviewers that authors have provided

d the editor helps the reviewers by giving an indication of his or her views beforehand

e the editor makes final decision.

a it sharpens the reviewer's critical appraisal skills

b it is well paid

c it contributes to the quality improvement process

d it can be enjoyable

e it helps the reviewer write their own publications.

MCQ answers

1 b 2 c 3 c 4 d 5 b

† For a discussion of the types of peer review, see Shoham N, Pitman A (2020) Open versus blind peer review: is anonymity better than transparency? BJPsych Advances , in press.

Halder et al. supplementary material

Halder et al. supplementary material 1

Halder et al. supplementary material 2

Halder et al. supplementary material 3

This article has been cited by the following publications. This list is generated based on data provided by Crossref .

• Google Scholar

View all Google Scholar citations for this article.

Save article to Kindle

To save this article to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

• Volume 27, Issue 4
• Neel Halder (a1) , Peter Tyrer (a2) and Patricia Casey (a3)
• DOI: https://doi.org/10.1192/bja.2020.62

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox .

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive .

Reply to: Submit a response

- No HTML tags allowed - Web page URLs will display as text only - Lines and paragraphs break automatically - Attachments, images or tables are not permitted

Your details

Your email address will be used in order to notify you when your comment has been reviewed by the moderator and in case the author(s) of the article or the moderator need to contact you directly.

You have entered the maximum number of contributors

Conflicting interests.

Please list any fees and grants from, employment by, consultancy for, shared ownership in or any close relationship with, at any time over the preceding 36 months, any organisation whose interests may be affected by the publication of the response. Please also list any non-financial associations or interests (personal, professional, political, institutional, religious or other) that a reasonable reader would want to know about in relation to the submitted work. This pertains to all the authors of the piece, their spouses or partners.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

Science-policy advisers shape programmes that solve real-world problems

Career Feature 18 SEP 24

Can AI be used to assess research quality?

Nature Index 18 SEP 24

The grassroots organizations continuing the fight for Ukrainian science

Career Feature 11 SEP 24

The human costs of the research-assessment culture

Career Feature 09 SEP 24

Why I’m committed to breaking the bias in large language models

Career Guide 04 SEP 24

Binning out-of-date chemicals? Somebody think about the carbon!

Correspondence 27 AUG 24

Rise of ChatGPT and other tools raises major questions for research

My identity was stolen by a predatory conference

Correspondence 17 SEP 24

Faculty Positions in Center of Bioelectronic Medicine, School of Life Sciences, Westlake University

SLS invites applications for multiple tenure-track/tenured faculty positions at all academic ranks.

Hangzhou, Zhejiang, China

School of Life Sciences, Westlake University

Faculty Positions, Aging and Neurodegeneration, Westlake Laboratory of Life Sciences and Biomedicine

Applicants with expertise in aging and neurodegeneration and related areas are particularly encouraged to apply.

Westlake Laboratory of Life Sciences and Biomedicine (WLLSB)

Faculty Positions in Chemical Biology, Westlake University

We are seeking outstanding scientists to lead vigorous independent research programs focusing on all aspects of chemical biology including...

Faculty Positions at the Center for Machine Learning Research (CMLR), Peking University

CMLR's goal is to advance machine learning-related research across a wide range of disciplines.

Beijing, China

Center for Machine Learning Research (CMLR), Peking University

Faculty Positions at Huairou campus of Peking University

The Beijing Laser Accelerator Innovation Center at Peking University invites applications for Applied Physics faculty, including tenured positions.

Peking University (PKU)

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Duration and quality of the peer review process: the author’s perspective

• Open access
• Published: 09 March 2017
• Volume 113 , pages 633–650, ( 2017 )

Cite this article

You have full access to this open access article

• Janine Huisman 1 , 2 &
• Jeroen Smits 2 , 3  

67k Accesses

156 Citations

111 Altmetric

12 Mentions

Explore all metrics

To gain insight into the duration and quality of the scientific peer review process, we analyzed data from 3500 review experiences submitted by authors to the SciRev.sc website. Aspects studied are duration of the first review round, total review duration, immediate rejection time, the number, quality, and difficulty of referee reports, the time it takes authors to revise and resubmit their manuscript, and overall quality of the experience. We find clear differences in these aspects between scientific fields, with Medicine, Public health, and Natural sciences showing the shortest durations and Mathematics and Computer sciences, Social sciences, Economics and Business, and Humanities the longest. One-third of journals take more than 2 weeks for an immediate (desk) rejection and one sixth even more than 4 weeks. This suggests that besides the time reviewers take, inefficient editorial processes also play an important role. As might be expected, shorter peer review processes and those of accepted papers are rated more positively by authors. More surprising is that peer review processes in the fields linked to long processes are rated highest and those in the fields linked to short processes lowest. Hence authors’ satisfaction is apparently influenced by their expectations regarding what is common in their field. Qualitative information provided by the authors indicates that editors can enhance author satisfaction by taking an independent position vis-à-vis reviewers and by communicating well with authors.

Similar content being viewed by others

Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts

Submitting the Manuscript for Formal Review: Efficient and Effective Strategies

We Are All Reviewer #2: A Window into the Secret World of Peer Review

Avoid common mistakes on your manuscript.

Introduction

The scientific peer review process is one of the weakest links in the process of scientific knowledge production. While it is possible to review a paper in less than a day (Ware and Mabe 2015 ), it may often lie untouched on reviewers’ desks and in editorial offices for extended periods before it is evaluated. This means a substantial loss of time for the scientific process, which has otherwise become much more efficient in the last decades. There are even indications that the duration of the peer review process may have increased in the last decades (Ellison 2002a ; Azar 2007 ). Hence there are good reasons for a critical look at this process.

To gain insight into the duration and other key aspects of the peer review process, we analyze data from 3500 review experiences submitted by authors to the SciRev.sc website ( www.scirev.sc ). On this website, researchers can share their experiences with the peer review process regarding manuscripts they have submitted to scientific journals. This information can subsequently be used by their colleagues when selecting a journal to submit their work. Information is available on several important aspects of the peer review process, including the duration of the first review round, total review duration, the time editors take to inform authors about an immediate (desk) rejection of a manuscript, the number and quality of referee reports, the time authors take to revise and resubmit their manuscript, and the overall quality of the process as experienced by the authors.

Duration of the first review round—or first response time (Azar 2007 )—is probably most important for scientific authors as it determines how much time may be lost if the outcome is negative (Solomon and Björk 2012 ). The number of review rounds and the time journals take to manage these rounds are also important, as these aspects significantly affect the time that elapses until author(s) are informed of the final editorial decision. Another important duration indicator is the immediate (desk) rejection time, i.e., the time taken by an editor to inform authors that the manuscript is not considered fitted for the journal. If this only takes a few days, authors can without much time loss send the manuscript to another journal. However, quite often, editors may take weeks or even months for a desk rejection. This seems unacceptable and may point to a less than efficient organization of the editorial process. If editors take much time to inform authors that they are not interested in the manuscript, they probably will also be rather slow in other aspects of manuscript handling, such as assigning reviewers and processing review reports. The immediate rejection time is thus a major indicator of a journal’s performance.

Besides by the duration of the different steps of the peer review process, total publication time is also influenced by revision time, i.e., the time taken by authors to revise and resubmit the manuscript. This factor is therefore also included in our analysis. It is influenced by the time authors are able and prepared to spend on the revision of the manuscript and by the difficulty of the revisions required. In this connection, it is important also to include aspects of the referee reports. Constructive comments by reviewers may substantially contribute to the quality of scientific papers, while low quality and contradictory referee reports may be a major source of frustration among authors (Nicholas et al. 2015 ). In the SciRev questionnaire, authors are asked about the number of reports they received and how they experienced the quality of the reports and the difficulty of the changes they were required to make.

Besides the measurable factors, such as the duration of the different phases of the peer review process and the number of referee reports, there are also aspects of the process that are more difficult to quantify. Does the editor take questions of the author(s) seriously? Is a reasonable motivation for a (desk) rejection given? Does the editor take an independent position vis-à-vis reviewers when making important decisions? Does the editor advise authors on the importance of specific reviewer comments? Together these aspects affect the author’s experience with the journal and to a certain extent may turn a rejection into a good experience or an acceptance into a bad one. We therefore also analyze the authors’ overall evaluation scores given to the journals for their peer review performance as well as the motivations given by authors for their scores. Because an author’s review experience is influenced by many factors (e.g., the outcome of the review process, the impact factor of the journal, and differences in expectations between scientific fields), we study the overall scores in a multivariate way and also analyze the authors’ scoring motivations.

There are around 28,000 scientific journals worldwide, which publish 2.5 million scientific articles annually, produced by a research community of 6–9 million scientists (Ware and Mabe 2015 ; Jinha 2010 ; Björk et al. 2009 ; Plume and Van Weijen 2014 ; Etkin 2014 ). Many of the published articles have been rejected at least once before they reached the editor’s desk of the journal in which they were published. This means that each year many more manuscripts pass through peer review than are published.

Although there is some variation among journals, the peer review process typically starts with a first evaluation of the manuscript by the editor, followed by a decision to accept the manuscript for peer review or immediately (desk) reject it. If desk rejected, the corresponding author receives a message from the editor that the manuscript is considered not fit for publication in the journal, with or without a brief motivation given for the rejection. A manuscript that has passed this first stage will then be send out for peer review, whereby experts in the field (peers of the authors) evaluate the manuscript and write a referee report. On the basis of these reports, the editor decides either to reject the manuscript or gives the author(s) an opportunity to revise and resubmit it, or—in exceptional cases,—directly accepts it. In case of a revise-and-resubmit, several additional review rounds may follow before a final decision regarding acceptance or rejection is made. If the process takes exceptionally long, the author may decide to withdraw the manuscript and submit it to another journal.

Process too slow

Given the fact that reviewers are often overloaded with academic work, that they are generally not paid for their review work, and that reviews are mostly anonymous, there are few incentives to give high priority to this work (Azar 2007 ; Moizer 2009 ). Hence, while the actual time it takes to write a referee report may vary between a few hours and a day (Ware and Mabe 2015 ), reviewers tend to take several weeks to several months to submit their reports. Apart from the time reviewers take to deliver their reports, the total manuscript processing time of journals is influenced by the duration of the various stages of manuscript handling at editorial offices. Given that these offices often have limited resources and many editors do this work besides busy academic careers, waiting times at the different stages are often (much) longer than strictly necessary.

It is therefore not surprising that one of the most important criticisms of the peer review system is that it is much too slow (Lotriet 2012 ). There are even indications that is has been getting slower in recent decades (Alberts et al. 2008 ). Ellison ( 2002a , 2002b ) documents a slowdown since the 1970s in submission-acceptance duration in economics and suggests a similar slowdown in other fields. A major cause for this is that authors are required to revise their manuscripts more often and more extensively (Ellison 2002a , 2002b ; Azar 2007 ; Cherkashin et al. 2009 ; Björk and Solomon 2013 ). According to Ellison ( 2002a ), review rounds are of quite recent date. In the early 1950s, ‘almost all submissions were either accepted or rejected: the noncommittal “revise-and-resubmit” was reserved for exceptional cases (p. 948).’

From the author’s perspective, first response time is particularly important, i.e., the time that elapses between submission and first response from the editor, be it rejection, acceptance, or a revise-and-resubmit. First response time is important because it often delays the publication of an article more than once, as many manuscripts are rejected once or several times before acceptance (Azar 2007 ; Etkin 2014 ; Pautasso and Schäfer 2010 ). There are indications that duration of the first review round has increased, at least in some fields. Azar ( 2007 ) finds that first response time for economic journals “grew from about 2 months circa 1960 to about 3–6 months in the early 2000s (Azar 2007 , p. 182)”. However, as Azar points out, a longer first response time is in itself not necessarily negative. Economics manuscripts have become longer over time and have more mathematical content, which means it is more time-consuming to evaluate them.

Field difference

Durations vary substantially between scientific fields and even within the same broader discipline. Kareiva et al. ( 2002 ), for instance, studying conservation biology, found that the process from submission to publication took on average 572 days for conservation and applied ecology journals compared to 249 days for genetics and evolution journals.

With respect to the number of times the average manuscript is rejected before it reaches the journal that will publish it, Azar ( 2004 ) arrives at a figure of three to six rejections. Similar to an increase in first response time, there also seems to be an increase in the number of rejections prior to publication. Thomson Reuters (in Ware and Mabe 2015 , p. 51) reports an increase in the rejection rate from 59 to 63% between 2005 and 2010. Regarding the desk rejection rate, Lewin ( 2014 ) reports an increase of up to three times for some journals. Lewin attributes this to increased publication pressure, whereby “governments in countries outside of the USA engage in a process of quantifying the scholarship of scientists in their countries as a way of rationalizing the allocation of national resources to institutions of higher learning in their countries. The unsurprising consequence has been a dramatic increase in submissions to the top journals by scholars from emerging economies as well as from European countries” (Lewin 2014 , p. 169).

Editors are also worried about these developments. ‘Amongst journal editors there are growing concerns that the quality—and duration—of the review process is being negatively affected as “referees are stretched thin by other professional commitments”. This often leads to “challenges in finding sufficient numbers of reviewers in a timely manner” (Lotriet 2012 , p. 27).’ Once reviewers have been found, other problems may emerge, such as poor reviewer agreement on submissions (Peters and Ceci 1982 ; Onitilo et al. 2014 ) or ethical problems (Resnik et al. 2008 ). Reviewers who make contradictory comments are a major source of frustration for authors as well as editors. Regarding unethical practices, Resnik et al. ( 2008 ) mention (in order of frequency) reviewers asking authors to include ‘unnecessary references to their publication(s), personal attacks, reviewers delaying publication to publish a paper on the same topic, breach of confidentiality and using ideas, data, or methods without permission (p. 305)’.

Ways to improve

Several suggestions have been done to make it more attractive for scientists to act as reviewers. Free subscription to journal content, annual acknowledgement on the journal’s website, more feedback about the outcome of the submission and quality of the review, appointment of reviewers to the journal’s editorial board and financial incentives (Tite and Schroter 2007 ). A noteworthy initiative in this respect is Publons ( www.publons.com ), a website where reviewers can upload information on anonymous review work they performed. This information is then verified with the journals and can subsequently be used as ‘proof’ of the peer review work done by the reviewer. This initiative provides a solution to the recognition problem. However, it does not help solve the problems of duration and quality as neither the time reviewers spent writing the reports nor the quality of their reports are registered.

As to financial incentives, Thompson et al. ( 2010 ) found a statistically significant reduction in review duration when referees were paid for their efforts. ‘Median first response time was reduced from 90 to 70 days, a 22% reduction in the presence of payments. With payments, only 1% of first response times exceeded 6 months; without payments, 16% exceeded 6 months (Thompson et al. 2010 , p. 678).’ Although it was not possible to compare the quality of referee reports submitted with or without payment, they thought it likely that if the length of referee reports was an indication of quality, payment might even have led to an increase in referee reports’ quality: “[r]eferees did not dash off shorter reports to meet the deadline for payment; in fact, reports were statistically significantly longer with payments than they were prior to payments” (Thompson et al. 2010 , p. 690).

Previous studies by Hamermesh ( 1994 ) for seven journals in 1989 also found an increase in timely referee reports for journals offering payments. However, since “some empirical evidence suggests that when voluntary economic activities—giving blood, volunteering to work for public or private institutions, and collecting donations for charity, for example—are rewarded with relatively low payment levels, low-paid performance is inferior to voluntary performance” (Thompson et al. 2010 , p. 680), most likely reviewers would have to receive a realistic rather than a symbolic payment for their efforts.

It seems natural to expect that authors of papers that have been accepted are happier with the review experience, when they look back at it in hindsight. Authors tend to suffer from attributional bias. If their paper is rejected, many authors tend to blame this on situational factors, such as incompetent reviewers or uninterested editors, but in case of acceptance tend to attribute this to their own expertise and competence in writing high-quality papers (Garcia et al. 2016 ). The difference in ratings between authors of accepted and rejected manuscripts might also be greater, the longer the duration of the peer review process. The more time and energy authors invest in a manuscript, the more likely it is they will be disappointed by a rejection, and even more so if rejection follows after several review rounds.

The data used in this paper are based on 3500 review experiences, reported by authors between 2013 and 2016, by filling in a questionnaire on the SciRev.sc website. The SciRev questionnaire contains questions on the duration of the different phases of the peer review process of research articles, on the number, quality, and difficulty of the received referee reports, on the outcome of the peer review process, and on whether the manuscript has previously been submitted to another journal. It also asks authors to provide an overall rating of the review experience and gives them the opportunity to motivate their rating. Research articles may include any paper submitted to a scientific journal (regular research papers, review articles, rapid communications, research notes, etc.), provided it has been subjected to peer review.

Authors who submitted a review to SciRev.sc were asked about their affiliation, which was checked by asking them for their institutional email address and sending a confirmation link to that address. Authors who registered with a noninstitutional email address, because for various reasons they could not provide an institutional one (e.g., job change or working in a non-Western institute without good ICT services), were asked for additional information to check their identity. Reviews were only accepted if the author’s identity was confirmed. Reviews of accepted papers were additionally checked at the journals’ websites; these reviews were only included if the author had indeed published a paper in the journal during the period mentioned.

Although the data are not based on a representative sample of author experiences, they are interesting because they paint a broad picture of the range of author experiences from different fields of study. Each submitted review represents the experience of an author and is important as such. If other authors report similar experiences, this would point toward a specific pattern. And if the resultant patterns differ among scientific fields, this would indicate that the prevalence of specific experiences differs among those fields.

There is little reason to expect authors from different fields to be fundamentally different in the way they experience the different aspects of the peer review process. However, there might be different expectations between fields about review duration and hence about what is considered a long process. Besides by field differences, experiences may also be colored by the process outcome and the journal’s impact factor. We therefore split the figures presented in this paper according to scientific field and process outcome (accepted/rejected) and also study relationships with the journal’s impact factor. Information on the impact factor was derived from the journal’s website and other Internet sources. This information could be found for 3126 reviews. In our analysis, we use the natural logarithm of the impact factor, as more journals are concentrated in the lower ranges of the impact factor.

Of the 3500 review experiences, 572 (16.3%) referred to manuscripts that were rejected without being sent to reviewers, 693 (19.8%) that were rejected after the first review round, 2128 (60.8%) that were accepted after one or more review rounds, 43 (1.2%) that were immediately accepted without peer review process, and 64 (1.8%) that were withdrawn by the author. Given the relatively small number of reported cases of manuscripts that were withdrawn or immediately accepted, these were not included in our analysis. We also removed some extreme cases regarding immediate rejection time (>62 days; 53 cases), duration of first review round and total review duration (>100 weeks; 15 cases), and duration of revision after first review round (>300 days; 6 cases). The extreme cases were not concentrated in specific fields.

Information on the various aspects of the peer review process is presented for all review experiences, separately for accepted and rejected papers and for ten major scientific fields: (1) General journals ( n  = 172), (2) Natural sciences ( n  = 1408), (3) Engineering (including technology; n  = 518), (4) Mathematics and Computer sciences ( n  = 375), (5) Medicine ( n  = 640), (6) Public health (including health professions; n  = 348), (7) Psychology (including education; n  = 355), (8) Economics and Business (including law; n  = 318), (9) Social sciences ( n  = 553), and (10) Humanities ( n  = 178). Given that a substantial number of journals have a broad scope and therefore include more than one scientific field, the sum of the reviews in the different fields is higher than the total number of reviews.

At the end of the SciRev questionnaire, authors are asked to give an overall rating of their review experience. Because this experience is influenced by many aspects of the peer review process, besides providing descriptive figures, also a multivariate regression analysis is performed. In this analysis, the variation in the rating is explained on the basis of relevant characteristics of the process, i.e., whether or not the paper was accepted or rejected, the duration of the first review round, the number of review rounds, the number of referee reports received in the first review round, whether the author is from an English-speaking country, and the scientific field of the journal. We present both direct effects of these factors and significant interactions between them. For journals covering several scientific fields, we only included the journal’s main field in this analysis.

In the multivariate analysis, we excluded reviews of papers that were withdrawn, immediately accepted, or desk rejected. Among the remaining 2821 reviews, there were some missing values. Five reviews for which duration of the first review round was missing were given the average duration of the first review round. Two reviews where the language of the reviewer was missing were included in the non-English (biggest) category. For 289 cases the impact factor was missing. These missings were addressed using the dummy variable adjustment procedure [imputing the mean and including a dummy indicating the missings (cf. Allison 2001 )]. Results of the analysis with missing values dealt with in this way were substantially the same as those with all missings removed from the data.

The overall rating of the review experience is measured on a scale running from 0 (very bad) to 5 (excellent). The outcome of the peer review process is a dummy indicating whether the paper was accepted (1) or rejected (0). The duration of the first review round is measured in days. To indicate language background, we included a dummy indicating whether (1) or not (0) the organization where the author works is located in a country where English is the main language used in daily life (i.e., United Kingdom, Ireland, USA, Canada, Australia, New Zealand, South Africa, and British Indian Ocean Territory). Of the 3500 reviews, 2516 were submitted by authors from non-English-speaking countries. Regarding the distribution of reviews over continents, 557 were obtained from Canada and the USA, 96 from Latin America and the Carribean, 2099 from Europe, 470 from Asia and the Pacific, 190 from the Middle East, 83 from Africa, and 5 of which the continent is not known. For the dummies for scientific field, deviation from mean (effects) coding is used. The dummies therefore indicate to what extent the overall rating within the field is higher or lower than the mean of the fields (Hardy 1993 ).

After rating the overall review experience, authors are given the opportunity to motivate their rating in a few words or sentences. These motivations are published online with the reviews, if permission is given by the author. They paint a sometimes revealing picture of what researchers experience in their attempts to get their work published. To supplement the figures presented in this paper with qualitative information, we analyzed the 1879 motivations available in the 3500 reviews studied.

First response time

For authors, the duration of the first review round, or first response time, is probably the factor they are mostly interested in, as this takes up a substantial part of the total manuscript evaluation time and to a large extent determines how much time is lost if the outcome is negative. First response time includes the time taken by the journal for a first evaluation of the manuscript, finding reviewers, the time the latter require to do their work, and the time the editor then requires to evaluate the manuscript in light of the referee reports and to inform authors about the decision.

As can be seen in Table  1 , the reported first response time in the SciRev data is on average 13 weeks and varies considerably among scientific fields. It took 8–9 weeks in Medicine and Public health related journals, 11 weeks in Natural sciences and General journals, 14 in Psychology, and 16–18 weeks in Social sciences, Humanities, Mathematics and Computer sciences, and Economics and Business. These figures differ between accepted and rejected manuscripts, with first response time of rejected manuscripts taking, on average, 4 weeks longer.

While writing a peer review may take between 4 and 8 h, in only 19% of all reported cases authors were informed about the outcome in less than a month. In about one third of the cases (32%) authors had to wait 3 months or more and in 10% of the cases even more than 6 months before being informed. Duration differs widely between scientific fields. In Social sciences and Humanities, only 7–8% of the authors were informed within 1 month versus 25% in Natural sciences and 27–28% in Medicine and Public health. In Economics and Business and Mathematics and Computer sciences over one sixth (18%) of authors had to wait 6 months or longer.

It is yet unclear to what extent the long duration of the first review round is the result of the peer review process as such and to what extent it is due to (in)efficient manuscript handling at editorial offices. Given that immediate rejection times are often long (see Table  3 and its discussion below), it seems that inefficiencies at editorial offices also play an important role. The finding that in Medicine and Public Health—where professionalization of journals is relatively high—first response times are the shortest, also points in this direction.

To test this idea further, we looked at the relationship between the journal’s impact factor and first response time. As highly ranked journals generally have more resources at their disposal and thus probably better organized editorial offices, and as reviewers are more motivated to review for those journals, we expected to find a negative relationship. Pearson correlations between first response time and impact factor indeed confirm this expectation. These correlations are significantly negative for all scientific fields combined ( P  = −0.29) as well as for all scientific fields separately, with General journals ( P  = −0.51), Mathematics and Computer sciences ( P  = −0.27), and Natural sciences ( P  = −0.26) having the highest correlations. The only exception was Humanities, where no significant correlation between first response time and impact factor was found. This might be because this field traditionally values publishing books more than publishing in journals (Ware and Mabe 2015 ).

Total review duration

Total review duration refers to the time a manuscript is under responsibility of the journal. Besides by the duration of the first review round, total review duration is also determined by the number and duration of subsequent review rounds. Total review duration does not include the time taken by authors to revise and resubmit their manuscript. Given that rejected manuscripts have on average less review rounds, we restrict this analysis to accepted papers.

Table  2 shows that the reported total review duration of accepted manuscripts is on average 17 weeks. Again there are substantial differences between scientific fields. With 12–14 weeks, average total review duration is shortest in Medicine, Public health, and the Natural sciences. It is longest in Economics and Business, where the process takes on average 25 weeks and is twice as long. In Mathematics and Computer sciences, Social sciences and Humanities, total review duration is also long, i.e., 22–23 weeks. Hence the differences in the duration of the review processes we observed for the first review round are also present in the other aspects of the process.

If we split out the data further, we note that in Natural sciences, Medicine, and Public health 13–16% of the manuscripts pass through the entire peer review process within 1 month, that this applies to about two thirds of the manuscripts after 3 months, and to 87–92% of the manuscripts after 6 months. In Mathematics and Computer sciences, Social sciences, and Humanities, these figures are 3–4%, one third and slightly above two thirds, respectively. Whereas only 8% of the authors in Medicine had to wait more than 6 months, this applies to one third of authors in Social sciences and Economics and Business.

The total time a manuscript is with the journal is determined by the time a journal takes for a review round and by the number of review rounds. As mentioned in the Background-section, there are indications that the number of review rounds has increased in recent years. In our data, the number of review rounds on average amounts to 2.03, with Psychology (2.23), General journals (2.18), Economics and Business (2.16), and Social sciences (2.15) showing a higher average number of review rounds.

Total review duration correlates significantly and negatively (−0.27) with a journal’s impact factor, thus indicating that total review duration is shorter for higher impact factor journals.

Immediate (desk) rejection time

Immediate rejection time is the time an editor takes to inform authors that he or she is not interested in the manuscript (and will therefore not send it to reviewers). Our figures clearly show that immediate rejection time is a major source of unnecessary time loss in the peer review process (Table  3 ). On average, an immediate rejection in Medicine takes 10 days, closely followed by Natural sciences, Public health, and Engineering, taking 11–12 days. Journals in Psychology, Social sciences and Mathematics and Computer sciences take half as long, i.e., 15–17 days. These are relatively high averages, given that in many cases an inspection of the abstract is sufficient to decide that a paper does not fit.

On the positive side, in half (50%) of the reported immediate rejection cases, the editor informed the author(s) within 1 week. However, the data also show that in 17% of cases authors had to wait more than 4 weeks to be informed of the rejection. Several authors even had to wait for more than 3 months, or withdrew their manuscripts after hearing nothing for an even longer period. These are clearly unacceptable practices.

The situation is best in Medicine, where 62% of authors are informed about an immediate rejection within 7 days, followed by Natural sciences and Public health where this figure is 54%. Immediate rejection time is longest for authors in the Social sciences and Mathematics and Computer sciences, where in about 30% of reported cases it took the editor 4 weeks or more to inform author(s) that he or she was not interested in the manuscript and would not to send it to reviewers. There is a significant negative correlation (−0.18) between immediate rejection time and the journal’s impact factor, which indicates that journals with a higher impact factor have editors who work faster and editorial offices that are more professionally organized.

Reviewers are generally blamed for long processing times, but our findings indicate that manuscript handling at editorial offices plays an important role too. If editors take a month for an immediate rejection decision, they are probably also slow in finding reviewers and processing referee reports.

Referee reports

The average number of referee reports is about 2.2 in all scientific fields (see Table  4 ). This correspondence is remarkable, given the substantial differences between fields in other respects. There is slight variation in the experienced quality of the referee reports between the fields [as indicated on a scale running from 0 (very bad) to 5 (excellent)]. Authors report the quality of the reports to be somewhat higher in Natural sciences, Engineering, and Public health (3.7), and lower in General journals, Psychology, and Economics and Business (3.4). It is interesting that the long review duration in Economics and Business did not translate into referee reports experienced of higher quality.

Authors who were given the opportunity to revise and resubmit their papers were also asked to what extent they perceived the requested changes as difficult and whether they thought their manuscript had improved as a result of the revision. There is a significant positive correlation (0.40) between these factors. When the revision was experienced as more difficult, authors were also more satisfied with the improvement. Regarding the difficulty experienced, revision processes were perceived as easiest in Mathematics and Computer sciences and in Public health (2.6), and as most difficult in Economics and Business (3.3). Regarding the experienced improvement of the manuscript as a result of the revision, authors from Social sciences, Economics and Business, and Humanities reported somewhat higher figures (3.8 and 3.9) compared to the other scientific fields (3.7).

There is a small positive correlation (0.07) between the difficulty experienced regarding the referee reports and the impact factor of the journal. Thus, reviewers of more highly ranked journals tend to make somewhat greater demands on the authors. The degree of improvement experienced regarding the manuscript is not significantly related to impact factor.

Revision time

The time from the first submission date to the final decision date is not only influenced by the time the manuscript is at the editorial office or being reviewed, but also by the time authors take to revise their manuscript. It is therefore important to look also at the duration of the revision time. Table  5 shows that authors who received a revise-and-resubmit on average take 39 days to revise their manuscript, but there is substantial variation among the fields. Authors in Economics and Business take longest to revise their manuscripts: on average 64 days to prepare and submit a revised version. This is substantially longer than authors in Natural sciences, Engineering and Mathematics and Computer sciences (32–34 days) and in Public health (29 days). Apparently, in Economics and Business it is not only the editors who take more time.

Table  5 also shows the percentage of manuscripts revised within a specific number of days. While 18% of authors in Engineering, Mathematics and Computer sciences and Public health revise their manuscript within 7 days, this applies to 9–10% of authors in Social sciences and Humanities and only 3% of authors in Economics and Business.

Regarding the relationship between the journal’s impact factor and the time authors take to revise their manuscript, we expected authors who received a revise-and-resubmit from a high-level journal to be more motivated to complete the revision of their manuscript quickly. However, no significant correlation was found between revision time and the journal’s impact factor.

Rating of peer review experience

The SciRev questionnaire gives authors the opportunity to provide an overall rating of the review experience on a scale from 0 (very bad) to 5 (excellent); see Table  6 for details. Authors of accepted manuscripts give the peer review process a much higher rating (4) than authors of rejected manuscripts (2.2). Moreover, the rating of the peer review process is negatively related to total review duration. This correlation is −0.43 for both accepted and rejected manuscripts.

To determine how the various factors might affect the satisfaction of authors with the peer review process, we turn to the results of multivariate analyses (see Table  7 ). The first columns show the results of Model 1, which contains all relevant variables. Model 2 contains the same variables but also the significant interactions between the variables.

As can be seen in Model 1, all variables, except impact factor, are significantly related to authors’ rating of the peer review process of their manuscript. As expected, authors of accepted manuscripts rate the process significantly more positive than authors of rejected manuscripts. Authors tend to suffer from attributional bias: if their paper is rejected, they often blame this on situational factors such as incompetent reviewers and uninterested editors; but if it is accepted they tend to attribute this to their own expertise and competence in writing high-quality papers (Garcia et al. 2016 ).

Authors also value speed of the peer review process. When the duration of the first review round is shorter and there are fewer review rounds, authors give the process a significantly higher rating. Authors who receive more referee reports also tend to be more positive about the process. Their perception might be that their manuscript has been dealt with more seriously and thoroughly. Authors from countries where English is the first language rate the peer review process less positive than authors from other countries. It is possible that these authors have higher expectations of the process and are more critical regarding aspects that do not meet their expectations.

Taking into account other factors, authors in Economics and Business, Social sciences, Psychology, and Mathematics and Computer sciences are more positive about the peer review process than authors in Natural sciences, Medicine, Public health, and especially General journals.

When we include the significant interactions in the model (Model 2), the sign and significance of the main effects stay the same. The interaction analysis shows that the negative effect of a longer duration of the first review round and the negative effect of more review rounds are less profound for accepted papers. Hence it seems that authors are willing to accept extensive revision work if this is rewarded with the acceptance of their paper. At the same time, they seem especially disappointed if the manuscript is still rejected after a long review process.

The negative interaction between a paper being accepted and the number of referee reports indicates that authors of rejected papers may consider a higher number of reports as a sign that their paper was taken seriously and might be content with extensive feedback. For obvious reasons, authors of accepted papers are more positive when the journal has a higher impact factor. Authors from English-speaking countries are less negative about the peer review process when their paper is accepted and when they receive more referee reports but find a long process more problematic. This might reflect that they have higher expectations that their paper will be accepted and that the peer review process will be short and efficient compared to authors from non-English-speaking countries.

When the duration of the first review round is longer, or when the impact factor of the journal is higher, authors are more concerned about a higher number of review rounds. In those cases, they might expect a smooth continuation of the process and be more disappointed when this proves not to be the case. A longer duration of the first review round is considered less negative by authors who receive more referee reports.

Qualitative findings

The motivations authors give for their rating of the peer review process on SciRev.sc contain important qualitative information on author experiences. We analyzed these motivations and registered the author’s major concern(s). A first important observation is that about half (918) of the 1879 comments is positive. Many authors, in particular of accepted papers, are satisfied with the process and express their gratitude in their motivations. Of the 961 comments with a negative connotation, 371 (39%) express concerns about the duration of the review process. This aspect of long review duration is included in the quantitative outcomes and has been discussed in the preceding sections.

A more informative source of discontent, mentioned 437 times (45%), concerns the role of editors and editorial offices. Poor communication of editors/offices—in particular not reacting to information requests—are a major source of frustration mentioned by authors. We received reports of authors who waited over 6 months without hearing anything of the journal or receiving reactions to information requests. Also editors who ‘hide’ behind reviewers and do not take an independent position vis-à-vis them are perceived as problematic. In particular when referee reports are contradictory—as often happens—it is important that editors provide guidance and indicate the comments on which authors should focus in their revision.

Poor quality of referee reports is mentioned in 141 (15%) of the critical comments. Referee reports are often perceived to be superficial, contradictory, unreadable, ask unreasonable modifications, or convey the impression that the reviewer did not read or understand the paper. Some other issues mentioned are the addition of completely new comments in the second review round, the theft of ideas, or asking for unnecessary references.

In this paper we study various aspects of the peer review process on the basis of 3500 review experiences reported in the last 3 years on the SciRev.sc website. Aspects discussed include the first response time (duration of the first review round), total review duration (the time the manuscript is at the editorial office or with reviewers), the immediate rejection time, the time authors take for their first revision (revision time), the number, quality, and difficulty of referee reports received, and the overall rating of the process.

We find considerable variation between the ten scientific fields distinguished. Whereas the reported first response time is 8–9 weeks for Medicine and Public health, it is 11–14 weeks in Natural sciences, Engineering, Psychology, and General journals and 16–18 weeks in Economics and Business, Social sciences, Mathematics and Computer sciences, and Humanities (Table  1 ). There is also considerable variation around these averages. While 27–28% of authors in Medicine and Public health were informed within a month, 18% of authors in Mathematics and Computer sciences and Economics and Business had to wait more than 6 months for a decision. As expected, these figures also translate into longer total review durations reported for the scientific fields with longer first review rounds (Table  2 ).

The long duration of the peer review process is often blamed on reviewers taking much time to complete their reports. However, our figures indicate that inefficient editorial processes are also important. The reported immediate rejection time (Table  3 ), which is not influenced by reviewers, shows substantial variation among the fields and is often unreasonably long. Whereas in half of the immediate rejection cases authors were informed within a week, in about one sixth of these cases authors had to wait for more than 4 weeks. Medicine performs best with an average of 10 days, Natural sciences, Public health and Engineering come second with 11–12 days. Psychology, Social sciences, and Mathematics and Computer sciences take longest with 15–17 days. If editors take much time for a desk rejection, it is likely they also take much time finding reviewers and processing incoming referee reports. Immediate rejection time is therefore a powerful indicator of the overall performance of editorial offices.

The total time between submission of a manuscript and the final decision of the editor is not only influenced by the time reviewers take to submit their reports and the time editorial offices take to handle the manuscript, but also by the time authors take to revise and resubmit their manuscript (Table  5 ). In this respect, the situation is similar to that of the other durations. While, on average, authors take 39 days to revise their manuscript, authors in Psychology and Social sciences take 50 days, and those in Economics and Business even 64 days. On the other hand, authors in Public Health, Engineering, Mathematics and Computer sciences, and Natural sciences take only 29–34 days for a revision. The longer duration in some fields is not associated with a higher number of referee reports (2.0–2.3) nor with more difficult referee reports (2.6–3.3).

Most characteristics of the peer review process studied are related to the journal’s impact factor. More highly ranked journals have a shorter duration of the first review round ( P  = −0.29), total review duration ( P  = −0.27), and immediate rejection time ( P  = −0.18), all indicating that review processes of more highly ranked journals are more efficient. We also found a small but significant positive correlation ( P  = 0.08) between experienced difficulty of the referee reports and impact factor, indicating that reviewers of more highly ranked journals are somewhat more demanding.

As expected, authors of accepted manuscripts are more satisfied with the peer review experience than authors of rejected papers (Table  6 ). On a scale from 0 (very bad) to 5 (excellent), they rate the process a 4, compared to a 2.2 for authors of rejected manuscripts. A longer duration of the process is negatively associated with the rating, independent of the process outcome. For both accepted and rejected manuscripts the Pearson correlation coefficient between total review duration and rating is −0.37.

To assess the independent associations between the characteristics of the process and the satisfaction of authors, a multivariate regression analysis was performed with the overall rating of the process as dependent variable (Table  7 ). This analysis shows that even when the other variables are taken into account, all three aspects, i.e., a shorter duration of the first review round, a lower number of review rounds, and acceptance of the paper, are associated with a significantly higher overall rating of the experience. Interestingly, it also shows that, in spite of the longer duration in Economics and Business, Social sciences, and Mathematics and Computer sciences, authors in those fields are more positive about the process than authors in the General journals, Medicine and Public health, where processes are shorter. Expectations thus clearly play a role.

As expected, authors of accepted papers are even more positive if the journal has a higher impact factor. They are (afterwards) also less bothered by a longer duration of the first review round and by more than one review round. We also find that authors rate the process more positive if they receive more referee reports, in particular after a long first review round and when the manuscript is rejected. This indicates that authors appreciate the work of reviewers and the feedback given on their manuscripts. Compared to authors from non-English-speaking countries, those from English-speaking countries are generally less satisfied with the process, particularly when their manuscript is rejected or in case of more than one review round. This suggests that authors from English-speaking countries have higher expectations of the peer review process.

Alberts, B., Hanson, B., & Kelner, K. L. (2008). Reviewing peer review. Science, 321, 15.

Article   Google Scholar  

Allison, P. (2001). Missing data . London: Sage Publications Ltd.

MATH   Google Scholar  

Azar, O. H. (2004). Rejections and the importance of first response times. International Journal of Social Economics, 31 (3), 259–274.

Azar, O. H. (2007). The slowdown in first-response times of economics Journals: Can it be beneficial? Economic Inquiry, 45 (1), 179–187.

Björk, B., Roos, A., & Lauri, M. (2009). Scientific journal publishing: Yearly volume and open access availability. Information Research, 14, 1.

Google Scholar  

Björk, B., & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetics, 7, 914–923.

Cherkashin, I., Demidova, S., Imai, S., & Krishna, K. (2009). The inside scoop: Acceptance and rejection at the journal of international economics. Journal of International Economics, 77, 120–132.

Ellison, G. (2002a). The slowdown of the economics publishing process. Journal of Political Economy, 110 (5), 947–993.

Ellison, G. (2002b). Evolving standards for academic publishing: A q-r theory. Journal of Political Economy, 110 (5), 994–1034.

Etkin, A. (2014). A new method and metric to evaluate the peer review process of scholarly journals. Pub Res Q, 30, 23–38.

Garcıa, J. A., Rodriguez-Sanchez, Rosa, & Fdez-Valdivia, J. (2016). Why the referees’ reports I receive as an editor are so much better than the reports I receive as an author? Scientometrics, 106, 967–986.

Hamermesh, D. S. (1994). Facts and myths about refereeing. Journal of Economic Perspectives, 8 (1), 153–163.

Hardy, M.A. (1993). Regression with dummy variables. Sage.

Jinha, A. E. (2010). Article 50 million: An estimate of the number of scholarly articles in existence. Learned Publishing, 23, 258–263.

Kareiva, P., Marvier, M., West, S., & Hornisher, J. (2002). Slow-moving journals hinder conservation efforts. Nature, 420, 15.

Lewin, A. Y. (2014). The peer-review process: The good, the bad, the ugly, and the extraordinary. Management and Organization Review, 10 (2), 167–173.

Lotriet, C. J. (2012). Reviewing the review process: Identifying sources of delay. Australasian Medical Journal, 5 (1), 26–29.

Moizer, P. (2009). Publishing in accounting journals: A fair game? Accounting, Organizations and Society, 34, 285–304.

Nicholas, D., Watkinson, A., Jamali, H. R., Herman, E., Tenopir, C., VolentineK, R., et al. (2015). Peer review: Still king in the digital age. Learned Publishing, 28 (1), 15–21.

Onitilo, A. A., Engel, J. M., Salzman-Scott, S. A., Stankowski, R. V., & Suhail, A. R. (2014). A core-item reviewer evaluation (CoRE) system for manuscript peer review. Accountability in Research: Policies and Quality Assurance, 21, 109–121.

Park, I.-U., Peacey, M. W., & Munafo, M. R. (2014). Modelling the effects of subjective and objective decision making in scientific peer review. Nature, 506, 93–98.

Pautasso, M., & Schäfer, H. (2010). Peer review delay and selectivity in ecology journals. Scientometrics, 84, 307–315.

Peters, D., & Ceci, S. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences, 5, 187–255.

Plume, A., & van Weijen, D. (2014). Publish or perish? (p. 38). The rise of the fractional author: Research Trends.

Resnik, D. B., Gutierrez-Ford, Ch., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. science eng Ethics, 14, 305–310.

Solomon, D., & Björk, B. (2012). Publication fees in open access publishing: Sources of funding and factors influencing choice of journal. Journal of the American Society for Information Science and Technology, 63 (1), 98–107.

Thompson, G. D., Aradhyula, S. V., Frisvold, G., & Frisvold, R. (2010). Does paying referees expedite reviews?: Results of a natural experiment. Southern Economic Journal, 76 (3), 678–692.

Tite, L., & Schroter, S. (2007). Why do peer reviewers decline to review? A Survey, Cintinuing Professional Education, 61, 9–12.

Ware, M., & Mabe, M. (2015). The STM report: An overview of scientific and scholarly journal publishing . The Hague: International Association of Scientific, Technical and Medical Publishers. http://www.stm-assoc.org/2012_12_11_STM_Report_2012.pdf .

Download references

Acknowledgement

This article is based upon work from COST Action TD1306 “New Frontiers of Peer Review”, supported by COST (European Cooperation in Science and Technology).

Author information

Authors and affiliations.

Department of Business Administration, Institute for Management Research, Radboud University, Nijmegen, The Netherlands

Janine Huisman

SciRev Foundation, Nijmegen, The Netherlands

Janine Huisman & Jeroen Smits

Department of Economics, Institute for Management Research, Radboud University, Nijmegen, The Netherlands

Jeroen Smits

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jeroen Smits .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Huisman, J., Smits, J. Duration and quality of the peer review process: the author’s perspective. Scientometrics 113 , 633–650 (2017). https://doi.org/10.1007/s11192-017-2310-5

Download citation

Received : 26 July 2016

Published : 09 March 2017

Issue Date : October 2017

DOI : https://doi.org/10.1007/s11192-017-2310-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Peer review process
• Author’s experience
• Find a journal
• Publish with us
• Track your research

Page Content

Overview of the review report format, the first read-through, first read considerations, spotting potential major flaws, concluding the first reading, rejection after the first reading, before starting the second read-through, doing the second read-through, the second read-through: section by section guidance, how to structure your report, on presentation and style, criticisms & confidential comments to editors, the recommendation, when recommending rejection, additional resources, step by step guide to reviewing a manuscript.

When you receive an invitation to peer review, you should be sent a copy of the paper's abstract to help you decide whether you wish to do the review. Try to respond to invitations promptly - it will prevent delays. It is also important at this stage to declare any potential Conflict of Interest.

The structure of the review report varies between journals. Some follow an informal structure, while others have a more formal approach.

" Number your comments!!! " (Jonathon Halbesleben, former Editor of Journal of Occupational and Organizational Psychology)

Informal Structure

Many journals don't provide criteria for reviews beyond asking for your 'analysis of merits'. In this case, you may wish to familiarize yourself with examples of other reviews done for the journal, which the editor should be able to provide or, as you gain experience, rely on your own evolving style.

Formal Structure

Other journals require a more formal approach. Sometimes they will ask you to address specific questions in your review via a questionnaire. Or they might want you to rate the manuscript on various attributes using a scorecard. Often you can't see these until you log in to submit your review. So when you agree to the work, it's worth checking for any journal-specific guidelines and requirements. If there are formal guidelines, let them direct the structure of your review.

In Both Cases

Whether specifically required by the reporting format or not, you should expect to compile comments to authors and possibly confidential ones to editors only.

Following the invitation to review, when you'll have received the article abstract, you should already understand the aims, key data and conclusions of the manuscript. If you don't, make a note now that you need to feedback on how to improve those sections.

The first read-through is a skim-read. It will help you form an initial impression of the paper and get a sense of whether your eventual recommendation will be to accept or reject the paper.

Keep a pen and paper handy when skim-reading.

Try to bear in mind the following questions - they'll help you form your overall impression:

• What is the main question addressed by the research? Is it relevant and interesting?
• How original is the topic? What does it add to the subject area compared with other published material?
• Is the paper well written? Is the text clear and easy to read?
• Are the conclusions consistent with the evidence and arguments presented? Do they address the main question posed?
• If the author is disagreeing significantly with the current academic consensus, do they have a substantial case? If not, what would be required to make their case credible?
• If the paper includes tables or figures, what do they add to the paper? Do they aid understanding or are they superfluous?

While you should read the whole paper, making the right choice of what to read first can save time by flagging major problems early on.

Editors say, " Specific recommendations for remedying flaws are VERY welcome ."

Examples of possibly major flaws include:

• Drawing a conclusion that is contradicted by the author's own statistical or qualitative evidence
• The use of a discredited method
• Ignoring a process that is known to have a strong influence on the area under study

If experimental design features prominently in the paper, first check that the methodology is sound - if not, this is likely to be a major flaw.

You might examine:

• The sampling in analytical papers
• The sufficient use of control experiments
• The precision of process data
• The regularity of sampling in time-dependent studies
• The validity of questions, the use of a detailed methodology and the data analysis being done systematically (in qualitative research)
• That qualitative research extends beyond the author's opinions, with sufficient descriptive elements and appropriate quotes from interviews or focus groups

Major Flaws in Information

If methodology is less of an issue, it's often a good idea to look at the data tables, figures or images first. Especially in science research, it's all about the information gathered. If there are critical flaws in this, it's very likely the manuscript will need to be rejected. Such issues include:

• Insufficient data
• Unclear data tables
• Contradictory data that either are not self-consistent or disagree with the conclusions
• Confirmatory data that adds little, if anything, to current understanding - unless strong arguments for such repetition are made

If you find a major problem, note your reasoning and clear supporting evidence (including citations).

After the initial read and using your notes, including those of any major flaws you found, draft the first two paragraphs of your review - the first summarizing the research question addressed and the second the contribution of the work. If the journal has a prescribed reporting format, this draft will still help you compose your thoughts.

The First Paragraph

This should state the main question addressed by the research and summarize the goals, approaches, and conclusions of the paper. It should:

• Help the editor properly contextualize the research and add weight to your judgement
• Show the author what key messages are conveyed to the reader, so they can be sure they are achieving what they set out to do
• Focus on successful aspects of the paper so the author gets a sense of what they've done well

The Second Paragraph

This should provide a conceptual overview of the contribution of the research. So consider:

• Is the paper's premise interesting and important?
• Are the methods used appropriate?
• Do the data support the conclusions?

After drafting these two paragraphs, you should be in a position to decide whether this manuscript is seriously flawed and should be rejected (see the next section). Or whether it is publishable in principle and merits a detailed, careful read through.

Even if you are coming to the opinion that an article has serious flaws, make sure you read the whole paper. This is very important because you may find some really positive aspects that can be communicated to the author. This could help them with future submissions.

A full read-through will also make sure that any initial concerns are indeed correct and fair. After all, you need the context of the whole paper before deciding to reject. If you still intend to recommend rejection, see the section "When recommending rejection."

Once the paper has passed your first read and you've decided the article is publishable in principle, one purpose of the second, detailed read-through is to help prepare the manuscript for publication. You may still decide to recommend rejection following a second reading.

" Offer clear suggestions for how the authors can address the concerns raised. In other words, if you're going to raise a problem, provide a solution ." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)

Preparation

To save time and simplify the review:

• Don't rely solely upon inserting comments on the manuscript document - make separate notes
• Try to group similar concerns or praise together
• If using a review program to note directly onto the manuscript, still try grouping the concerns and praise in separate notes - it helps later
• Note line numbers of text upon which your notes are based - this helps you find items again and also aids those reading your review

Now that you have completed your preparations, you're ready to spend an hour or so reading carefully through the manuscript.

As you're reading through the manuscript for a second time, you'll need to keep in mind the argument's construction, the clarity of the language and content.

With regard to the argument’s construction, you should identify:

• Any places where the meaning is unclear or ambiguous
• Any factual errors
• Any invalid arguments

You may also wish to consider:

• Does the title properly reflect the subject of the paper?
• Does the abstract provide an accessible summary of the paper?
• Do the keywords accurately reflect the content?
• Is the paper an appropriate length?
• Are the key messages short, accurate and clear?

Not every submission is well written. Part of your role is to make sure that the text’s meaning is clear.

Editors say, " If a manuscript has many English language and editing issues, please do not try and fix it. If it is too bad, note that in your review and it should be up to the authors to have the manuscript edited ."

If the article is difficult to understand, you should have rejected it already. However, if the language is poor but you understand the core message, see if you can suggest improvements to fix the problem:

• Are there certain aspects that could be communicated better, such as parts of the discussion?
• Should the authors consider resubmitting to the same journal after language improvements?
• Would you consider looking at the paper again once these issues are dealt with?

On Grammar and Punctuation

Your primary role is judging the research content. Don't spend time polishing grammar or spelling. Editors will make sure that the text is at a high standard before publication. However, if you spot grammatical errors that affect clarity of meaning, then it's important to highlight these. Expect to suggest such amendments - it's rare for a manuscript to pass review with no corrections.

A 2010 study of nursing journals found that 79% of recommendations by reviewers were influenced by grammar and writing style (Shattel, et al., 2010).

1. The Introduction

A well-written introduction:

• Sets out the argument
• Summarizes recent research related to the topic
• Highlights gaps in current understanding or conflicts in current knowledge
• Establishes the originality of the research aims by demonstrating the need for investigations in the topic area
• Gives a clear idea of the target readership, why the research was carried out and the novelty and topicality of the manuscript

Originality and Topicality

Originality and topicality can only be established in the light of recent authoritative research. For example, it's impossible to argue that there is a conflict in current understanding by referencing articles that are 10 years old.

Authors may make the case that a topic hasn't been investigated in several years and that new research is required. This point is only valid if researchers can point to recent developments in data gathering techniques or to research in indirectly related fields that suggest the topic needs revisiting. Clearly, authors can only do this by referencing recent literature. Obviously, where older research is seminal or where aspects of the methodology rely upon it, then it is perfectly appropriate for authors to cite some older papers.

Editors say, "Is the report providing new information; is it novel or just confirmatory of well-known outcomes ?"

It's common for the introduction to end by stating the research aims. By this point you should already have a good impression of them - if the explicit aims come as a surprise, then the introduction needs improvement.

2. Materials and Methods

Academic research should be replicable, repeatable and robust - and follow best practice.

Replicable Research

This makes sufficient use of:

• Control experiments
• Repeated analyses
• Repeated experiments

These are used to make sure observed trends are not due to chance and that the same experiment could be repeated by other researchers - and result in the same outcome. Statistical analyses will not be sound if methods are not replicable. Where research is not replicable, the paper should be recommended for rejection.

Repeatable Methods

These give enough detail so that other researchers are able to carry out the same research. For example, equipment used or sampling methods should all be described in detail so that others could follow the same steps. Where methods are not detailed enough, it's usual to ask for the methods section to be revised.

Robust Research

This has enough data points to make sure the data are reliable. If there are insufficient data, it might be appropriate to recommend revision. You should also consider whether there is any in-built bias not nullified by the control experiments.

Best Practice

During these checks you should keep in mind best practice:

• Standard guidelines were followed (e.g. the CONSORT Statement for reporting randomized trials)
• The health and safety of all participants in the study was not compromised
• Ethical standards were maintained

If the research fails to reach relevant best practice standards, it's usual to recommend rejection. What's more, you don't then need to read any further.

3. Results and Discussion

This section should tell a coherent story - What happened? What was discovered or confirmed?

Certain patterns of good reporting need to be followed by the author:

• They should start by describing in simple terms what the data show
• They should make reference to statistical analyses, such as significance or goodness of fit
• Once described, they should evaluate the trends observed and explain the significance of the results to wider understanding. This can only be done by referencing published research
• The outcome should be a critical analysis of the data collected

Discussion should always, at some point, gather all the information together into a single whole. Authors should describe and discuss the overall story formed. If there are gaps or inconsistencies in the story, they should address these and suggest ways future research might confirm the findings or take the research forward.

4. Conclusions

This section is usually no more than a few paragraphs and may be presented as part of the results and discussion, or in a separate section. The conclusions should reflect upon the aims - whether they were achieved or not - and, just like the aims, should not be surprising. If the conclusions are not evidence-based, it's appropriate to ask for them to be re-written.

5. Information Gathered: Images, Graphs and Data Tables

If you find yourself looking at a piece of information from which you cannot discern a story, then you should ask for improvements in presentation. This could be an issue with titles, labels, statistical notation or image quality.

Where information is clear, you should check that:

• The results seem plausible, in case there is an error in data gathering
• The trends you can see support the paper's discussion and conclusions
• There are sufficient data. For example, in studies carried out over time are there sufficient data points to support the trends described by the author?

You should also check whether images have been edited or manipulated to emphasize the story they tell. This may be appropriate but only if authors report on how the image has been edited (e.g. by highlighting certain parts of an image). Where you feel that an image has been edited or manipulated without explanation, you should highlight this in a confidential comment to the editor in your report.

6. List of References

You will need to check referencing for accuracy, adequacy and balance.

Where a cited article is central to the author's argument, you should check the accuracy and format of the reference - and bear in mind different subject areas may use citations differently. Otherwise, it's the editor’s role to exhaustively check the reference section for accuracy and format.

You should consider if the referencing is adequate:

• Are important parts of the argument poorly supported?
• Are there published studies that show similar or dissimilar trends that should be discussed?
• If a manuscript only uses half the citations typical in its field, this may be an indicator that referencing should be improved - but don't be guided solely by quantity
• References should be relevant, recent and readily retrievable

Check for a well-balanced list of references that is:

• Helpful to the reader
• Fair to competing authors
• Not over-reliant on self-citation
• Gives due recognition to the initial discoveries and related work that led to the work under assessment

You should be able to evaluate whether the article meets the criteria for balanced referencing without looking up every reference.

7. Plagiarism

By now you will have a deep understanding of the paper's content - and you may have some concerns about plagiarism.

Identified Concern

If you find - or already knew of - a very similar paper, this may be because the author overlooked it in their own literature search. Or it may be because it is very recent or published in a journal slightly outside their usual field.

You may feel you can advise the author how to emphasize the novel aspects of their own study, so as to better differentiate it from similar research. If so, you may ask the author to discuss their aims and results, or modify their conclusions, in light of the similar article. Of course, the research similarities may be so great that they render the work unoriginal and you have no choice but to recommend rejection.

"It's very helpful when a reviewer can point out recent similar publications on the same topic by other groups, or that the authors have already published some data elsewhere ." (Editor feedback)

Suspected Concern

If you suspect plagiarism, including self-plagiarism, but cannot recall or locate exactly what is being plagiarized, notify the editor of your suspicion and ask for guidance.

Most editors have access to software that can check for plagiarism.

Editors are not out to police every paper, but when plagiarism is discovered during peer review it can be properly addressed ahead of publication. If plagiarism is discovered only after publication, the consequences are worse for both authors and readers, because a retraction may be necessary.

For detailed guidelines see COPE's Ethical guidelines for reviewers and Wiley's Best Practice Guidelines on Publishing Ethics .

8. Search Engine Optimization (SEO)

After the detailed read-through, you will be in a position to advise whether the title, abstract and key words are optimized for search purposes. In order to be effective, good SEO terms will reflect the aims of the research.

A clear title and abstract will improve the paper's search engine rankings and will influence whether the user finds and then decides to navigate to the main article. The title should contain the relevant SEO terms early on. This has a major effect on the impact of a paper, since it helps it appear in search results. A poor abstract can then lose the reader's interest and undo the benefit of an effective title - whilst the paper's abstract may appear in search results, the potential reader may go no further.

So ask yourself, while the abstract may have seemed adequate during earlier checks, does it:

• Do justice to the manuscript in this context?
• Highlight important findings sufficiently?
• Present the most interesting data?

Editors say, " Does the Abstract highlight the important findings of the study ?"

If there is a formal report format, remember to follow it. This will often comprise a range of questions followed by comment sections. Try to answer all the questions. They are there because the editor felt that they are important. If you're following an informal report format you could structure your report in three sections: summary, major issues, minor issues.

• Give positive feedback first. Authors are more likely to read your review if you do so. But don't overdo it if you will be recommending rejection
• Briefly summarize what the paper is about and what the findings are
• Try to put the findings of the paper into the context of the existing literature and current knowledge
• Indicate the significance of the work and if it is novel or mainly confirmatory
• Indicate the work's strengths, its quality and completeness
• State any major flaws or weaknesses and note any special considerations. For example, if previously held theories are being overlooked

Major Issues

• Are there any major flaws? State what they are and what the severity of their impact is on the paper
• Has similar work already been published without the authors acknowledging this?
• Are the authors presenting findings that challenge current thinking? Is the evidence they present strong enough to prove their case? Have they cited all the relevant work that would contradict their thinking and addressed it appropriately?
• If major revisions are required, try to indicate clearly what they are
• Are there any major presentational problems? Are figures & tables, language and manuscript structure all clear enough for you to accurately assess the work?
• Are there any ethical issues? If you are unsure it may be better to disclose these in the confidential comments section

Minor Issues

• Are there places where meaning is ambiguous? How can this be corrected?
• Are the correct references cited? If not, which should be cited instead/also? Are citations excessive, limited, or biased?
• Are there any factual, numerical or unit errors? If so, what are they?
• Are all tables and figures appropriate, sufficient, and correctly labelled? If not, say which are not

Your review should ultimately help the author improve their article. So be polite, honest and clear. You should also try to be objective and constructive, not subjective and destructive.

You should also:

• Write clearly and so you can be understood by people whose first language is not English
• Avoid complex or unusual words, especially ones that would even confuse native speakers
• Number your points and refer to page and line numbers in the manuscript when making specific comments
• If you have been asked to only comment on specific parts or aspects of the manuscript, you should indicate clearly which these are
• Treat the author's work the way you would like your own to be treated

Most journals give reviewers the option to provide some confidential comments to editors. Often this is where editors will want reviewers to state their recommendation - see the next section - but otherwise this area is best reserved for communicating malpractice such as suspected plagiarism, fraud, unattributed work, unethical procedures, duplicate publication, bias or other conflicts of interest.

However, this doesn't give reviewers permission to 'backstab' the author. Authors can't see this feedback and are unable to give their side of the story unless the editor asks them to. So in the spirit of fairness, write comments to editors as though authors might read them too.

Reviewers should check the preferences of individual journals as to where they want review decisions to be stated. In particular, bear in mind that some journals will not want the recommendation included in any comments to authors, as this can cause editors difficulty later - see Section 11 for more advice about working with editors.

You will normally be asked to indicate your recommendation (e.g. accept, reject, revise and resubmit, etc.) from a fixed-choice list and then to enter your comments into a separate text box.

Recommending Acceptance

If you're recommending acceptance, give details outlining why, and if there are any areas that could be improved. Don't just give a short, cursory remark such as 'great, accept'. See Improving the Manuscript

Recommending Revision

Where improvements are needed, a recommendation for major or minor revision is typical. You may also choose to state whether you opt in or out of the post-revision review too. If recommending revision, state specific changes you feel need to be made. The author can then reply to each point in turn.

Some journals offer the option to recommend rejection with the possibility of resubmission – this is most relevant where substantial, major revision is necessary.

What can reviewers do to help? " Be clear in their comments to the author (or editor) which points are absolutely critical if the paper is given an opportunity for revisio n." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)

Recommending Rejection

If recommending rejection or major revision, state this clearly in your review (and see the next section, 'When recommending rejection').

Where manuscripts have serious flaws you should not spend any time polishing the review you've drafted or give detailed advice on presentation.

Editors say, " If a reviewer suggests a rejection, but her/his comments are not detailed or helpful, it does not help the editor in making a decision ."

In your recommendations for the author, you should:

• Give constructive feedback describing ways that they could improve the research
• Keep the focus on the research and not the author. This is an extremely important part of your job as a reviewer
• Avoid making critical confidential comments to the editor while being polite and encouraging to the author - the latter may not understand why their manuscript has been rejected. Also, they won't get feedback on how to improve their research and it could trigger an appeal

Remember to give constructive criticism even if recommending rejection. This helps developing researchers improve their work and explains to the editor why you felt the manuscript should not be published.

" When the comments seem really positive, but the recommendation is rejection…it puts the editor in a tough position of having to reject a paper when the comments make it sound like a great paper ." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)

Visit our Wiley Author Learning and Training Channel for expert advice on peer review.

Watch the video, Ethical considerations of Peer Review

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PLoS Comput Biol
• v.9(7); 2013 Jul

Ten Simple Rules for Writing a Literature Review

Marco pautasso.

1 Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France

2 Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications [1] . For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively [2] . Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests [3] . Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read [4] . For such summaries to be useful, however, they need to be compiled in a professional way [5] .

When starting from scratch, reviewing the literature can require a titanic amount of work. That is why researchers who have spent their career working on a certain research issue are in a perfect position to review that literature. Some graduate schools are now offering courses in reviewing the literature, given that most research students start their project by producing an overview of what has already been done on their research issue [6] . However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.

Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills [7] . In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.

Rule 1: Define a Topic and Audience

How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review. On the one hand, if you take several years to choose, several other people may have had the same idea in the meantime. On the other hand, only a well-considered topic is likely to lead to a brilliant literature review [8] . The topic must at least be:

• interesting to you (ideally, you should have come across a series of recent papers related to your line of work that call for a critical summary),
• an important aspect of the field (so that many readers will be interested in the review and there will be enough material to write it), and
• a well-defined issue (otherwise you could potentially include thousands of publications, which would make the review unhelpful).

Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9] , but also from serendipitous moments during desultory reading and discussions. In addition to choosing your topic, you should also select a target audience. In many cases, the topic (e.g., web services in computational biology) will automatically define an audience (e.g., computational biologists), but that same topic may also be of interest to neighbouring fields (e.g., computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

After having chosen your topic and audience, start by checking the literature and downloading relevant papers. Five pieces of advice here:

• keep track of the search items you use (so that your search can be replicated [10] ),
• keep a list of papers whose pdfs you cannot access immediately (so as to retrieve them later with alternative strategies),
• use a paper management system (e.g., Mendeley, Papers, Qiqqa, Sente),
• define early in the process some criteria for exclusion of irrelevant papers (these criteria can then be described in the review to help define its scope), and
• do not just look for research papers in the area you wish to review, but also seek previous reviews.

The chances are high that someone will already have published a literature review ( Figure 1 ), if not exactly on the issue you are planning to tackle, at least on a related topic. If there are already a few or several reviews of the literature on your issue, my advice is not to give up, but to carry on with your own literature review,

The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33] .

• discussing in your review the approaches, limitations, and conclusions of past reviews,
• trying to find a new angle that has not been covered adequately in the previous reviews, and
• incorporating new material that has inevitably accumulated since their appearance.

When searching the literature for pertinent papers and reviews, the usual rules apply:

• be thorough,
• use different keywords and database sources (e.g., DBLP, Google Scholar, ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science), and
• look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

If you read the papers first, and only afterwards start writing the review, you will need a very good memory to remember who wrote what, and what your impressions and associations were while reading each single paper. My advice is, while reading, to start writing down interesting pieces of information, insights about how to organize the review, and thoughts on what to write. This way, by the time you have read the literature you selected, you will already have a rough draft of the review.

Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11] , but you will have avoided the danger posed by staring at a blank document. Be careful when taking notes to use quotation marks if you are provisionally copying verbatim from the literature. It is advisable then to reformulate such quotes with your own words in the final draft. It is important to be careful in noting the references already at this stage, so as to avoid misattributions. Using referencing software from the very beginning of your endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

After having taken notes while reading the literature, you will have a rough idea of the amount of material available for the review. This is probably a good time to decide whether to go for a mini- or a full review. Some journals are now favouring the publication of rather short reviews focusing on the last few years, with a limit on the number of words and citations. A mini-review is not necessarily a minor review: it may well attract more attention from busy readers, although it will inevitably simplify some issues and leave out some relevant material due to space limitations. A full review will have the advantage of more freedom to cover in detail the complexities of a particular scientific development, but may then be left in the pile of the very important papers “to be read” by readers with little time to spare for major monographs.

There is probably a continuum between mini- and full reviews. The same point applies to the dichotomy of descriptive vs. integrative reviews. While descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, integrative reviews attempt to find common ideas and concepts from the reviewed material [12] . A similar distinction exists between narrative and systematic reviews: while narrative reviews are qualitative, systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias [13] , [14] . When systematic reviews analyse quantitative results in a quantitative way, they become meta-analyses. The choice between different review types will have to be made on a case-by-case basis, depending not just on the nature of the material found and the preferences of the target journal(s), but also on the time available to write the review and the number of coauthors [15] .

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16 , 17 . Including material just for the sake of it can easily lead to reviews that are trying to do too many things at once. The need to keep a review focused can be problematic for interdisciplinary reviews, where the aim is to bridge the gap between fields [18] . If you are writing a review on, for example, how epidemiological approaches are used in modelling the spread of ideas, you may be inclined to include material from both parent fields, epidemiology and the study of cultural diffusion. This may be necessary to some extent, but in this case a focused review would only deal in detail with those studies at the interface between epidemiology and the spread of ideas.

While focus is an important feature of a successful review, this requirement has to be balanced with the need to make the review relevant to a broad audience. This square may be circled by discussing the wider implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing the literature is not stamp collecting. A good review does not just summarize the literature, but discusses it critically, identifies methodological problems, and points out research gaps [19] . After having read a review of the literature, a reader should have a rough idea of:

• the major achievements in the reviewed field,
• the main areas of debate, and
• the outstanding research questions.

It is challenging to achieve a successful review on all these fronts. A solution can be to involve a set of complementary coauthors: some people are excellent at mapping what has been achieved, some others are very good at identifying dark clouds on the horizon, and some have instead a knack at predicting where solutions are going to come from. If your journal club has exactly this sort of team, then you should definitely write a review of the literature! In addition to critical thinking, a literature review needs consistency, for example in the choice of passive vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review has a number of telling features: it is worth the reader's time, timely, systematic, well written, focused, and critical. It also needs a good structure. With reviews, the usual subdivision of research papers into introduction, methods, results, and discussion does not work or is rarely used. However, a general introduction of the context and, toward the end, a recapitulation of the main points covered and take-home messages make sense also in the case of reviews. For systematic reviews, there is a trend towards including information about how the literature was searched (database, keywords, time limits) [20] .

How can you organize the flow of the main body of the review so that the reader will be drawn into and guided through it? It is generally helpful to draw a conceptual scheme of the review, e.g., with mind-mapping techniques. Such diagrams can help recognize a logical way to order and link the various sections of a review [21] . This is the case not just at the writing stage, but also for readers if the diagram is included in the review as a figure. A careful selection of diagrams and figures relevant to the reviewed topic can be very helpful to structure the text too [22] .

Rule 8: Make Use of Feedback

Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23] . As a rule, incorporating feedback from reviewers greatly helps improve a review draft. Having read the review with a fresh mind, reviewers may spot inaccuracies, inconsistencies, and ambiguities that had not been noticed by the writers due to rereading the typescript too many times. It is however advisable to reread the draft one more time before submission, as a last-minute correction of typos, leaps, and muddled sentences may enable the reviewers to focus on providing advice on the content rather than the form.

Feedback is vital to writing a good review, and should be sought from a variety of colleagues, so as to obtain a diversity of views on the draft. This may lead in some cases to conflicting views on the merits of the paper, and on how to improve it, but such a situation is better than the absence of feedback. A diversity of feedback perspectives on a literature review can help identify where the consensus view stands in the landscape of the current scientific understanding of an issue [24] .

Rule 9: Include Your Own Relevant Research, but Be Objective

In many cases, reviewers of the literature will have published studies relevant to the review they are writing. This could create a conflict of interest: how can reviewers report objectively on their own work [25] ? Some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. However, bias could also occur in the other direction: some scientists may be unduly dismissive of their own achievements, so that they will tend to downplay their contribution (if any) to a field when reviewing it.

In general, a review of the literature should neither be a public relations brochure nor an exercise in competitive self-denial. If a reviewer is up to the job of producing a well-organized and methodical review, which flows well and provides a service to the readership, then it should be possible to be objective in reviewing one's own relevant findings. In reviews written by multiple authors, this may be achieved by assigning the review of the results of a coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

Given the progressive acceleration in the publication of scientific papers, today's reviews of the literature need awareness not just of the overall direction and achievements of a field of inquiry, but also of the latest studies, so as not to become out-of-date before they have been published. Ideally, a literature review should not identify as a major research gap an issue that has just been addressed in a series of papers in press (the same applies, of course, to older, overlooked studies (“sleeping beauties” [26] )). This implies that literature reviewers would do well to keep an eye on electronic lists of papers in press, given that it can take months before these appear in scientific databases. Some reviews declare that they have scanned the literature up to a certain point in time, but given that peer review can be a rather lengthy process, a full search for newly appeared literature at the revision stage may be worthwhile. Assessing the contribution of papers that have just appeared is particularly challenging, because there is little perspective with which to gauge their significance and impact on further research and society.

Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from all quarters after the review has been published, so that there may soon be the need for an updated review. But this is the nature of science [27] – [32] . I wish everybody good luck with writing a review of the literature.

Acknowledgments

Many thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.

Funding Statement

This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

Username or Email Address

Remember Me

An editor’s guide to the peer review process

Introduction, editorial decision making and the peer review process.

The peer review process is a fundamental part of research publishing. It’s a way of ensuring only articles of the highest quality, which describe sound research methods and results, are published.

The process involves both the journal editors and independent expert reviewers, who evaluate the submitted articles. Peer reviewers can recommend whether or not they believe an article should be accepted or rejected by the journal. However, the ultimate authority to make the final decision rests solely with journal editors or the journal’s editorial board.

Let’s take a look at the key steps in the decision-making process.

Initial screening

After an article is submitted to a journal, a journal editor screens the manuscript and decides whether or not to send it for full peer review. Only after clearing the initial screening is the article sent to two or more independent peer reviewers . Editors will consider the following aspects:

• Is the manuscript good enough quality to be sent for peer review?
• Does it conform to the aims and scope of the journal and has it followed the style guidelines and instructions for authors?
• Does it make a significant contribution to the existing literature?

Unsuitable articles may be rejected without peer review at the editor’s discretion. If the article passes these initial checks, it will be sent for peer review.

Benefits of carrying out this initial screening include:

• A quick decision for authors – if the manuscript clearly lies outside the scope of the journal, then a rapid rejection allows the author to submit their article to another journal more quickly.
• Peer reviewers’ time is not wasted – reviewers don’t have to spend time evaluating and giving feedback for a manuscript of clearly inferior quality.

The peer review process

Once an article has passed the initial screening process, it’s sent for peer review. As an editor, you’ll appreciate the amount of effort that goes into the peer review process. There are many people involved, including:

• You as an editor
• A team of associate editors
• An administrator
• The reviewers
• Editorial board members
• The Taylor & Francis Peer Review and Online Submissions teams
• Your Taylor & Francis Portfolio Managers

There are different types of peer review operated by different journals. But no matter what type of peer review your journal uses, there are plenty of intricate parts to keep everyone busy.

Taking care of the initial checks, assigning the right associate editor, finding enough willing reviewers, assigning reviewers, and checking for ethical issues are all key parts of the process. Not to mention the actual task of writing the review.

Editors need to be careful to select reviewers who have sufficient subject matter expertise to do justice to the article they’re reviewing. You can read more about finding reviewers on the ‘How to find reviewers’ section below.

Given all this, it’s no wonder that peer review takes time.

How long does peer review take?

A question often asked by authors, but also important to editors, is how long does it take between submission and publication of an article. This is a hard question to answer, but often peer review is the lengthiest part of this process.

Journals usually ask reviewers to complete their reviews within 3-4 weeks. However, few journals have a mechanism to enforce the deadline, which is why it can be hard to predict how long the peer review process will take. It’s also worth bearing in mind that highly technical papers or papers from niche subject areas could take longer to review because it often takes editors more time to find appropriate reviewers.

However, there are things you can do as an editor to make peer review more effective and efficient. Focus your efforts on good time management and supplying high-quality reviews. Being aware of the following potential delays can help you limit their effects:

• Difficulty in finding appropriate reviewers
• Delayed response from reviewers
• Unhelpful review reports – reviews that are a single sentence or paragraph are unhelpful to authors or editors. A normal review report should be two to three pages in length, sometimes longer. (Read how to write a review report .)

The final decision

Editors have various options when it comes to making a decision on an article. The following are the most common decisions made:

• Accept without any changes (acceptance) : the journal will publish the article in its original form.
• Accept with minor revisions (acceptance) : the journal will publish the article once the author has made some small corrections.
• Accept after major revisions (conditional acceptance) : the journal will publish the article if the authors make changes suggested by the reviewers and/or editors.
• Revise and resubmit (conditional rejection) : the journal will reconsider the article in another round of decision making once the authors have made major changes.
• Reject the paper (outright rejection) : the journal won’t publish the article or reconsider it.

How to find peer reviewers

Finding peer reviewers is a challenging and time-consuming task. You need to find reviewers with the right expertise to assess a submitted article. And they also need to be willing to write the review to a deadline.

This can limit the reviewer pool immediately, particularly in niche research fields. But on top of that, editors also need to consider conflicts of interest, the diversity of the reviewers, and whether a reviewer has been asked too many times (leading to ‘reviewer fatigue’).

The good news is that there are some simple steps you can take to find reviewers . In brief, these are:

1. Check the references in the article

The reference section of a submitted article is an excellent place to start when looking for peer reviewers.

2. Use search tools and databases to find researchers working on similar topics

There are a number of different search tools and resources you can use to find reviewers. Including:

• Publons’ Reviewer Connect
• Taylor & Francis reviewer locator tools
• Web of Science

3. Use the editorial board

The journal’s editorial board is a great source for reviewers, both as reviewers themselves and via their networks.

4. Consider previous authors and guest editors

Authors of previously published articles and journal guest editors could be just what you’re looking for in a reviewer.

5. Ask reviewers who decline for suggestions

Invited reviewers who decline to review could still help you find alternatives.

6. Use predefined keywords

Predefined keywords are part of the article submission process. When submitting an article, authors select from a predefined list of keywords that describe their expertise areas. This saves editors time on deciding which keywords to use when searching for peer reviewers.

7. Use previous reviewers

Previous reviewers are, of course, obvious people to go to when you need new reviews. But beware of ‘reviewer fatigue’.

8. Use the editor’s personal network

An editor’s personal network is bound to include researchers from relevant subject areas for their journal, who could make excellent reviewers.

9. Consider using early career researchers or junior colleagues

Just bear in mind they might need mentoring or formal training – particularly if it’s their first time carrying out peer review.

It’s also important to keep growing a journal’s reviewer pool. Not only will this help find reviewers faster, it’ll also ensure the journal isn’t always leaning on the same people.

How to retain and reward reviewers

Without reviewers, the peer review process and journal publishing as a whole would fall apart. They provide a vital and important service that ensures the quality and integrity of published research.

While most reviewers see their work as providing service to the academic community, it’s still a voluntary service and one of many demands on their time. Yet peer review can help researchers advance their careers, that is if they have the right evidence to show the expertise they’ve built up. Journal editors can support peer reviewers by both recognizing and validating their work.

Doing so isn’t just beneficial for reviewers either. A case study produced by Publons with the American Society for Microbiology (ASM), showed that researchers are more willing to review and provide useful, constructive feedback if they know their contributions will be formally recognized.

There are a few different ways that Taylor & Francis editors can reward and retain their reviewers:

Provide support and guidance

Our reviewer guidelines provide an important source of support for reviewers about what to expect during the peer review process, how to write review reports, and ethical considerations.

In addition, it’s important for editors to take the time to provide feedback to reviewers and encourage authors to do the same. This is particularly important when supporting early career researchers, as they will undoubtedly be feeling unsure about how well they’ve carried out their reviews. Offering encouragement and tips for improvement will be invaluable to them.

Publish the names of your reviewers as a thank you

Many journals will publish the names of all their reviewers on a regular basis (for example, annually during Peer Review Week). This provides public acknowledgment of the service they’ve delivered.

Offer reviewer rewards or discounts

We have two options for rewarding reviewers in this way:

• Many of our journals now give peer reviewers 30-days’ free access to Taylor & Francis journal content upon agreeing to review. This provides them with resources to assist in the creation of quality reviews.
• We offer a 30% book discount to all our reviewers when purchasing any Taylor & Francis Group books. This includes those under the Routledge and CRC Press imprints.

Present a reviewer certificate or confirmation letter

Our certificate of recognition serves as a formal acknowledgment of a reviewer’s role. Reviewers can request the certificate from their Taylor & Francis contact. It gives them something they can present to employers or their institutions (or simply use to decorate their office). A reviewer confirmation letter is also available on request.

Use Publons

We’ve extended our partnership with Publons and now the majority of our journals are included. Through Publons, researchers can showcase a complete record of their reviewing activity as evidence of their subject-area expertise. They can also earn ‘merit’ points for their contributions.

The ethics of peer review

As there’s been a steady rise in the number of journals using electronic peer review, there has unfortunately also been a rise in ethical concerns about the peer review process. We’ve put together some detailed information on the issues editors need to be aware of.  You can also read a summary of the key points below.

Best practice guidelines for editors

1. be aware of fake reviews and safeguarding peer review integrity.

One of the most high-profile ethical issues in peer review is the increase in cases of ‘fake reviewers’. As a responsible publisher, Taylor & Francis has taken the decision to safeguard the integrity of our peer review by removing the ‘Preferred reviewers’ function from our ScholarOne Manuscripts and Editorial Manager sites. See here for more information on this.  

2. Clarify peer review policies for the journal

• State the types of peer review offered.
• State whether an article has been peer reviewed. For example, it may be the case for a journal that editorials and letters are not peer reviewed, but original articles and reviews always are, by a minimum of two independent reviewers – this needs to be clear.

3. Apply consistent peer review standards

Editors should apply consistent standards in their peer review processes, including for special issues, supplements, or where peer review has been managed by a guest editor. For journals where research content is published, a minimum of two independent expert referee reports should be obtained before an accept decision is made. Reject decisions may be made without seeking advice from independent referees. Content that is not considered research, for example: book reviews, editorials and letters to the editor may also be accepted for publication without independent peer review.

4. Ensure confidentiality

Keep submission and peer review details confidential, as required. Do not upload files, images or information from unpublished manuscripts into databases or tools that do not guarantee confidentiality, are accessible by the public and/or may store or use this information for their own purposes (for example, generative AI tools like ChatGPT).

Editors should ensure confidential handling of article manuscripts. No details should be disclosed to anyone except the peer reviewers without permission from the author.

5. Be aware of reviewer bias and conflicts of interest

Before agreeing to review an article, reviewers must declare any conflicts of interest. This includes any relationship with the author that may potentially bias their review. Editors are also responsible for checking for potential reviewer biases, rather than relying solely on declarations.

Indicators that might suggest a lack of independence include frequent collaborations or working at the same institute. To reduce bias, reviewers should be independent of the authors and each other, and should not be the person who makes the final decision on the manuscript .

6. Manage a reviewers’ database

To ensure journals have a pool of trusted reviewers, it’s important to establish and maintain a secure database of suitably qualified peer reviewers. This needs to be compliant with data protection legislation.

7. Ensure reviewers are aware of guidelines

Our reviewer guidelines provide an important source of support for reviewers about what to expect during the peer review process.

8. Be aware of the need for co-author verification

Editors need to ensure that all co-authors listed for an article have made a genuine contribution to the research.

9. Know where to go for support

There are a variety of support sources to help navigate the ins and outs of peer review ethics.

Read more on our best practice guidelines for peer review to understand why you need at least two independent peer reviewers.

For editors

The following ethical guidelines are also available from the  Committee on Publication Ethics (COPE) :

• Sharing of information among editors-in-chief regarding possible misconduct
• Text recycling guidelines for editors
• A short guide to ethical editing for new editors
• Guidance for editors: research, audit and service evaluations
• Journals’ best practices for ensuring consent for publishing medical case reports: guidance from COPE

For authors – read our guide on Ethics for authors .

For reviewers – visit our Guide to becoming a peer reviewer and the COPE Ethical Guidelines for Peer Reviewers .

Save page as PDF

Save a PDF version of this page.

For the best results we recommend using Google Chrome.

• Getting Published
• Open Research
• Communicating Research
• Life in Research
• For Editors
• For Peer Reviewers
• Research Integrity

Five Steps to a Timely Review

Author: Penny Freedman

Peer review is an important part of the publishing process. Journals would not be able to publish the research that they do without the dedication of peer reviewers. Serving as the checkpoint to academic credibility, reviewers provide crucial feedback for authors and editors that influences the future of a paper’s publication status. With the world facing an unprecedented time of crisis, the work of reviewers is more essential than ever. 

We asked researchers about their experience as peer reviewers. We hope their stories provide a sounding board for others, and offer insight into a process that almost all researchers will take part in during their career. 

Reviewing papers for the past two years, Dr. Berdugo finds that the most challenging part of the process can be setting aside time to review. As the mother of a 3-year old with a 65% appointment, she balances everyday work and life responsibilities alongside periods of time where she is working in remote locations developing field work. To incorporate review responsibilities into her schedule she has shared the efficient system that she uses to complete her reviews in a timely manner.

• First, after glancing at the paper’s abstract to determine if it is a fit, I take a look at my deadline calendar before answering any requests for review
• Once I accept an invitation, I take another glance at the manuscript abstract, and a first glance to the hypotheses, tables, figures, and supporting information (Time allotted: 30+ mins)
•  Some days later, I read the full manuscript, and reserve a couple of additional days to think about the ideas presented in it.
• Usually one week before the review deadline, I read the manuscript again, and take notes section by section (Time allotted: 1+ hour sessions)
• One or two days before the deadline, I write the comments, fill out and submit the review forms

To help facilitate her review process, Dr. Berdugo likes to use a standard paper and pencil along with her laptop. Using her kindle helps her to read over research while on public transport, and she also finds that Acrobat, Word, Google Scholar, Wikipedia, and a thesaurus are helpful tools to utilize when completing a review.

“I think that once the world overpasses the COVID crisis, the global scientific community should reorganize the publication system with deep changes in its structure as the entire society should do. “

Dr. Berdugo hopes that this period of time where our priorities have shifted also helps the scientific community; publishers alike; to re-think, discuss, and implement change to the review process.

Penny Freedman is a Marketing Manager on the Author Experience & Services team based in the New York office. She works closely on sharing insight and guidance on the benefits and services available to our editors, reviewers, and authors.

• Open science
• Tools & Services
• Account Development
• Sales and account contacts
• Professional
• Press office
• Locations & Contact

We are a world leading research, educational and professional publisher. Visit our main website for more information.

• © 2024 Springer Nature
• General terms and conditions
• Your US State Privacy Rights
• Your Privacy Choices / Manage Cookies
• Accessibility
• Legal notice
• Help us to improve this site, send feedback.