Part of being a professional social scientist is to write peer reviews for article in academic journals. This post discusses how to write them.
Basic Principles
Peer review serves to ensure the integrity and quality of scholarly research. It’s a process designed to evaluate the value of a research work. The double blind peer review process is the standard in the social sciences. In these review types, confidentiality is essential, and in all types of peer review, the reviewer must act ethically and civilly.
See also: Writing Peer Review in the Social Sciences: Constructive, Honest, and Respectful
You must disclose to the journal any potential conflicts of interest that could affect your impartiality in the review process.
There is a general structure and content to reviews, but each journal has its own focus, audience, and expectations; thus, you should know the journal’s review criteria and standards. Part of this assessment is whether the article is a fit to the journal’s scope. While that step should be taken by the journal’s editorial team, sometimes they miss it.
The tone of their review should be respectful and professional, even when pointing out flaws. The goal is to foster scientific dialogue, not to discourage authors. Always say something good about the paper, even for papers that have serious flaws. Try to encourage authors and acknowledge their effort. Focus your criticisms on the content and never the author.
Criticize what the paper is, not what you wish the paper would be. For example, suggesting a quantitative study when the paper is qualitative is not appropriate. If the paper’s research question and objectives are out of alignment with the methods, then suggest that the paper revise the question and objectives, e.g. if the research question is about causality, but the methods reveal only correlations, then suggest that the question be rephrased appropriately.
Be specific in your constructive criticism. Feedback should be objective, substantiated, and free from personal bias. Be clear about the weaknesses and provide concrete suggestions for improvement where possible.
Remember that peer reviewing is a skill that improves with practice.
Structure of reviews
If the journal does not provide specific guidelines on the structure of the review, then it is up to you to provide that structure. Here is a generic structure.
Start with a summary
Start with a brief, ca. 100 word summary of the paper. The topic, the research question, the data and methods, and the main findings. If the contribution is clearly stated, then include it in the summary.
Next, discuss the strengths of the paper. Point out what the paper does well, even if it is mainly about the topic and the potential of such an article.
Separate major from minor criticisms
It is important to separate major from minor criticisms. The author (and the editor) will be mainly concerned with major criticisms, and therefore your review should clearly state whether the criticisms you express are major or minor. You can do this with subheadings (“Major Criticisms”) or in text “My major criticisms are as follows…”).
Start with your major criticisms, loosely following the paper’s structure. If possible, start with the issues with how the study contributes to the literature with a focus on the research question, definition of concepts, and theory and hypotheses. Then, discuss major criticisms with the data and methods. End with criticisms of the results and conclusions, if you consider them major. As always, be specific. If you criticize the data, for example, you must be specific about what the data problems are.
Next, write about any minor criticisms. Perhaps there are some issues with grammar, or the wording of the title and abstract, etc.
End with suggestions for improvement
How can the paper be improved overall? This does not have to be a separate section. You may, if you’d like, mix in the suggestions for improvement with the major and minor criticisms.
What to critique?
Remember: criticize the article that you are reading, not the one you wish you were reading. Be fair by being realistic in your criticisms.
The following is a list of questions to ask about the article that you are reading.
Introduction
- What is the research question? Is it clearly stated?
- Is the literature review comprehensive and up-to-date? Does it effectively frame the problem that the study aims to address?
- Does the introduction articulate the study’s contribution to the field? What, specifically, are these contributions? How does it advance or challenge existing knowledge?
- Are the research objectives and hypotheses clearly stated and logically derived from the literature review?
- Are the concepts clearly defined and consistent throughout the paper?
Theory
- Is the theoretical framework clearly defined and relevant to the research question?
- Does the theory section provide a thorough review of the relevant theories? Is the literature review up-to-date, reflecting current knowledge in the field?
- Are key theories and concepts critically analyzed rather than just described? How well does the review engage with different perspectives or debates within the literature? Does the theory section identify gaps or inconsistencies in the literature that the study aims to address?
- Are all key concepts clearly defined and consistently used?
- Does the study contribute to the development or refinement of concepts or theory?
Methods
- Does the article explain how the theory guides the methodology, analysis, and interpretation of results?
- How are concepts operationalized in the study? Are the measures and methods of data collection consistent with the theoretical underpinnings?
- Is the research design appropriate for the research question?
- Are the methods of data collection clearly described and justified?
- Is the sample size, selection process, and unit of observation adequately explained? Are they appropriate for the study’s aims?
- Are the tools, measures, and techniques adequately described and appropriate?
- Is the procedure clearly outlined so that the study could be replicated?
- Are ethical considerations and approvals mentioned and appropriately addressed?
Results
- Are the results presented clearly and systematically?
- Is the data analysis appropriate and adequately supported by the data?
- Are tables, figures, and graphs used effectively to summarize and explain the findings?
- Does the section avoid interpreting or extrapolating the results beyond what the data can support?
Conclusion
- Does the conclusion clearly summarize the key findings and their relevance to the research question and broader field?
- Does this section answer the research question?
- Does the conclusion logically follow from the results?
- Is the theoretical framework effectively used to interpret findings?
- Are theoretical implications discussed in the conclusion? Does the study propose new avenues for theoretical advancement or application?
- Are limitations of the study openly discussed, and do they appear to undermine the main message of the paper?
- Does the discussion offer theoretical or methodological implications of the findings?
Overall Assessment
- Does the article contribute new knowledge or insights to the field?
- Is the argument coherent and well-supported across all sections?
- Are there any critical flaws or gaps in the research?
- Is the writing clear, concise, and well-organized?
- Would the findings be of interest to the readership of the journal?
Other advice on peer review
- How to write a critique
- I am reviewing a very bad paper – do I have to be nice? — I don’t agree with some of the advice, but this is an honest discussion about a common problem
- Step by step guide to reviewing a manuscript
Joshua K. Dubrow is a PhD from The Ohio State University and a Professor of Sociology at the Polish Academy of Sciences.
The Sociology Place 