how to critically appraise research papers

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Perspect Clin Res
• v.12(2); Apr-Jun 2021

Critical appraisal of published research papers – A reinforcing tool for research methodology: Questionnaire-based study

Snehalata gajbhiye.

Department of Pharmacology and Therapeutics, Seth GS Medical College and KEM Hospital, Mumbai, Maharashtra, India

Raakhi Tripathi

Urwashi parmar, nishtha khatri, anirudha potey.

1 Department of Clinical Trials, Serum Institute of India, Pune, Maharashtra, India

Background and Objectives:

Critical appraisal of published research papers is routinely conducted as a journal club (JC) activity in pharmacology departments of various medical colleges across Maharashtra, and it forms an important part of their postgraduate curriculum. The objective of this study was to evaluate the perception of pharmacology postgraduate students and teachers toward use of critical appraisal as a reinforcing tool for research methodology. Evaluation of performance of the in-house pharmacology postgraduate students in the critical appraisal activity constituted secondary objective of the study.

Materials and Methods:

The study was conducted in two parts. In Part I, a cross-sectional questionnaire-based evaluation on perception toward critical appraisal activity was carried out among pharmacology postgraduate students and teachers. In Part II of the study, JC score sheets of 2 nd - and 3 rd -year pharmacology students over the past 4 years were evaluated.

One hundred and twenty-seven postgraduate students and 32 teachers participated in Part I of the study. About 118 (92.9%) students and 28 (87.5%) faculties considered the critical appraisal activity to be beneficial for the students. JC score sheet assessments suggested that there was a statistically significant improvement in overall scores obtained by postgraduate students ( n = 25) in their last JC as compared to the first JC.

Conclusion:

Journal article criticism is a crucial tool to develop a research attitude among postgraduate students. Participation in the JC activity led to the improvement in the skill of critical appraisal of published research articles, but this improvement was not educationally relevant.

INTRODUCTION

Critical appraisal of a research paper is defined as “The process of carefully and systematically examining research to judge its trustworthiness, value and relevance in a particular context.”[ 1 ] Since scientific literature is rapidly expanding with more than 12,000 articles being added to the MEDLINE database per week,[ 2 ] critical appraisal is very important to distinguish scientifically useful and well-written articles from imprecise articles.

Educational authorities like the Medical Council of India (MCI) and Maharashtra University of Health Sciences (MUHS) have stated in pharmacology postgraduate curriculum that students must critically appraise research papers. To impart training toward these skills, MCI and MUHS have emphasized on the introduction of journal club (JC) activity for postgraduate (PG) students, wherein students review a published original research paper and state the merits and demerits of the paper. Abiding by this, pharmacology departments across various medical colleges in Maharashtra organize JC at frequent intervals[ 3 , 4 ] and students discuss varied aspects of the article with teaching faculty of the department.[ 5 ] Moreover, this activity carries a significant weightage of marks in the pharmacology university examination. As postgraduate students attend this activity throughout their 3-year tenure, it was perceived by the authors that this activity of critical appraisal of research papers could emerge as a tool for reinforcing the knowledge of research methodology. Hence, a questionnaire-based study was designed to find out the perceptions from PG students and teachers.

There have been studies that have laid emphasis on the procedure of conducting critical appraisal of research papers and its application into clinical practice.[ 6 , 7 ] However, there are no studies that have evaluated how well students are able to critically appraise a research paper. The Department of Pharmacology and Therapeutics at Seth GS Medical College has developed an evaluation method to score the PG students on this skill and this tool has been implemented for the last 5 years. Since there are no research data available on the performance of PG Pharmacology students in JC, capturing the critical appraisal activity evaluation scores of in-house PG students was chosen as another objective of the study.

MATERIALS AND METHODS

Description of the journal club activity.

JC is conducted in the Department of Pharmacology and Therapeutics at Seth GS Medical College once in every 2 weeks. During the JC activity, postgraduate students critically appraise published original research articles on their completeness and aptness in terms of the following: study title, rationale, objectives, study design, methodology-study population, inclusion/exclusion criteria, duration, intervention and safety/efficacy variables, randomization, blinding, statistical analysis, results, discussion, conclusion, references, and abstract. All postgraduate students attend this activity, while one of them critically appraises the article (who has received the research paper given by one of the faculty members 5 days before the day of JC). Other faculties also attend these sessions and facilitate the discussions. As the student comments on various sections of the paper, the same predecided faculty who gave the article (single assessor) evaluates the student on a total score of 100 which is split per section as follows: Introduction –20 marks, Methodology –20 marks, Discussion – 20 marks, Results and Conclusion –20 marks, References –10 marks, and Title, Abstract, and Keywords – 10 marks. However, there are no standard operating procedures to assess the performance of students at JC.

Methodology

After seeking permission from the Institutional Ethics Committee, the study was conducted in two parts. Part I consisted of a cross-sectional questionnaire-based survey that was conducted from October 2016 to September 2017. A questionnaire to evaluate perception towards the activity of critical appraisal of published papers as research methodology reinforcing tool was developed by the study investigators. The questionnaire consisted of 20 questions: 14 questions [refer Figure 1 ] graded on a 3-point Likert scale (agree, neutral, and disagree), 1 multiple choice selection question, 2 dichotomous questions, 1 semi-open-ended questions, and 2 open-ended questions. Content validation for this questionnaire was carried out with the help of eight pharmacology teachers. The content validity ratio per item was calculated and each item in the questionnaire had a CVR ratio (CVR) of >0.75.[ 8 ] The perception questionnaire was either E-mailed or sent through WhatsApp to PG pharmacology students and teaching faculty in pharmacology departments at various medical colleges across Maharashtra. Informed consent was obtained on E-mail from all the participants.

Graphical representation of the percentage of students/teachers who agreed that critical appraisal of research helped them improve their knowledge on various aspects of research, perceived that faculty participation is important in this activity, and considered critical appraisal activity beneficial for students. The numbers adjacent to the bar diagrams indicate the raw number of students/faculty who agreed, while brackets indicate %

Part II of the study consisted of evaluating the performance of postgraduate students toward skills of critical appraisal of published papers. For this purpose, marks obtained by 2 nd - and 3 rd -year residents during JC sessions conducted over a period of 4 years from October 2013 to September 2017 were recorded and analyzed. No data on personal identifiers of the students were captured.

Statistical analysis

Marks obtained by postgraduate students in their first and last JC were compared using Wilcoxon signed-rank test, while marks obtained by 2 nd - and 3 rd -year postgraduate students were compared using Mann–Whitney test since the data were nonparametric. These statistical analyses were performed using GraphPad Prism statistical software, San Diego, Calfornia, USA, Version 7.0d. Data obtained from the perception questionnaire were entered in Microsoft Excel sheet and were expressed as frequencies (percentages) using descriptive statistics.

Participants who answered all items of the questionnaire were considered as complete responders and only completed questionnaires were analyzed. The questionnaire was sent through an E-mail to 100 students and through WhatsApp to 68 students. Out of the 100 students who received the questionnaire through E-mail, 79 responded completely and 8 were incomplete responders, while 13 students did not revert back. Out of the 68 students who received the questionnaire through WhatsApp, 48 responded completely, 6 gave an incomplete response, and 14 students did not revert back. Hence, of the 168 postgraduate students who received the questionnaire, 127 responded completely (student response rate for analysis = 75.6%). The questionnaire was E-mailed to 33 faculties and was sent through WhatsApp to 25 faculties. Out of the 33 faculties who received the questionnaire through E-mail, 19 responded completely, 5 responded incompletely, and 9 did not respond at all. Out of the 25 faculties who received the questionnaire through WhatsApp, 13 responded completely, 3 were incomplete responders, and 9 did not respond at all. Hence, of a total of 58 faculties who were contacted, 32 responded completely (faculty response rate for analysis = 55%). For Part I of the study, responses on the perception questionnaire from 127 postgraduate students and 32 postgraduate teachers were recorded and analyzed. None of the faculty who participated in the validation of the questionnaire participated in the survey. Number of responses obtained region wise (Mumbai region and rest of Maharashtra region) have been depicted in Table 1 .

Region-wise distribution of responses

Number of responses obtained from students/faculty belonging to Mumbai colleges and rest of Maharashtra colleges. Brackets indicate percentages

As per the data obtained on the Likert scale questions, 102 (80.3%) students and 29 (90.6%) teachers agreed that critical appraisal trains the students in doing a review of literature before selecting a particular research topic. Majority of the participants, i.e., 104 (81.9%) students and 29 (90.6%) teachers also believed that the activity increases student's knowledge regarding various experimental evaluation techniques. Moreover, 112 (88.2%) students and 27 (84.4%) faculty considered that critical appraisal activity results in improved skills of writing and understanding methodology section of research articles in terms of inclusion/exclusion criteria, endpoints, and safety/efficacy variables. About 103 (81.1%) students and 24 (75%) teachers perceived that this activity results in refinement of the student's research work. About 118 (92.9%) students and 28 (87.5%) faculty considered the critical appraisal activity to be beneficial for the students. Responses to 14 individual Likert scale items of the questionnaire have been depicted in Figure 1 .

With respect to the multiple choice selection question, 66 (52%) students and 16 (50%) teachers opined that faculty should select the paper, 53 (41.7%) students and 9 (28.1%) teachers stated that the papers should be selected by the presenting student himself/herself, while 8 (6.3%) students and 7 (21.9%) teachers expressed that some other student should select the paper to be presented at the JC.

The responses to dichotomous questions were as follows: majority of the students, that is, 109 (85.8%) and 23 (71.9%) teachers perceived that a standard checklist for article review should be given to the students before critical appraisal of journal article. Open-ended questions of the questionnaire invited suggestions from the participants regarding ways of getting trained on critical appraisal skills and of improving JC activity. Some of the suggestions given by faculty were as follows: increasing the frequency of JC activity, discussion of cited articles and new guidelines related to it, selecting all types of articles for criticism rather than only randomized controlled trials, and regular yearly exams on article criticism. Students stated that regular and frequent article criticism activity, practice of writing letter to the editor after criticism, active participation by peers and faculty, increasing weightage of marks for critical appraisal of papers in university examinations (at present marks are 50 out of 400), and a formal training for research criticism from 1 st year of postgraduation could improve critical appraisal program.

In Part II of this study, performance of the students on the skill of critical appraisal of papers was evaluated. Complete data of the first and last JC scores of a total of 25 students of the department were available, and when these scores were compared, it was seen that there was a statistically significant improvement in the overall scores ( P = 0.04), as well as in the scores obtained in methodology ( P = 0.03) and results section ( P = 0.02). This is depicted in Table 2 . Although statistically significant, the differences in scores in the methodology section, results section, and overall scores were 1.28/20, 1.28/20, and 4.36/100, respectively, amounting to 5.4%, 5.4%, and 4.36% higher scores in the last JC, which may not be considered educationally relevant (practically significant). The quantum of difference that would be considered practically significant was not decided a priori .

Comparison of marks obtained by pharmacology residents in their first and last journal club

Marks have been represented as mean±SD. The maximum marks that can be obtained in each section have been stated as maximum. *Indicates statistically significant ( P <0.05). IQR=Interquartile range, SD=Standard deviation

Scores of two groups, one group consisting of 2 nd -year postgraduate students ( n = 44) and second group consisting of 3 rd -year postgraduate students ( n = 32) were compared and revealed no statistically significant difference in overall score ( P = 0.84). This is depicted in Table 3 . Since the quantum of difference in the overall scores was meager 0.84/100 (0.84%), it cannot be considered practically significant.

Comparison of marks obtained by 2 nd - and 3 rd -year pharmacology residents in the activity of critical appraisal of research articles

Marks have been represented as mean±SD. The maximum marks that can be obtained in each section have been stated as maximum. P <0.05 was considered to be statistically significant. IQR=Interquartile range, SD=Standard deviation

The present study gauged the perception of the pharmacology postgraduate students and teachers toward the use of critical appraisal activity as a reinforcing tool for research methodology. Both students and faculties (>50%) believed that critical appraisal activity increases student's knowledge on principles of ethics, experimental evaluation techniques, CONSORT guidelines, statistical analysis, concept of conflict of interest, current trends and recent advances in Pharmacology and trains on doing a review of literature, and improves skills on protocol writing and referencing. In the study conducted by Crank-Patton et al ., a survey on 278 general surgery program directors was carried out and more than 50% indicated that JC was important to their training program.[ 9 ]

The grading template used in Part II of the study was based on the IMRaD structure. Hence, equal weightage was given to the Introduction, Methodology, Results, and Discussion sections and lesser weightage was given to the references and title, abstract, and keywords sections.[ 10 ] While evaluating the scores obtained by 25 students in their first and last JC, it was seen that there was a statistically significant improvement in the overall scores of the students in their last JC. However, the meager improvement in scores cannot be considered educationally relevant, as the authors expected the students to score >90% for the upgrade to be considered educationally impactful. The above findings suggest that even though participation in the JC activity led to a steady increase in student's performance (~4%), the increment was not as expected. In addition, the students did not portray an excellent performance (>90%), with average scores being around 72% even in the last JC. This can be probably explained by the fact that students perform this activity in a routine setting and not in an examination setting. Unlike the scenario in an examination, students were aware that even if they performed at a mediocre level, there would be no repercussions.

A separate comparison of scores obtained by 44 students in their 2 nd year and 32 students in their 3 rd year of postgraduation students was also done. The number of student evaluation sheets reviewed for this analysis was greater than the number of student evaluation sheets reviewed to compare first and last JC scores. This can be spelled out by the fact that many students were still in 2 nd year when this analysis was done and the score data for their last JC, which would take place in 3 rd year, was not available. In addition, few students were asked to present at JC multiple times during the 2 nd /3 rd year of their postgraduation.

While evaluating the critical appraisal scores obtained by 2 nd - and 3 rd -year postgraduate students, it was found that although the 3 rd -year students had a mean overall score greater than the 2 nd -year students, this difference was not statistically significant. During the 1 st year of MD Pharmacology course, students at the study center attend JC once in every 2 weeks. Even though the 1 st -year students do not themselves present in JC, they listen and observe the criticism points stated by senior peers presenting at the JC, and thereby, incur substantial amount of knowledge required to critically appraise papers. By the time, they become 2 nd -year students, they are already well versed with the program and this could have led to similar overall mean scores between the 2 nd -year students (71.50 ± 10.71) and 3 rd -year students (72.34 ± 10.85). This finding suggests that attentive listening is as important as active participation in the JC. Moreover, although students are well acquainted with the process of criticism when they are in their 3 rd year, there is certainly a scope for improvement in terms of the mean overall scores.

Similar results were obtained in a study conducted by Stern et al ., in which 62 students in the internal medicine program at the New England Medical Center were asked to respond to a questionnaire, evaluate a sample article, and complete a self-assessment of competence in evaluation of research. Twenty-eight residents returned the questionnaire and the composite score for the resident's objective assessment was not significantly correlated with the postgraduate year or self-assessed critical appraisal skill.[ 11 ]

Article criticism activity provides the students with practical experience of techniques taught in research methodology workshop. However, this should be supplemented with activities that assess the improvement of designing and presenting studies, such as protocol and paper writing. Thus, critical appraisal plays a significant role in reinforcing good research practices among the new generation of physicians. Moreover, critical appraisal is an integral part of PG assessment, and although the current format of conducting JCs did not portray a clinically meaningful improvement, the authors believe that it is important to continue this activity with certain modifications suggested by students who participated in this study. Students suggested that an increase in the frequency of critical appraisal activity accompanied by the display of active participation by peers and faculty could help in the betterment of this activity. This should be brought to attention of the faculty, as students seem to be interested to learn. Critical appraisal should be a two-way teaching–learning process between the students and faculty and not a dire need for satisfying the students' eligibility criteria for postgraduate university examinations. This activity is not only for the trainee doctors but also a part of the overall faculty development program.[ 12 ]

In the present era, JCs have been used as a tool to not only teach critical appraisal skills but also to teach other necessary aspects such as research design, medical statistics, clinical epidemiology, and clinical decision-making.[ 13 , 14 ] A study conducted by Khan in 2013 suggested that success of JC program can be ensured if institutes develop a defined JC objective for the development of learning capability of students and also if they cultivate more skilled faculties.[ 15 ] A good JC is believed to facilitate relevant, meaningful scientific discussion, and evaluation of the research updates that will eventually benefit the patient care.[ 12 ]

Although there is a lot of literature emphasizing the importance of JC, there is a lack of studies that have evaluated the outcome of such activity. One such study conducted by Ibrahim et al . assessed the importance of critical appraisal as an activity in surgical trainees in Nigeria. They reported that 92.42% trainees considered the activity to be important or very important and 48% trainees stated that the activity helped in improving literature search.[ 16 ]

This study is unique since it is the first of its kind to evaluate how well students are able to critically appraise a research paper. Moreover, the study has taken into consideration the due opinions of the students as well as faculties, unlike the previous literature which has laid emphasis on only student's perception. A limitation of this study is that sample size for faculties was smaller than the students, as it was not possible to convince the distant faculty in other cities to fill the survey. Besides, there may be a variation in the manner of conduct of the critical appraisal activity in pharmacology departments across the various medical colleges in the country. Another limitation of this study was that a single assessor graded a single student during one particular JC. Nevertheless, each student presented at multiple JC and thereby came across multiple assessors. Since the articles addressed at different JC were disparate, interobserver variability was not taken into account in this study. Furthermore, the authors did not make an a priori decision on the quantum of increase in scores that would be considered educationally meaningful.

Pharmacology students and teachers acknowledge the role of critical appraisal in improving the ability to understand the crucial concepts of research methodology and research conduct. In our institute, participation in the JC activity led to an improvement in the skill of critical appraisal of published research articles among the pharmacology postgraduate students. However, this improvement was not educationally relevant. The scores obtained by final-year postgraduate students in this activity were nearly 72% indicating that there is still scope of betterment in this skill.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

Acknowledgments

We would like to acknowledge the support rendered by the entire Department of Pharmacology and Therapeutics at Seth GS Medical College.

• En español – ExME
• Em português – EME

Critical Appraisal: A Checklist

Posted on 6th September 2016 by Robert Will

Critical appraisal of scientific literature is a necessary skill for healthcare students. Students can be overwhelmed by the vastness of search results. Database searching is a skill in itself, but will not be covered in this blog. This blog assumes that you have found a relevant journal article to answer a clinical question. After selecting an article, you must be able to sit with the article and critically appraise it. Critical appraisal of a journal article is a literary and scientific systematic dissection in an attempt to assign merit to the conclusions of an article. Ideally, an article will be able to undergo scrutiny and retain its findings as valid.

The specific questions used to assess validity change slightly with different study designs and article types. However, in an attempt to provide a generalized checklist, no specific subtype of article has been chosen. Rather, the 20 questions below should be used as a quick reference to appraise any journal article. The first four checklist questions should be answered “Yes.” If any of the four questions are answered “no,” then you should return to your search and attempt to find an article that will meet these criteria.

Critical appraisal of…the Introduction

• Does the article attempt to answer the same question as your clinical question?
• Is the article recently published (within 5 years) or is it seminal (i.e. an earlier article but which has strongly influenced later developments)?
• Is the journal peer-reviewed?
• Do the authors present a hypothesis?

Critical appraisal of…the Methods

• Is the study design valid for your question?
• Are both inclusion and exclusion criteria described?
• Is there an attempt to limit bias in the selection of participant groups?
• Are there methodological protocols (i.e. blinding) used to limit other possible bias?
• Do the research methods limit the influence of confounding variables?
• Are the outcome measures valid for the health condition you are researching?

Critical appraisal of…the Results

• Is there a table that describes the subjects’ demographics?
• Are the baseline demographics between groups similar?
• Are the subjects generalizable to your patient?
• Are the statistical tests appropriate for the study design and clinical question?
• Are the results presented within the paper?
• Are the results statistically significant and how large is the difference between groups?
• Is there evidence of significance fishing (i.e. changing statistical tests to ensure significance)?

Critical appraisal of…the Discussion/Conclusion

• Do the authors attempt to contextualise non-significant data in an attempt to portray significance? (e.g. talking about findings which had a  trend  towards significance as if they were significant).
• Do the authors acknowledge limitations in the article?
• Are there any conflicts of interests noted?

This is by no means a comprehensive checklist of how to critically appraise a scientific journal article. However, by answering the previous 20 questions based on a detailed reading of an article, you can appraise most articles for their merit, and thus determine whether the results are valid. I have attempted to list the questions based on the sections most commonly present in a journal article, starting at the introduction and progressing to the conclusion. I believe some of these items are weighted heavier than others (i.e. methodological questions vs journal reputation). However, without taking this list through rigorous testing, I cannot assign a weight to them. Maybe one day, you will be able to critically appraise my future paper:  How Online Checklists Influence Healthcare Students’ Ability to Critically Appraise Journal Articles.

Feature Image by Arek Socha from Pixabay

Robert Will

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

No Comments on Critical Appraisal: A Checklist

Hi Ella, I have found a checklist here for before and after study design: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools and you may also find a checklist from this blog, which has a huge number of tools listed: https://s4be.cochrane.org/blog/2018/01/12/appraising-the-appraisal/

What kind of critical appraisal tool can be used for before and after study design article? Thanks

Hello, I am currently writing a book chapter on critical appraisal skills. This chapter is limited to 1000 words so your simple 20 questions framework would be the perfect format to cite within this text. May I please have your permission to use your checklist with full acknowledgement given to you as author? Many thanks

Thank you Robert, I came across your checklist via the Royal College of Surgeons of England website; https://www.rcseng.ac.uk/library-and-publications/library/blog/dissecting-the-literature-the-importance-of-critical-appraisal/ . I really liked it and I have made reference to it for our students. I really appreciate your checklist and it is still current, thank you.

Hi Kirsten. Thank you so much for letting us know that Robert’s checklist has been used in that article – that’s so good to see. If any of your students have any comments about the blog, then do let us know. If you also note any topics that you would like to see on the website, then we can add this to the list of suggested blogs for students to write about. Thank you again. Emma.

i am really happy with it. thank you very much

A really useful guide for helping you ask questions about the studies you are reviewing BRAVO

Dr.Suryanujella,

Thank you for the comment. I’m glad you find it helpful.

Feel free to use the checklist. S4BE asks that you cite the page when you use it.

I have read your article and found it very useful , crisp with all relevant information.I would like to use it in my presentation with your permission

That’s great thank you very much. I will definitely give that a go.

I find the MEAL writing approach very versatile. You can use it to plan the entire paper and each paragraph within the paper. There are a lot of helpful MEAL resources online. But understanding the acronym can get you started.

M-Main Idea (What are you arguing?) E-Evidence (What does the literature say?) A-Analysis (Why does the literature matter to your argument?) L-Link (Transition to next paragraph or section)

I hope that is somewhat helpful. -Robert

Hi, I am a university student at Portsmouth University, UK. I understand the premise of a critical appraisal however I am unsure how to structure an essay critically appraising a paper. Do you have any pointers to help me get started?

Thank you. I’m glad that you find this helpful.

Very informative & to the point for all medical students

How can I know what is the name of this checklist or tool?

This is a checklist that the author, Robert Will, has designed himself.

Thank you for asking. I am glad you found it helpful. As Emma said, please cite the source when you use it.

Greetings Robert, I am a postgraduate student at QMUL in the UK and I have just read this comprehensive critical appraisal checklist of your. I really appreciate you. if I may ask, can I have it downloaded?

Please feel free to use the information from this blog – if you could please cite the source then that would be much appreciated.

Robert Thank you for your comptrehensive account of critical appraisal. I have just completed a teaching module on critical appraisal as part of a four module Evidence Based Medicine programme for undergraduate Meducal students at RCSI Perdana medical school in Malaysia. If you are agreeable I would like to cite it as a reference in our module.

Anthony, Please feel free to cite my checklist. Thank you for asking. I hope that your students find it helpful. They should also browse around S4BE. There are numerous other helpful articles on this site.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

Risk Communication in Public Health

Learn why effective risk communication in public health matters and where you can get started in learning how to better communicate research evidence.

Why was the CONSORT Statement introduced?

The CONSORT statement aims at comprehensive and complete reporting of randomized controlled trials. This blog introduces you to the statement and why it is an important tool in the research world.

Measures of central tendency in clinical research papers: what we should know whilst analysing them

Learn more about the measures of central tendency (mean, mode, median) and how these need to be critically appraised when reading a paper.

• CASP Checklists
• How to use our CASP Checklists
• Referencing and Creative Commons
• Online Training Courses
• CASP Workshops
• What is Critical Appraisal
• Study Designs
• Useful Links
• Bibliography
• View all Tools and Resources
• Testimonials

How to Critically Appraise a Research Paper

Research papers are a powerful means through which millions of researchers around the globe pass on knowledge about our world.

However, the quality of research can be highly variable. To avoid being misled, it is vital to perform critical appraisals of research studies to assess the validity, results and relevance of the published research. Critical appraisal skills are essential to be able to identify whether published research provides results that can be used as evidence to help improve your practice.

What is a critical appraisal?

Most of us know not to believe everything we read in the newspaper or on various media channels. But when it comes to research literature and journals, they must be critically appraised due to the nature of the context. In order for us to trust research papers, we want to be safe in the knowledge that they have been efficiently and professionally checked to confirm what they are saying. This is where a critical appraisal comes in.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, value and relevance in a particular context. We have put together a more detailed page to explain what critical appraisal is to give you more information.

Why is a critical appraisal of research required?

Critical appraisal skills are important as they enable you to systematically and objectively assess the trustworthiness, relevance and results of published papers. When a research article is published, who wrote it should not be an indication of its trustworthiness and relevance.

What are the benefits of performing critical appraisals for research papers?

Performing a critical appraisal helps to:

• Reduce information overload by eliminating irrelevant or weak studies
• Identify the most relevant papers
• Distinguish evidence from opinion, assumptions, misreporting, and belief
• Assess the validity of the study
• Check the usefulness and clinical applicability of the study

How to critically appraise a research paper

There are some key questions to consider when critically appraising a paper. These include:

• Is the study relevant to my field of practice?
• What research question is being asked?
• Was the study design appropriate for the research question?

CASP has several checklists to help with performing a critical appraisal which we believe are crucial because:

• They help the user to undertake a complex task involving many steps
• They support the user in being systematic by ensuring that all important factors or considerations are taken into account
• They increase consistency in decision-making by providing a framework

In addition to our free checklists, CASP has developed a number of valuable online e-learning modules designed to increase your knowledge and confidence in conducting a critical appraisal.

Introduction To Critical Appraisal & CASP

This Module covers the following:

• Challenges using evidence to change practice
• 5 steps of evidence-based practice
• Developing critical appraisal skills
• Integrating and acting on the evidence
• The Critical Appraisal Skills Programme (CASP)
• Online Learning
• What Is a Subgroup Analysis?
• What Is Evidence-Based Practice?
• What Is A Cross-Sectional Study?
• What Is A PICO Tool?
• What Is A Pilot Study?
• Different Types of Bias in Research
• What is Qualitative Research?
• What is a Case-Control Study in Research?
• What Are Systematic Reviews? Why Are They Important?
• How to Critically Appraise a Medical Research Paper
• Privacy Policy

Critical Appraisal Skills Programme

Critical Appraisal Skills Programme (CASP) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Copyright 2024 CASP UK - OAP Ltd. All rights reserved Website by Beyond Your Brand

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 20 January 2009

How to critically appraise an article

• Jane M Young 1 &
• Michael J Solomon 2  

Nature Clinical Practice Gastroenterology & Hepatology volume  6 ,  pages 82–91 ( 2009 ) Cite this article

52k Accesses

100 Citations

431 Altmetric

Metrics details

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article in order to assess the usefulness and validity of research findings. The most important components of a critical appraisal are an evaluation of the appropriateness of the study design for the research question and a careful assessment of the key methodological features of this design. Other factors that also should be considered include the suitability of the statistical methods used and their subsequent interpretation, potential conflicts of interest and the relevance of the research to one's own practice. This Review presents a 10-step guide to critical appraisal that aims to assist clinicians to identify the most relevant high-quality studies available to guide their clinical practice.

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article

Critical appraisal provides a basis for decisions on whether to use the results of a study in clinical practice

Different study designs are prone to various sources of systematic bias

Design-specific, critical-appraisal checklists are useful tools to help assess study quality

Assessments of other factors, including the importance of the research question, the appropriateness of statistical analysis, the legitimacy of conclusions and potential conflicts of interest are an important part of the critical appraisal process

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 12 print issues and online access

195,33 € per year

only 16,28 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Making sense of the literature: an introduction to critical appraisal for the primary care practitioner

How to appraise the literature: basic principles for the busy clinician - part 2: systematic reviews and meta-analyses

How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials

Druss BG and Marcus SC (2005) Growth and decentralisation of the medical literature: implications for evidence-based medicine. J Med Libr Assoc 93 : 499–501

PubMed   PubMed Central   Google Scholar  

Glasziou PP (2008) Information overload: what's behind it, what's beyond it? Med J Aust 189 : 84–85

PubMed   Google Scholar  

Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University Press

Google Scholar  

Sackett DL et al . (2000). Evidence-based Medicine. How to Practice and Teach EBM . London: Churchill Livingstone

Guyatt G and Rennie D (Eds; 2002). Users' Guides to the Medical Literature: a Manual for Evidence-based Clinical Practice . Chicago: American Medical Association

Greenhalgh T (2000) How to Read a Paper: the Basics of Evidence-based Medicine . London: Blackwell Medicine Books

MacAuley D (1994) READER: an acronym to aid critical reading by general practitioners. Br J Gen Pract 44 : 83–85

CAS   PubMed   PubMed Central   Google Scholar  

Hill A and Spittlehouse C (2001) What is critical appraisal. Evidence-based Medicine 3 : 1–8 [ http://www.evidence-based-medicine.co.uk ] (accessed 25 November 2008)

Public Health Resource Unit (2008) Critical Appraisal Skills Programme (CASP) . [ http://www.phru.nhs.uk/Pages/PHD/CASP.htm ] (accessed 8 August 2008)

National Health and Medical Research Council (2000) How to Review the Evidence: Systematic Identification and Review of the Scientific Literature . Canberra: NHMRC

Elwood JM (1998) Critical Appraisal of Epidemiological Studies and Clinical Trials (2nd Edn). Oxford: Oxford University Press

Agency for Healthcare Research and Quality (2002) Systems to rate the strength of scientific evidence? Evidence Report/Technology Assessment No 47, Publication No 02-E019 Rockville: Agency for Healthcare Research and Quality

Crombie IK (1996) The Pocket Guide to Critical Appraisal: a Handbook for Health Care Professionals . London: Blackwell Medicine Publishing Group

Heller RF et al . (2008) Critical appraisal for public health: a new checklist. Public Health 122 : 92–98

Article   Google Scholar  

MacAuley D et al . (1998) Randomised controlled trial of the READER method of critical appraisal in general practice. BMJ 316 : 1134–37

Article   CAS   Google Scholar  

Parkes J et al . Teaching critical appraisal skills in health care settings (Review). Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: cd001270. 10.1002/14651858.cd001270

Mays N and Pope C (2000) Assessing quality in qualitative research. BMJ 320 : 50–52

Hawking SW (2003) On the Shoulders of Giants: the Great Works of Physics and Astronomy . Philadelphia, PN: Penguin

National Health and Medical Research Council (1999) A Guide to the Development, Implementation and Evaluation of Clinical Practice Guidelines . Canberra: National Health and Medical Research Council

US Preventive Services Taskforce (1996) Guide to clinical preventive services (2nd Edn). Baltimore, MD: Williams & Wilkins

Solomon MJ and McLeod RS (1995) Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 118 : 456–467

Rothman KJ (2002) Epidemiology: an Introduction . Oxford: Oxford University Press

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: sources of bias in surgical studies. ANZ J Surg 73 : 504–506

Margitic SE et al . (1995) Lessons learned from a prospective meta-analysis. J Am Geriatr Soc 43 : 435–439

Shea B et al . (2001) Assessing the quality of reports of systematic reviews: the QUORUM statement compared to other tools. In Systematic Reviews in Health Care: Meta-analysis in Context 2nd Edition, 122–139 (Eds Egger M. et al .) London: BMJ Books

Chapter   Google Scholar  

Easterbrook PH et al . (1991) Publication bias in clinical research. Lancet 337 : 867–872

Begg CB and Berlin JA (1989) Publication bias and dissemination of clinical research. J Natl Cancer Inst 81 : 107–115

Moher D et al . (2000) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement. Br J Surg 87 : 1448–1454

Shea BJ et al . (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7 : 10 [10.1186/1471-2288-7-10]

Stroup DF et al . (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283 : 2008–2012

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: evaluating surgical effectiveness. ANZ J Surg 73 : 507–510

Schulz KF (1995) Subverting randomization in controlled trials. JAMA 274 : 1456–1458

Schulz KF et al . (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273 : 408–412

Moher D et al . (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Medical Research Methodology 1 : 2 [ http://www.biomedcentral.com/ 1471-2288/1/2 ] (accessed 25 November 2008)

Rochon PA et al . (2005) Reader's guide to critical appraisal of cohort studies: 1. Role and design. BMJ 330 : 895–897

Mamdani M et al . (2005) Reader's guide to critical appraisal of cohort studies: 2. Assessing potential for confounding. BMJ 330 : 960–962

Normand S et al . (2005) Reader's guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding. BMJ 330 : 1021–1023

von Elm E et al . (2007) Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 335 : 806–808

Sutton-Tyrrell K (1991) Assessing bias in case-control studies: proper selection of cases and controls. Stroke 22 : 938–942

Knottnerus J (2003) Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol 56 : 1118–1128

Furukawa TA and Guyatt GH (2006) Sources of bias in diagnostic accuracy studies and the diagnostic process. CMAJ 174 : 481–482

Bossyut PM et al . (2003)The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138 : W1–W12

STARD statement (Standards for the Reporting of Diagnostic Accuracy Studies). [ http://www.stard-statement.org/ ] (accessed 10 September 2008)

Raftery J (1998) Economic evaluation: an introduction. BMJ 316 : 1013–1014

Palmer S et al . (1999) Economics notes: types of economic evaluation. BMJ 318 : 1349

Russ S et al . (1999) Barriers to participation in randomized controlled trials: a systematic review. J Clin Epidemiol 52 : 1143–1156

Tinmouth JM et al . (2004) Are claims of equivalency in digestive diseases trials supported by the evidence? Gastroentrology 126 : 1700–1710

Kaul S and Diamond GA (2006) Good enough: a primer on the analysis and interpretation of noninferiority trials. Ann Intern Med 145 : 62–69

Piaggio G et al . (2006) Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA 295 : 1152–1160

Heritier SR et al . (2007) Inclusion of patients in clinical trial analysis: the intention to treat principle. In Interpreting and Reporting Clinical Trials: a Guide to the CONSORT Statement and the Principles of Randomized Controlled Trials , 92–98 (Eds Keech A. et al .) Strawberry Hills, NSW: Australian Medical Publishing Company

National Health and Medical Research Council (2007) National Statement on Ethical Conduct in Human Research 89–90 Canberra: NHMRC

Lo B et al . (2000) Conflict-of-interest policies for investigators in clinical trials. N Engl J Med 343 : 1616–1620

Kim SYH et al . (2004) Potential research participants' views regarding researcher and institutional financial conflicts of interests. J Med Ethics 30 : 73–79

Komesaroff PA and Kerridge IH (2002) Ethical issues concerning the relationships between medical practitioners and the pharmaceutical industry. Med J Aust 176 : 118–121

Little M (1999) Research, ethics and conflicts of interest. J Med Ethics 25 : 259–262

Lemmens T and Singer PA (1998) Bioethics for clinicians: 17. Conflict of interest in research, education and patient care. CMAJ 159 : 960–965

Download references

Author information

Authors and affiliations.

JM Young is an Associate Professor of Public Health and the Executive Director of the Surgical Outcomes Research Centre at the University of Sydney and Sydney South-West Area Health Service, Sydney,

Jane M Young

MJ Solomon is Head of the Surgical Outcomes Research Centre and Director of Colorectal Research at the University of Sydney and Sydney South-West Area Health Service, Sydney, Australia.,

Michael J Solomon

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jane M Young .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Young, J., Solomon, M. How to critically appraise an article. Nat Rev Gastroenterol Hepatol 6 , 82–91 (2009). https://doi.org/10.1038/ncpgasthep1331

Download citation

Received : 10 August 2008

Accepted : 03 November 2008

Published : 20 January 2009

Issue Date : February 2009

DOI : https://doi.org/10.1038/ncpgasthep1331

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Emergency physicians’ perceptions of critical appraisal skills: a qualitative study.

• Sumintra Wood
• Jacqueline Paulis
• Angela Chen

BMC Medical Education (2022)

An integrative review on individual determinants of enrolment in National Health Insurance Scheme among older adults in Ghana

• Anthony Kwame Morgan
• Anthony Acquah Mensah

BMC Primary Care (2022)

Autopsy findings of COVID-19 in children: a systematic review and meta-analysis

• Anju Khairwa
• Kana Ram Jat

Forensic Science, Medicine and Pathology (2022)

The use of a modified Delphi technique to develop a critical appraisal tool for clinical pharmacokinetic studies

• Alaa Bahaa Eldeen Soliman
• Shane Ashley Pawluk
• Ousama Rachid

International Journal of Clinical Pharmacy (2022)

Critical Appraisal: Analysis of a Prospective Comparative Study Published in IJS

• Ramakrishna Ramakrishna HK
• Swarnalatha MC

Indian Journal of Surgery (2021)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Critically appraising...

Critically appraising qualitative research

• Related content
• Peer review
• Ayelet Kuper , assistant professor 1 ,
• Lorelei Lingard , associate professor 2 ,
• Wendy Levinson , Sir John and Lady Eaton professor and chair 3
• 1 Department of Medicine, Sunnybrook Health Sciences Centre, and Wilson Centre for Research in Education, University of Toronto, 2075 Bayview Avenue, Room HG 08, Toronto, ON, Canada M4N 3M5
• 2 Department of Paediatrics and Wilson Centre for Research in Education, University of Toronto and SickKids Learning Institute; BMO Financial Group Professor in Health Professions Education Research, University Health Network, 200 Elizabeth Street, Eaton South 1-565, Toronto
• 3 Department of Medicine, Sunnybrook Health Sciences Centre
• Correspondence to: A Kuper ayelet94{at}post.harvard.edu

Six key questions will help readers to assess qualitative research

Summary points

Appraising qualitative research is different from appraising quantitative research

Qualitative research papers should show appropriate sampling, data collection, and data analysis

Transferability of qualitative research depends on context and may be enhanced by using theory

Ethics in qualitative research goes beyond review boards’ requirements to involve complex issues of confidentiality, reflexivity, and power

Over the past decade, readers of medical journals have gained skills in critically appraising studies to determine whether the results can be trusted and applied to their own practice settings. Criteria have been designed to assess studies that use quantitative methods, and these are now in common use.

In this article we offer guidance for readers on how to assess a study that uses qualitative research methods by providing six key questions to ask when reading qualitative research (box 1). However, the thorough assessment of qualitative research is an interpretive act and requires informed reflective thought rather than the simple application of a scoring system.

Box 1 Key questions to ask when reading qualitative research studies

Was the sample used in the study appropriate to its research question, were the data collected appropriately, were the data analysed appropriately, can i transfer the results of this study to my own setting, does the study adequately address potential ethical issues, including reflexivity.

Overall: is what the researchers did clear?

One of the critical decisions in a qualitative study is whom or what to include in the sample—whom to interview, whom to observe, what texts to analyse. An understanding that qualitative research is based in experience and in the construction of meaning, combined with the specific research question, should guide the sampling process. For example, a study of the experience of survivors of domestic violence that examined their reasons for not seeking help from healthcare providers might focus on interviewing a sample of such survivors (rather than, for example, healthcare providers, social services workers, or academics in the field). The sample should be broad enough to capture the many facets of a phenomenon, and limitations to the sample should be clearly justified. Since the answers to questions of experience and meaning also relate to people’s social affiliations (culture, religion, socioeconomic group, profession, etc), it is also important that the researcher acknowledges these contexts in the selection of a study sample.

In contrast with quantitative approaches, qualitative studies do not usually have predetermined sample sizes. Sampling stops when a thorough understanding of the phenomenon under study has been reached, an end point that is often called saturation. Researchers consider samples to be saturated when encounters (interviews, observations, etc) with new participants no longer elicit trends or themes not already raised by previous participants. Thus, to sample to saturation, data analysis has to happen while new data are still being collected. Multiple sampling methods may be used to broaden the understanding achieved in a study (box 2). These sampling issues should be clearly articulated in the methods section.

Box 2 Qualitative sampling methods for interviews and focus groups 9

Examples are for a hypothetical study of financial concerns among adult patients with chronic renal failure receiving ongoing haemodialysis in a single hospital outpatient unit.

Typical case sampling —sampling the most ordinary, usual cases of a phenomenon

The sample would include patients likely to have had typical experiences for that haemodialysis unit and patients who fit the profile of patients in the unit for factors found on literature review. Other typical cases could be found via snowball sampling (see below)

Deviant case sampling —sampling the most extreme cases of a phenomenon

The sample would include patients likely to have had different experiences of relevant aspects of haemodialysis. For example, if most patients in the unit are 60-70 years old and recently began haemodialysis for diabetic nephropathy, researchers might sample the unmarried university student in his 20s on haemodialysis since childhood, the 32 year old woman with lupus who is now trying to get pregnant, and the 90 year old who newly started haemodialysis due to an adverse reaction to radio-opaque contrast dye. Other deviant cases could be found via theoretical and/or snowball sampling (see below)

Critical case sampling —sampling cases that are predicted (based on theoretical models or previous research) to be especially information-rich and thus particularly illuminating

The nature of this sample depends on previous research. For example, if research showed that marital status was a major determinant of financial concerns for haemodialysis patients, then critical cases might include patients whose marital status changed while on haemodialysis

Maximum-variation sampling —sampling as wide a range of perspectives as possible to capture the broadest set of information and experiences)

The sample would include typical, deviant, and critical cases (as above), plus any other perspectives identified

Confirming-disconfirming sampling —Sampling both individuals or texts whose perspectives are likely to confirm the researcher’s developing understanding of the phenomenon under study and those whose perspectives are likely to challenge that understanding

The sample would include patients whose experiences would likely either confirm or disconfirm what the researchers had already learnt (from other patients) about financial concerns among patients in the haemodialysis unit. This could be accomplished via theoretical and/or snowball sampling (see below)

Snowball sampling —sampling participants found by asking current participants in a study to recommend others whose experiences would be relevant to the study

Current participants could be asked to provide the names of others in the unit who they thought, when asked about financial concerns, would either share their views (confirming), disagree with their views (disconfirming), have views typical of patients on their unit (typical cases), or have views different from most other patients on their unit (deviant cases)

Theoretical sampling —sampling individuals or texts whom the researchers predict (based on theoretical models or previous research) would add new perspectives to those already represented in the sample

Researchers could use their understanding of known issues for haemodialysis patients that would, in theory, relate to financial concerns to ensure that the relevant perspectives were represented in the study. For example, if, as the research progressed, it turned out that none of the patients in the sample had had to change or leave a job in order to accommodate haemodialysis scheduling, the researchers might (based on previous research) choose to intentionally sample patients who had left their jobs because of the time commitment of haemodialysis (but who could not do peritoneal dialysis) and others who had switched to jobs with more flexible scheduling because of their need for haemodialysis

It is important that a qualitative study carefully describes the methods used in collecting data. The appropriateness of the method(s) selected to use for the specific research question should be justified, ideally with reference to the research literature. It should be clear that methods were used systematically and in an organised manner. Attention should be paid to specific methodological challenges such as the Hawthorne effect, 1 whereby the presence of an observer may influence participants’ behaviours. By using a technique called thick description, qualitative studies often aim to include enough contextual information to provide readers with a sense of what it was like to have been in the research setting.

Another technique that is often used is triangulation, with which a researcher uses multiple methods or perspectives to help produce a more comprehensive set of findings. A study can triangulate data, using different sources of data to examine a phenomenon in different contexts (for example, interviewing palliative patients who are at home, those who are in acute care hospitals, and those who are in specialist palliative care units); it can also triangulate methods, collecting different types of data (for example, interviews, focus groups, observations) to increase insight into a phenomenon.

Another common technique is the use of an iterative process, whereby concurrent data analysis is used to inform data collection. For example, concurrent analysis of an interview study about lack of adherence to medications among a particular social group might show that early participants seem to be dismissive of the efforts of their local pharmacists; the interview script might then be changed to include an exploration of this phenomenon. The iterative process constitutes a distinctive qualitative tradition, in contrast to the tradition of stable processes and measures in quantitative studies. Iterations should be explicit and justified with reference to the research question and sampling techniques so that the reader understands how data collection shaped the resulting insights.

Qualitative studies should include a clear description of a systematic form of data analysis. Many legitimate analytical approaches exist; regardless of which is used, the study should report what was done, how, and by whom. If an iterative process was used, it should be clearly delineated. If more than one researcher analysed the data (which depends on the methodology used) it should be clear how differences between analyses were negotiated. Many studies make reference to a technique called member checking, wherein the researcher shows all or part of the study’s findings to participants to determine if they are in accord with their experiences. 2 Studies may also describe an audit trail, which might include researchers’ analysis notes, minutes of researchers’ meetings, and other materials that could be used to follow the research process.

The contextual nature of qualitative research means that careful thought must be given to the potential transferability of its results to other sociocultural settings. Though the study should discuss the extent of the findings’ resonance with the published literature, 3 much of the onus of assessing transferability is left to readers, who must decide if the setting of the study is sufficiently similar for its results to be transferable to their own context. In doing so, the reader looks for resonance—the extent that research findings have meaning for the reader.

Transferability may be helped by the study’s discussion of how its results advance theoretical understandings that are relevant to multiple situations. For example, a study of patients’ preferences in palliative care may contribute to theories of ethics and humanity in medicine, thus suggesting relevance to other clinical situations such as the informed consent exchange before treatment. We have explained elsewhere in this series the importance of theory in qualitative research, and there are many who believe that a key indicator of quality in qualitative research is its contribution to advancing theoretical understanding as well as useful knowledge. This debate continues in the literature, 4 but from a pragmatic perspective most qualitative studies in health professions journals emphasise results that relate to practice; theoretical discussions tend to be published elsewhere.

Reflexivity is particularly important within the qualitative paradigm. Reflexivity refers to recognition of the influence a researcher brings to the research process. It highlights potential power relationships between the researcher and research participants that might shape the data being collected, particularly when the researcher is a healthcare professional or educator and the participant is a patient, client, or student. 5 It also acknowledges how a researcher’s gender, ethnic background, profession, and social status influence the choices made within the study, such as the research question itself and the methods of data collection. 6 7

Research articles written in the qualitative paradigm should show evidence both of reflexive practice and of consideration of other relevant ethical issues. Ethics in qualitative research should extend beyond prescriptive guidelines and research ethics boards into a thorough exploration of the ethical consequences of collecting personal experiences and opening those experiences to public scrutiny (a detailed discussion of this problem within a research report may, however, be limited by the practicalities of word count limitations). 8 Issues of confidentiality and anonymity can become quite complex when data constitute personal reports of experience or perception; the need to minimise harm may involve not only protection from external scrutiny but also mechanisms to mitigate potential distress to participants from sharing their personal stories.

In conclusion: is what the researchers did clear?

The qualitative paradigm includes a wide range of theoretical and methodological options, and qualitative studies must include clear descriptions of how they were conducted, including the selection of the study sample, the data collection methods, and the analysis process. The list of key questions for beginning readers to ask when reading qualitative research articles (see box 1) is intended not as a finite checklist, but rather as a beginner’s guide to a complex topic. Critical appraisal of particular qualitative articles may differ according to the theories and methodologies used, and achieving a nuanced understanding in this area is fairly complex.

Further reading

Crabtree F, Miller WL, eds. Doing qualitative research . 2nd ed. Thousand Oaks, CA: Sage, 1999.

Denzin NK, Lincoln YS, eds. Handbook of qualitative research . 2nd ed. Thousand Oaks, CA: Sage, 2000.

Finlay L, Ballinger C, eds. Qualitative research for allied health professionals: challenging choices . Chichester: Wiley, 2006.

Flick U. An introduction to qualitative research . 2nd ed. London: Sage, 2002.

Green J, Thorogood N. Qualitative methods for health research . London: Sage, 2004.

Lingard L, Kennedy TJ. Qualitative research in medical education . Edinburgh: Association for the Study of Medical Education, 2007.

Mauthner M, Birch M, Jessop J, Miller T, eds. Ethics in Qualitative Research . Thousand Oaks, CA: Sage, 2002.

Seale C. The quality of qualitative research . London: Sage, 1999.

Silverman D. Doing qualitative research . Thousand Oaks, CA: Sage, 2000.

Journal articles

Greenhalgh T. How to read a paper: papers that go beyond numbers. BMJ 1997;315:740-3.

Mays N, Pope C. Qualitative research: Rigour and qualitative research. BMJ 1995;311:109-12.

Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMJ 2000;320:50-2.

Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res 1998;8:341-51.

Internet resources

National Health Service Public Health Resource Unit. Critical appraisal skills programme: qualitative research appraisal tool . 2006. www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf

Cite this as: BMJ 2008;337:a1035

• Related to doi: , 10.1136/bmj.a288
• doi: , 10.1136/bmj.39602.690162.47
• doi: , 10.1136/bmj.a1020
• doi: , 10.1136/bmj.a879
• doi: 10.1136/bmj.a949

This is the last in a series of six articles that aim to help readers to critically appraise the increasing number of qualitative research articles in clinical journals. The series editors are Ayelet Kuper and Scott Reeves.

For a definition of general terms relating to qualitative research, see the first article in this series.

Contributors: AK wrote the first draft of the article and collated comments for subsequent iterations. LL and WL made substantial contributions to the structure and content, provided examples, and gave feedback on successive drafts. AK is the guarantor.

Funding: None.

Competing interests: None declared.

Provenance and peer review: Commissioned; externally peer reviewed.

• ↵ Holden JD. Hawthorne effects and research into professional practice. J Evaluation Clin Pract 2001 ; 7 : 65 -70. OpenUrl CrossRef PubMed Web of Science
• ↵ Hammersley M, Atkinson P. Ethnography: principles in practice . 2nd ed. London: Routledge, 1995 .
• ↵ Silverman D. Doing qualitative research . Thousand Oaks, CA: Sage, 2000 .
• ↵ Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMJ 2000 ; 320 : 50 -2. OpenUrl FREE Full Text
• ↵ Lingard L, Kennedy TJ. Qualitative research in medical education . Edinburgh: Association for the Study of Medical Education, 2007 .
• ↵ Seale C. The quality of qualitative research . London: Sage, 1999 .
• ↵ Wallerstein N. Power between evaluator and community: research relationships within New Mexico’s healthier communities. Soc Sci Med 1999 ; 49 : 39 -54. OpenUrl CrossRef PubMed Web of Science
• ↵ Mauthner M, Birch M, Jessop J, Miller T, eds. Ethics in qualitative research . Thousand Oaks, CA: Sage, 2002 .
• ↵ Kuzel AJ. Sampling in qualitative inquiry. In: Crabtree F, Miller WL, eds. Doing qualitative research . 2nd ed. Thousand Oaks, CA: Sage, 1999 :33-45.

• Teesside University Student & Library Services
• Learning Hub Group

Critical Appraisal for Health Students

• Critical Appraisal of a qualitative paper
• Critical Appraisal: Help
• Critical Appraisal of a quantitative paper
• Useful resources

Appraisal of a Qualitative paper : Top tips

• Introduction

Critical appraisal of a qualitative paper

This guide aimed at health students, provides basic level support for appraising qualitative research papers. It's designed for students who have already attended lectures on critical appraisal. One framework  for appraising qualitative research (based on 4 aspects of trustworthiness) is  provided and there is an opportunity to practise the technique on a sample article.

Support Materials

• Framework for reading qualitative papers
• Critical appraisal of a qualitative paper PowerPoint

To practise following this framework for critically appraising a qualitative article, please look at the following article:

Schellekens, M.P.J.  et al  (2016) 'A qualitative study on mindfulness-based stress reduction for breast cancer patients: how women experience participating with fellow patients',  Support Care Cancer , 24(4), pp. 1813-1820.

Critical appraisal of a qualitative paper: practical example.

• Credibility
• Transferability
• Dependability
• Confirmability

How to use this practical example 

Using the framework, you can have a go at appraising a qualitative paper - we are going to look at the following article: 

Step 1.  take a quick look at the article, step 2.  click on the credibility tab above - there are questions to help you appraise the trustworthiness of the article, read the questions and look for the answers in the article. , step 3.   click on each question and our answers will appear., step 4.    repeat with the other aspects of trustworthiness: transferability, dependability and confirmability ., questioning the credibility:, who is the researcher what has been their experience how well do they know this research area, was the best method chosen what method did they use was there any justification was the method scrutinised by peers is it a recognisable method was there triangulation ( more than one method used), how was the data collected was data collected from the participants at more than one time point how long were the interviews were questions asked to the participants in different ways, is the research reporting what the participants actually said were the participants shown transcripts / notes of the interviews / observations to ‘check’ for accuracy are direct quotes used from a variety of participants, how would you rate the overall credibility, questioning the transferability, was a meaningful sample obtained how many people were included is the sample diverse how were they selected, are the demographics given, does the research cover diverse viewpoints do the results include negative cases was data saturation reached, what is the overall transferability can the research be transferred to other settings , questioning the dependability :, how transparent is the audit trail can you follow the research steps are the decisions made transparent is the whole process explained in enough detail did the researcher keep a field diary is there a clear limitations section, was there peer scrutiny of the researchwas the research plan shown to peers / colleagues for approval and/or feedback did two or more researchers independently judge data, how would you rate the overall dependability would the results be similar if the study was repeated how consistent are the data and findings, questioning the confirmability :, is the process of analysis described in detail is a method of analysis named or described is there sufficient detail, have any checks taken place was there cross-checking of themes was there a team of researchers, has the researcher reflected on possible bias is there a reflexive diary, giving a detailed log of thoughts, ideas and assumptions, how do you rate the overall confirmability has the researcher attempted to limit bias, questioning the overall trustworthiness :, overall how trustworthy is the research, further information.

See Useful resources  for links, books and LibGuides to help with Critical appraisal.

• << Previous: Critical Appraisal: Help
• Next: Critical Appraisal of a quantitative paper >>
• Last Updated: May 29, 2024 10:15 AM
• URL: https://libguides.tees.ac.uk/critical_appraisal

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 21, Issue 4
• How to appraise quantitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

This article has a correction. Please see:

• Correction: How to appraise quantitative research - April 01, 2019

• Xabi Cathala 1 ,
• Calvin Moorley 2
• 1 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
• 2 Nursing Research and Diversity in Care , School of Health and Social Care, London South Bank University , London , UK
• Correspondence to Mr Xabi Cathala, Institute of Vocational Learning, School of Health and Social Care, London South Bank University London UK ; cathalax{at}lsbu.ac.uk and Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/eb-2018-102996

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Some nurses feel that they lack the necessary skills to read a research paper and to then decide if they should implement the findings into their practice. This is particularly the case when considering the results of quantitative research, which often contains the results of statistical testing. However, nurses have a professional responsibility to critique research to improve their practice, care and patient safety. 1  This article provides a step by step guide on how to critically appraise a quantitative paper.

Title, keywords and the authors

The authors’ names may not mean much, but knowing the following will be helpful:

Their position, for example, academic, researcher or healthcare practitioner.

Their qualification, both professional, for example, a nurse or physiotherapist and academic (eg, degree, masters, doctorate).

This can indicate how the research has been conducted and the authors’ competence on the subject. Basically, do you want to read a paper on quantum physics written by a plumber?

The abstract is a resume of the article and should contain:

Introduction.

Research question/hypothesis.

Methods including sample design, tests used and the statistical analysis (of course! Remember we love numbers).

Main findings.

Conclusion.

The subheadings in the abstract will vary depending on the journal. An abstract should not usually be more than 300 words but this varies depending on specific journal requirements. If the above information is contained in the abstract, it can give you an idea about whether the study is relevant to your area of practice. However, before deciding if the results of a research paper are relevant to your practice, it is important to review the overall quality of the article. This can only be done by reading and critically appraising the entire article.

The introduction

Example: the effect of paracetamol on levels of pain.

My hypothesis is that A has an effect on B, for example, paracetamol has an effect on levels of pain.

My null hypothesis is that A has no effect on B, for example, paracetamol has no effect on pain.

My study will test the null hypothesis and if the null hypothesis is validated then the hypothesis is false (A has no effect on B). This means paracetamol has no effect on the level of pain. If the null hypothesis is rejected then the hypothesis is true (A has an effect on B). This means that paracetamol has an effect on the level of pain.

Background/literature review

The literature review should include reference to recent and relevant research in the area. It should summarise what is already known about the topic and why the research study is needed and state what the study will contribute to new knowledge. 5 The literature review should be up to date, usually 5–8 years, but it will depend on the topic and sometimes it is acceptable to include older (seminal) studies.

Methodology

In quantitative studies, the data analysis varies between studies depending on the type of design used. For example, descriptive, correlative or experimental studies all vary. A descriptive study will describe the pattern of a topic related to one or more variable. 6 A correlational study examines the link (correlation) between two variables 7  and focuses on how a variable will react to a change of another variable. In experimental studies, the researchers manipulate variables looking at outcomes 8  and the sample is commonly assigned into different groups (known as randomisation) to determine the effect (causal) of a condition (independent variable) on a certain outcome. This is a common method used in clinical trials.

There should be sufficient detail provided in the methods section for you to replicate the study (should you want to). To enable you to do this, the following sections are normally included:

Overview and rationale for the methodology.

Participants or sample.

Data collection tools.

Methods of data analysis.

Ethical issues.

Data collection should be clearly explained and the article should discuss how this process was undertaken. Data collection should be systematic, objective, precise, repeatable, valid and reliable. Any tool (eg, a questionnaire) used for data collection should have been piloted (or pretested and/or adjusted) to ensure the quality, validity and reliability of the tool. 9 The participants (the sample) and any randomisation technique used should be identified. The sample size is central in quantitative research, as the findings should be able to be generalised for the wider population. 10 The data analysis can be done manually or more complex analyses performed using computer software sometimes with advice of a statistician. From this analysis, results like mode, mean, median, p value, CI and so on are always presented in a numerical format.

The author(s) should present the results clearly. These may be presented in graphs, charts or tables alongside some text. You should perform your own critique of the data analysis process; just because a paper has been published, it does not mean it is perfect. Your findings may be different from the author’s. Through critical analysis the reader may find an error in the study process that authors have not seen or highlighted. These errors can change the study result or change a study you thought was strong to weak. To help you critique a quantitative research paper, some guidance on understanding statistical terminology is provided in  table 1 .

• View inline

Some basic guidance for understanding statistics

Quantitative studies examine the relationship between variables, and the p value illustrates this objectively.  11  If the p value is less than 0.05, the null hypothesis is rejected and the hypothesis is accepted and the study will say there is a significant difference. If the p value is more than 0.05, the null hypothesis is accepted then the hypothesis is rejected. The study will say there is no significant difference. As a general rule, a p value of less than 0.05 means, the hypothesis is accepted and if it is more than 0.05 the hypothesis is rejected.

The CI is a number between 0 and 1 or is written as a per cent, demonstrating the level of confidence the reader can have in the result. 12  The CI is calculated by subtracting the p value to 1 (1–p). If there is a p value of 0.05, the CI will be 1–0.05=0.95=95%. A CI over 95% means, we can be confident the result is statistically significant. A CI below 95% means, the result is not statistically significant. The p values and CI highlight the confidence and robustness of a result.

Discussion, recommendations and conclusion

The final section of the paper is where the authors discuss their results and link them to other literature in the area (some of which may have been included in the literature review at the start of the paper). This reminds the reader of what is already known, what the study has found and what new information it adds. The discussion should demonstrate how the authors interpreted their results and how they contribute to new knowledge in the area. Implications for practice and future research should also be highlighted in this section of the paper.

A few other areas you may find helpful are:

Limitations of the study.

Conflicts of interest.

Table 2 provides a useful tool to help you apply the learning in this paper to the critiquing of quantitative research papers.

Quantitative paper appraisal checklist

• 1. ↵ Nursing and Midwifery Council , 2015 . The code: standard of conduct, performance and ethics for nurses and midwives https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21.8.18 ).
• Gerrish K ,
• Moorley C ,
• Tunariu A , et al
• Shorten A ,

Competing interests None declared.

Patient consent Not required.

Provenance and peer review Commissioned; internally peer reviewed.

Correction notice This article has been updated since its original publication to update p values from 0.5 to 0.05 throughout.

Linked Articles

• Miscellaneous Correction: How to appraise quantitative research BMJ Publishing Group Ltd and RCN Publishing Company Ltd Evidence-Based Nursing 2019; 22 62-62 Published Online First: 31 Jan 2019. doi: 10.1136/eb-2018-102996corr1

Read the full text or download the PDF: