definition of systematic review in research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 17 October 2022.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesise the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimise research b ias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinised by others.
• They’re thorough : they summarise all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomised control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective(s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgement of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

• Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2022, October 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 29 April 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

1.2.2  What is a systematic review?

A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question.  It  uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993) . The key characteristics of a systematic review are:

a clearly stated set of objectives with pre-defined eligibility criteria for studies;

an explicit, reproducible methodology;

a systematic search that attempts to identify all studies that would meet the eligibility criteria;

an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias; and

a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

Many systematic reviews contain meta-analyses. Meta-analysis is the use of statistical methods to summarize the results of independent studies (Glass 1976). By combining information from all relevant studies, meta-analyses can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review (see Chapter 9, Section 9.1.3 ). They also facilitate investigations of the consistency of evidence across studies, and the exploration of differences across studies.

Systematic Review

• Library Help
• What is a Systematic Review (SR)?
• Steps of a Systematic Review
• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Introduction to Systematic Review

• Introduction
• Types of literature reviews
• Other Libguides
• Systematic review as part of a dissertation
• Tutorials & Guidelines & Examples from non-Medical Disciplines

Depending on your learning style, please explore the resources in various formats on the tabs above.

For additional tutorials, visit the SR Workshop Videos  from UNC at Chapel Hill outlining each stage of the systematic review process.

Know the difference! Systematic review vs. literature review

Types of literature reviews along with associated methodologies

JBI Manual for Evidence Synthesis .  Find definitions and methodological guidance.

- Systematic Reviews - Chapters 1-7

- Mixed Methods Systematic Reviews -  Chapter 8

- Diagnostic Test Accuracy Systematic Reviews -  Chapter 9

- Umbrella Reviews -  Chapter 10

- Scoping Reviews -  Chapter 11

- Systematic Reviews of Measurement Properties -  Chapter 12

Systematic reviews vs scoping reviews - 

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal , 26 (2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1 (28). htt p s://doi.org/ 10.1186/2046-4053-1-28

Munn, Z., Peters, M., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018).  Systematic review or  scoping review ?  Guidance for authors when choosing between a systematic or scoping review approach.  BMC medical research methodology, 18 (1), 143. https://doi.org/10.1186/s12874-018-0611-x. Also, check out the  Libguide from Weill Cornell Medicine  for the  differences between a systematic review and a scoping review  and when to embark on either one of them.

Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements . Health Information & Libraries Journal , 36 (3), 202–222. https://doi.org/10.1111/hir.12276

Temple University. Review Types . - This guide provides useful descriptions of some of the types of reviews listed in the above article.

UMD Health Sciences and Human Services Library.  Review Types . - Guide describing Literature Reviews, Scoping Reviews, and Rapid Reviews.

Whittemore, R., Chao, A., Jang, M., Minges, K. E., & Park, C. (2014). Methods for knowledge synthesis: An overview. Heart & Lung: The Journal of Acute and Critical Care, 43 (5), 453–461. https://doi.org/10.1016/j.hrtlng.2014.05.014

Differences between a systematic review and other types of reviews

Armstrong, R., Hall, B. J., Doyle, J., & Waters, E. (2011). ‘ Scoping the scope ’ of a cochrane review. Journal of Public Health , 33 (1), 147–150. https://doi.org/10.1093/pubmed/fdr015

Kowalczyk, N., & Truluck, C. (2013). Literature reviews and systematic reviews: What is the difference? Radiologic Technology , 85 (2), 219–222.

White, H., Albers, B., Gaarder, M., Kornør, H., Littell, J., Marshall, Z., Matthew, C., Pigott, T., Snilstveit, B., Waddington, H., & Welch, V. (2020). Guidance for producing a Campbell evidence and gap map . Campbell Systematic Reviews, 16 (4), e1125. https://doi.org/10.1002/cl2.1125. Check also this comparison between evidence and gaps maps and systematic reviews.

Rapid Reviews Tutorials

Rapid Review Guidebook  by the National Collaborating Centre of Methods and Tools (NCCMT)

Hamel, C., Michaud, A., Thuku, M., Skidmore, B., Stevens, A., Nussbaumer-Streit, B., & Garritty, C. (2021). Defining Rapid Reviews: a systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews.  Journal of clinical epidemiology ,  129 , 74–85. https://doi.org/10.1016/j.jclinepi.2020.09.041

• Müller, C., Lautenschläger, S., Meyer, G., & Stephan, A. (2017). Interventions to support people with dementia and their caregivers during the transition from home care to nursing home care: A systematic review . International Journal of Nursing Studies, 71 , 139–152. https://doi.org/10.1016/j.ijnurstu.2017.03.013
• Bhui, K. S., Aslam, R. W., Palinski, A., McCabe, R., Johnson, M. R. D., Weich, S., … Szczepura, A. (2015). Interventions to improve therapeutic communications between Black and minority ethnic patients and professionals in psychiatric services: Systematic review . The British Journal of Psychiatry, 207 (2), 95–103. https://doi.org/10.1192/bjp.bp.114.158899
• Rosen, L. J., Noach, M. B., Winickoff, J. P., & Hovell, M. F. (2012). Parental smoking cessation to protect young children: A systematic review and meta-analysis . Pediatrics, 129 (1), 141–152. https://doi.org/10.1542/peds.2010-3209

Scoping Review

• Hyshka, E., Karekezi, K., Tan, B., Slater, L. G., Jahrig, J., & Wild, T. C. (2017). The role of consumer perspectives in estimating population need for substance use services: A scoping review . BMC Health Services Research, 171-14.  https://doi.org/10.1186/s12913-017-2153-z
• Olson, K., Hewit, J., Slater, L.G., Chambers, T., Hicks, D., Farmer, A., & ... Kolb, B. (2016). Assessing cognitive function in adults during or following chemotherapy: A scoping review . Supportive Care In Cancer, 24 (7), 3223-3234. https://doi.org/10.1007/s00520-016-3215-1
• Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency . Research Synthesis Methods, 5 (4), 371–385. https://doi.org/10.1002/jrsm.1123
• Scoping Review Tutorial from UNC at Chapel Hill

Qualitative Systematic Review/Meta-Synthesis

• Lee, H., Tamminen, K. A., Clark, A. M., Slater, L., Spence, J. C., & Holt, N. L. (2015). A meta-study of qualitative research examining determinants of children's independent active free play . International Journal Of Behavioral Nutrition & Physical Activity, 12 (5), 121-12. https://doi.org/10.1186/s12966-015-0165-9

Videos on systematic reviews

Systematic Reviews: What are they? Are they right for my research? - 47 min. video recording with a closed caption option.

More training videos  on systematic reviews:   

Books on Systematic Reviews

Books on Meta-analysis

• University of Toronto Libraries  - very detailed with good tips on the sensitivity and specificity of searches.
• Monash University  - includes an interactive case study tutorial. 
• Dalhousie University Libraries - a comprehensive How-To Guide on conducting a systematic review.

Guidelines for a systematic review as part of the dissertation

• Guidelines for Systematic Reviews in the Context of Doctoral Education Background  by University of Victoria (PDF)
• Can I conduct a Systematic Review as my Master’s dissertation or PhD thesis? Yes, It Depends!  by Farhad (blog)
• What is a Systematic Review Dissertation Like? by the University of Edinburgh (50 min video) 

Further readings on experiences of PhD students and doctoral programs with systematic reviews

Puljak, L., & Sapunar, D. (2017). Acceptance of a systematic review as a thesis: Survey of biomedical doctoral programs in Europe . Systematic Reviews , 6 (1), 253. https://doi.org/10.1186/s13643-017-0653-x

Perry, A., & Hammond, N. (2002). Systematic reviews: The experiences of a PhD Student . Psychology Learning & Teaching , 2 (1), 32–35. https://doi.org/10.2304/plat.2002.2.1.32

Daigneault, P.-M., Jacob, S., & Ouimet, M. (2014). Using systematic review methods within a Ph.D. dissertation in political science: Challenges and lessons learned from practice . International Journal of Social Research Methodology , 17 (3), 267–283. https://doi.org/10.1080/13645579.2012.730704

UMD Doctor of Philosophy Degree Policies

Before you embark on a systematic review research project, check the UMD PhD Policies to make sure you are on the right path. Systematic reviews require a team of at least two reviewers and an information specialist or a librarian. Discuss with your advisor the authorship roles of the involved team members. Keep in mind that the  UMD Doctor of Philosophy Degree Policies (scroll down to the section, Inclusion of one's own previously published materials in a dissertation ) outline such cases, specifically the following: 

" It is recognized that a graduate student may co-author work with faculty members and colleagues that should be included in a dissertation . In such an event, a letter should be sent to the Dean of the Graduate School certifying that the student's examining committee has determined that the student made a substantial contribution to that work. This letter should also note that the inclusion of the work has the approval of the dissertation advisor and the program chair or Graduate Director. The letter should be included with the dissertation at the time of submission.  The format of such inclusions must conform to the standard dissertation format. A foreword to the dissertation, as approved by the Dissertation Committee, must state that the student made substantial contributions to the relevant aspects of the jointly authored work included in the dissertation."

• Cochrane Handbook for Systematic Reviews of Interventions - See Part 2: General methods for Cochrane reviews
• Systematic Searches - Yale library video tutorial series 
• Using PubMed's Clinical Queries to Find Systematic Reviews  - From the U.S. National Library of Medicine
• Systematic reviews and meta-analyses: A step-by-step guide - From the University of Edinsburgh, Centre for Cognitive Ageing and Cognitive Epidemiology

Bioinformatics

• Mariano, D. C., Leite, C., Santos, L. H., Rocha, R. E., & de Melo-Minardi, R. C. (2017). A guide to performing systematic literature reviews in bioinformatics .  arXiv preprint arXiv:1707.05813.

Environmental Sciences

Collaboration for Environmental Evidence. 2018.  Guidelines and Standards for Evidence synthesis in Environmental Management. Version 5.0 (AS Pullin, GK Frampton, B Livoreil & G Petrokofsky, Eds) www.environmentalevidence.org/information-for-authors .

Pullin, A. S., & Stewart, G. B. (2006). Guidelines for systematic review in conservation and environmental management. Conservation Biology, 20 (6), 1647–1656. https://doi.org/10.1111/j.1523-1739.2006.00485.x

Engineering Education

• Borrego, M., Foster, M. J., & Froyd, J. E. (2014). Systematic literature reviews in engineering education and other developing interdisciplinary fields. Journal of Engineering Education, 103 (1), 45–76. https://doi.org/10.1002/jee.20038

Public Health

• Hannes, K., & Claes, L. (2007). Learn to read and write systematic reviews: The Belgian Campbell Group . Research on Social Work Practice, 17 (6), 748–753. https://doi.org/10.1177/1049731507303106
• McLeroy, K. R., Northridge, M. E., Balcazar, H., Greenberg, M. R., & Landers, S. J. (2012). Reporting guidelines and the American Journal of Public Health’s adoption of preferred reporting items for systematic reviews and meta-analyses . American Journal of Public Health, 102 (5), 780–784. https://doi.org/10.2105/AJPH.2011.300630
• Pollock, A., & Berge, E. (2018). How to do a systematic review.   International Journal of Stroke, 13 (2), 138–156. https://doi.org/10.1177/1747493017743796
• Institute of Medicine. (2011). Finding what works in health care: Standards for systematic reviews . https://doi.org/10.17226/13059
• Wanden-Berghe, C., & Sanz-Valero, J. (2012). Systematic reviews in nutrition: Standardized methodology . The British Journal of Nutrition, 107 Suppl 2, S3-7. https://doi.org/10.1017/S0007114512001432

Social Sciences

• Bronson, D., & Davis, T. (2012).  Finding and evaluating evidence: Systematic reviews and evidence-based practice (Pocket guides to social work research methods). Oxford: Oxford University Press.
• Petticrew, M., & Roberts, H. (2006).  Systematic reviews in the social sciences: A practical guide . Malden, MA: Blackwell Pub.
• Cornell University Library Guide -  Systematic literature reviews in engineering: Example: Software Engineering
• Biolchini, J., Mian, P. G., Natali, A. C. C., & Travassos, G. H. (2005). Systematic review in software engineering .  System Engineering and Computer Science Department COPPE/UFRJ, Technical Report ES, 679 (05), 45.
• Biolchini, J. C., Mian, P. G., Natali, A. C. C., Conte, T. U., & Travassos, G. H. (2007). Scientific research ontology to support systematic review in software engineering . Advanced Engineering Informatics, 21 (2), 133–151.
• Kitchenham, B. (2007). Guidelines for performing systematic literature reviews in software engineering . [Technical Report]. Keele, UK, Keele University, 33(2004), 1-26.
• Weidt, F., & Silva, R. (2016). Systematic literature review in computer science: A practical guide .  Relatórios Técnicos do DCC/UFJF ,  1 .
• Academic Phrasebank - Get some inspiration and find some terms and phrases for writing your research paper
• Oxford English Dictionary  - Use to locate word variants and proper spelling
• << Previous: Library Help
• Next: Steps of a Systematic Review >>
• Last Updated: Apr 19, 2024 12:47 PM
• URL: https://lib.guides.umd.edu/SR

Systematic Reviews

• What is a Systematic Review?

A systematic review is an evidence synthesis that uses explicit, reproducible methods to perform a comprehensive literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.

Key Characteristics of a Systematic Review:

Generally, systematic reviews must have:

• a clearly stated set of objectives with pre-defined eligibility criteria for studies
• an explicit, reproducible methodology
• a systematic search that attempts to identify all studies that would meet the eligibility criteria
• an assessment of the validity of the findings of the included studies, for example through the assessment of the risk of bias
• a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

A meta-analysis is a systematic review that uses quantitative methods to synthesize and summarize the pooled data from included studies.

Additional Information

• How-to Books
• Beyond Health Sciences

• Cochrane Handbook For Systematic Reviews of Interventions Provides guidance to authors for the preparation of Cochrane Intervention reviews. Chapter 6 covers searching for reviews.
• Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care From The University of York Centre for Reviews and Dissemination: Provides practical guidance for undertaking evidence synthesis based on a thorough understanding of systematic review methodology. It presents the core principles of systematic reviewing, and in complementary chapters, highlights issues that are specific to reviews of clinical tests, public health interventions, adverse effects, and economic evaluations.
• Cornell, Sytematic Reviews and Evidence Synthesis Beyond the Health Sciences Video series geared for librarians but very informative about searching outside medicine.
• << Previous: Getting Started
• Next: Levels of Evidence >>
• Getting Started
• Levels of Evidence
• Locating Systematic Reviews
• Searching Systematically
• Developing Answerable Questions
• Identifying Synonyms & Related Terms
• Using Truncation and Wildcards
• Identifying Search Limits/Exclusion Criteria
• Keyword vs. Subject Searching
• Where to Search
• Search Filters
• Sensitivity vs. Precision
• Core Databases
• Other Databases
• Clinical Trial Registries
• Conference Presentations
• Databases Indexing Grey Literature
• Web Searching
• Handsearching
• Citation Indexes
• Documenting the Search Process
• Managing your Review

Research Support

• Last Updated: May 1, 2024 4:09 PM
• URL: https://guides.library.ucdavis.edu/systematic-reviews

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 14, Issue 3
• What is a systematic review?
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Jane Clarke
• Correspondence to Jane Clarke 4 Prime Road, Grey Lynn, Auckland, New Zealand; janeclarkehome{at}gmail.com

https://doi.org/10.1136/ebn.2011.0049

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

A high-quality systematic review is described as the most reliable source of evidence to guide clinical practice. The purpose of a systematic review is to deliver a meticulous summary of all the available primary research in response to a research question. A systematic review uses all the existing research and is sometime called ‘secondary research’ (research on research). They are often required by research funders to establish the state of existing knowledge and are frequently used in guideline development. Systematic review findings are often used within the healthcare setting but may be applied elsewhere. For example, the Campbell Collaboration advocates the application of systematic reviews for policy-making in education, justice and social work.

Systematic reviews can be conducted on all types of primary research. Many are reviews of randomised trials (addressing questions of effectiveness), cross-sectional studies (addressing questions about prevalence or diagnostic accuracy, for example) or cohort studies (addressing questions about prognosis). When qualitative research is reviewed systematically, it may be described as a systematic review, but more often other terms such as meta-synthesis are used.

Systematic review methodology is explicit and precise and aims to minimise bias, thus enhancing the reliability of the conclusions drawn. 1 , 2 The features of a systematic review include:

■ clear aims with predetermined eligibility and relevance criteria for studies;

■ transparent, reproducible methods;

■ rigorous search designed to locate all eligible studies;

■ an assessment of the validity of the findings of the included studies and

■ a systematic presentation, and synthesis, of the included studies. 3

The first step in a systematic review is a meticulous search of all sources of evidence for relevant studies. The databases and citation indexes searched are listed in the methodology section of the review. Next, using predetermined reproducible criteria to screen for eligibility and relevance assessment of titles and the abstracts is completed. Each study is then assessed in terms of methodological quality.

Finally, the evidence is synthesised. This process may or may not include a meta-analysis. A meta-analysis is a statistical summary of the findings of independent studies. 4 Meta-analyses can potentially present more precise estimates of the effects of interventions than those derived from the individual studies alone. These strategies are used to limit bias and random error which may arise during this process. Without these safeguards, then, reviews can mislead, such that we gain an unreliable summary of the available knowledge.

The Cochrane Collaboration is a leader in the production of systematic reviews. Cochrane reviews are published on a monthly basis in the Cochrane Database of Systematic Reviews in The Cochrane Library (see: http://www.thecochranelibrary.com ).

• Antman EM ,
• Kupelnick B ,
• Higgins JPT ,

Competing interests None.

Read the full text or download the PDF:

Jump to navigation

• Bahasa Malaysia

What are systematic reviews?

Watch this video from   Cochrane Consumers and Communication to learn what systematic reviews are, how researchers prepare them, and why they’re an important part of making informed decisions about health - for everyone. 

Cochrane evidence, including our systematic reviews, provides a powerful tool to enhance your healthcare knowledge and decision making. This video from Cochrane Sweden explains a bit about how we create health evidence and what Cochrane does. 

• Search our Plain Language Summaries of health evidence
• Learn more about Cochrane and our work

• Research article
• Open access
• Published: 04 November 2019

Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks

• Marina Krnic Martinic 1 ,
• Dawid Pieper 2 ,
• Angelina Glatt 2 &
• Livia Puljak   ORCID: orcid.org/0000-0002-8467-6061 3  

BMC Medical Research Methodology volume  19 , Article number:  203 ( 2019 ) Cite this article

41k Accesses

113 Citations

31 Altmetric

Metrics details

A standard or consensus definition of a systematic review does not exist. Therefore, if there is no definition about a systematic review in secondary studies that analyse them or the definition is too broad, inappropriate studies might be included in such evidence synthesis. The aim of this study was to analyse the definition of a systematic review (SR) in health care literature, elements of the definitions that are used and to propose a starting point for an explicit and non-ambiguous SR definition.

We included overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks. We extracted the definitions of SRs, as well as the inclusion and exclusion criteria that could indicate which definition of a SR the authors used. We extracted individual elements of SR definitions, categorised and quantified them.

Among the 535 analysed sources of information, 188 (35%) provided a definition of a SR. The most commonly used reference points for the definitions of SRs were Cochrane and the PRISMA statement. We found 188 different elements of SR definitions and divided them into 14 categories. The highest number of SR definition elements was found in categories related to searching ( N  = 51), analysis/synthesis ( N  = 23), overall methods ( N  = 22), quality/bias/appraisal/validity ( N  = 22) and aim/question ( N  = 13). The same five categories were also the most commonly used combination of categories in the SR definitions.

Currently used definitions of SRs are vague and ambiguous, often using terms such as clear, explicit and systematic, without further elaboration. In this manuscript we propose a more specific definition of a systematic review, with the ultimate aim of motivating the research community to establish a clear and unambiguous definition of this type of research.

Peer Review reports

In 1990, the term evidence-based medicine (EBM) was coined [ 1 ]. It was hailed as a new approach for teaching and practising clinical medicine [ 2 ], incorporating “the best available external clinical evidence from a systematic search” [ 3 ]. When it comes to the best available evidence about treatment, randomised controlled trials (RCTs) and a systematic review (SR)/meta-analysis are considered the “gold standard” [ 1 ].

The EBM movement has been widely adopted, and evidence syntheses are regularly used to support clinical guidelines and recommendations for practice. However, it has been suggested that the EBM might be a movement in crisis [ 4 ], as there is “too much evidence” [ 4 ]. A study published in 2016 indicated that more than 8000 systematic reviews were being indexed annually in MEDLINE, corresponding to a three-fold increase over the last decade [ 5 ]. A search conducted in October 2019 showed that more than 15,000 studies published in 2018 were marked with a systematic review tag in PubMed.

Furthermore, some SRs might actually be misleading, redundant and conflicted [ 6 ]. A recent overview of systematic reviews found 12 systematic reviews and two major guidelines about thrombolytic therapy for pulmonary embolism published within less than 2 years. The results of those evidence syntheses were discordant, and the benefit-to-risk ratio was elusive [ 7 ]. Inclusion and exclusion criteria played a part in the origin of the discordant results [ 7 ].

Just as different inclusion and exclusion criteria might be a problem when conducting a systematic review, the same can happen in overviews of systematic reviews (OSRs) or other types of studies analysing systematic reviews, where results will depend upon inclusion criteria. The problem here is that a standard or consensus definition of a systematic review does not exist.

For example, in a study that reported about the increasing popularity of SRs, Page et al. [ 5 ] used the PRISMA-P explanation of a SR [ 8 ]. Using a definition when searching for SRs is important because there are studies that may call themselves SRs but are not SRs; we can only speculate that authors use a descriptor SR to label their studies because they are not aware of what a systematic review is, or because systematic reviews are considered to be a higher standard of review.

Therefore, if there is no definition about a systematic review in secondary studies that analyse them or the definition is too broad, inappropriate studies might be included in such evidence synthesis. The aim of this study was to explore and analyse the definition of a systematic review (SR) in health care literature, the elements of definitions that are used and propose a starting point for a new, explicit SR definition.

This was a methodological study, for which we developed a protocol a priori. The study protocol is available from the corresponding author on request.

Included studies

We aimed to collect definitions of systematic reviews in the health care literature. As numerous collections of SRs have already been published in the past, we relied on existing resources. We used three different sources: i) OSRs about healthcare interventions, ii) studies that have analysed the methodological quality of systematic reviews and iii) relevant textbooks/internet sources that define systematic reviews.

We included OSRs and methodological studies identified previously by Pieper et al. [ 9 , 10 ]. We used a validated filter for retrieval [ 11 ]. Additionally, we searched for EBM-related and epidemiology handbooks published in English or German. There was no systematic search for the handbooks. We compiled a list of relevant handbooks known to us, using the same methodological approach as described by other authors in similar projects [ 12 ].

Furthermore, we searched Google Scholar between the 24th January, 2018 and the 7th February, 2018 using the following search phrases: “definition of a systematic review”, “definition of systematic review”, “definition of the systematic review”, “defined a systematic review”, “defined the systematic review”, “systematic review was defined”. Those phrases were used to search any part of the manuscript – without any restrictions. We analysed the first 50 search results for each search phrase if there were more than 50 search phrases retrieved for a phrase. We excluded duplicate manuscripts found by searching multiple sources before starting the analysis.

Data extraction analysis

We piloted a data extraction form in Microsoft Excel on a sample of ten manuscripts. Two authors piloted the data extraction form (LP, MKM). Furthermore, based on the advice of a third author (DP), the form was further refined. Thus, in an iterative process between the authors, the form was modified where necessary to avoid any misunderstandings or later disagreements.

We extracted the following information: i) whether the analysed literature sources reported a definition of a systematic review and ii) inclusion and exclusion criteria defining systematic reviews. We extracted the relevant exclusion criteria when they had explicit statements about studies that were not included because certain aspects of them were not considered to be characteristics of a systematic review.

When we found a definition or inclusion/exclusion criteria defining systematic reviews, the text was extracted verbatim. Subsequently, from those definitions and the inclusion/exclusion criteria, we extracted elements of a systematic review definition. The definition elements were defined as distinct methodological components and their attributes. Elements described with similar adjectives were not combined; instead, we presented all unique elements separately in order to present a wide variety of adjectives and attributes used in the definitions of SRs. We did not use an a priori defined list of those elements; instead, we presented elements that we found in the analysed sources of information and we kept expanding the list of elements as we found new variations of the elements of the SR definition.

One of the elements of a definition we used was the presence of a meta-analysis (MA), but only if the authors explicitly indicated that the MA was considered as defining characteristic of a SR. For example, in a study published in 2013, Aziz wrote explicitly that SRs without MA were not included because “these were not considered SRs” [ 13 ].

Extracted individual elements of the SR definition were then categorised into groups. For example, if a SR definition was: “systematic search”, “reproducible search” or “keywords searched”, those elements were sorted into a category called “Search”. The process of forming categorisations was iterative between the authors until we reached a consensus about the categories that will be used.

We extracted reference(s) for a definition of a systematic review or inclusion criteria referring to the systematic review, if available. We recorded the 2017 Journal Citation Reports (JCR) Journal Impact Factor (JIF) of a journal from the Web of Science. We hypothesised that manuscripts published in journals with a higher JIF would have a higher prevalence of SR definitions, due to the higher reporting standards.

For all data, one author (MKM) extracted data and the second author (LP) verified the extractions. Furthermore, one author (LP) categorised the definition elements and the second author verified the categorisations (MKM). Any discrepancies in opinion were resolved via discussion.

For the analysis of definitions from textbooks and Internet sources, we extracted the definitions verbatim and indicated the field from which the definition came from, such as medicine, psychology and social sciences. During the analyses of the textbooks, if the definition in the text was supplemented with a table, we treated this as one source of information and the extracted elements of the SR definition from both text and table. One author extracted data and the second author verified the extractions from the textbooks and Internet sources.

Descriptive statistics, including frequencies and percentages, were used to describe the categories of elements of a systematic review definition/inclusion criteria. We also analysed the frequency of each category by counting the categories of elements that were used in each source. If at least one element was used in a certain category, we considered that this category of elements was present in the information source. We expressed the JIF as the mean and standard deviation (M ± SD), we used a t-test to analyse the difference in the JIR between the information sources with and without a SR definition. For the analyses, we used the MedCalc statistical software, v 15.2.1 (©MedCalc Software bvba, Ostend, Belgium). The statistical significance was set at P  < 0.05.

Search results

After searching for OSRs and methodological studies, from the 347 identified full texts, we included 308 studies. We excluded 39 studies because 31 were duplicates and an additional eight manuscripts were excluded because they were written in Chinese or did not fit our inclusion criteria (commentaries, traditional narrative reviews, reviews of an unspecified type of reviews or analysed rapid reviews).

By searching Google Scholar we found 531 hits. Based on the limits we set, analysing 50 hits per search phrase, we analysed a total of 238 bibliographic records from Google Scholar. After removing the duplicates that we already had in the first cohort of the included studies, we included the remaining 200 manuscripts from this cohort of studies. Additionally, we analysed 27 textbooks. In total, we analysed 535 sources of information: 508 manuscripts from peer-reviewed journals and 27 from textbooks.

Prevalence of definitions of SRs

Among the 535 analysed sources of information, 188 (35%) defined what they consider to be a systematic review, 62 (18%) had an inclusion criteria in the methods that allowed us to extract information about what the authors considered to be a systematic review and 59 (18%) had exclusion criteria that we used as well for determining the authors’ definition of a SR. Some sources of information had both a definition of a SR and/or inclusion/exclusion criteria; in total there were 226 sources of information from which we could extract information related to the authors’ definition of a SR.

Among the 508 manuscripts, we found a JIF for 401 manuscripts, of which 113 had a SR definition, and 288 did not. Journals that did not provide SR a definition had a higher JIF (4.4 ± 5.1) than those with a definition (3.7 ± 4.5), but this difference was not significant ( P  = 0.099).

Organisations, databases and checklists used as a reference for SR definition

Many of the analysed sources explicitly mentioned relevant organisations, checklists and databases for defining what they considered to be a SR, some of the analysed sources of information only provided literature references to support their definitions or inclusion/exclusion criteria.

Explicit mentions of the names of the organisations, checklists, databases associated with a definition of SRs or criteria for the inclusion of SRs were found in 43 out of 535 (8%) analysed sources of information. Those were Cochrane ( N  = 24), the PRISMA statement ( N  = 13), criteria of Database of Reviews of Effect (DARE) ( N  = 5), National Institute for Health and Care Excellence (NICE) ( N  = 3), NHS Centre for Reviews and Dissemination (N = 3), Campbell collaboration (N = 2) National Health and Medical Research Council ( N  = 1), QUOROM (QUality Of Reporting of Meta-analyses) recommendations (N = 1), Guidelines from Agency for Healthcare Research and Quality (AHRQ) (N = 1), Institute of Medicine (IOM) (N = 1) and author Andy Oxman (N = 1), referred to as the “Oxman criteria“. Cochrane was mentioned most commonly, either as a reference to a whole organisation, the Cochrane Handbook for Systematic Reviews of Interventions or a specific Cochrane entity: the Dutch Cochrane Centre in one source of information. Details about the definitions and references provided in those 43 studies are shown in Additional file 1 : Table S1. The most commonly used supporting references in those studies were the manuscripts by Moher et al. and Liberati et al. describing the PRISMA statement, the PRISMA-P checklist, and the Cochrane Handbook (Additional file 1 : Table S1).

The most commonly used literature references that were used to support the statements provided in the definitions of SRs or the inclusion/exclusion criteria were also manuscripts describing the PRISMA statement and Cochrane Handbook (Additional file 1 : Table S2).

Elements of systematic review definitions

After analysing all the definitions of SRs and the inclusion/exclusion criteria for SRs, we extracted 188 individual elements of a SR definition; we categorised them into the following 14 categories: self-identification, indexing, aim/question, overall methods, search, identification of studies, selection of studies, study eligibility, data extraction, quality/bias/appraisal/validity, analysis/synthesis, describing included studies, reporting and unclear (Table 1 ).

Elements were sorted according to those categories (Table 1 ). The highest number of SR definition elements was found in categories related to searching ( N  = 51), analysis/synthesis ( N  = 23), overall methods ( N  = 22), quality/bias/appraisal/validity (N = 22) and aim/question ( N  = 13) (Table 1 ).

Categories of systematic review elements

Among the 226 sources of information that had a SR definition or inclusion/exclusion criteria that could be used for extracting individual elements of a SR definition, 59 used only one category, 62 used two categories, while 105 used from three to ten categories of the SR definition elements. When we looked at the combinations that were used, none of the combinations of various categories was used more than ten times. The most commonly used combination of SR definition categories was used in nine of the manuscripts/books, and it used the following five categories: i) aim/research question, ii) search, iii) study eligibility, iv) quality, bias, appraisal, validity and v) analysis/synthesis. However, those nine manuscripts had different wording of the SR definition, as shown in Additional file 2 : Table S3; they did not use one consistent definition.

The same five categories were the most commonly used SR definition categories in our sample of information sources, with the following frequencies: i) search ( N  = 122), ii) aim/research question ( N  = 93), iii) analysis/synthesis ( N  = 90), iv) study eligibility ( N  = 89) and v) quality, bias, appraisal, validity ( N  = 81).

We found that authors of manuscripts and textbooks use various definitions of systematic reviews; in 535 sources of information, we found 188 different elements of a SR definition. The most commonly used categories of SR definition elements were related to searching, analysis/synthesis, overall methods, quality/bias/appraisal/validity and aim/question. The most commonly used reference resources were the Cochrane and PRISMA statement [ 14 , 15 ].

However, as our study showed, there is no uniformly used definition of a SR. We analysed various sources of information, including overviews of SRs and methodological studies about SRs because those studies included SRs and we expected that therefore they should provide a definition of a SR. Our expectations were not met; as we found that one-third of those information sources used an explicit definition of a SR. In another one-third of the information sources, we found either inclusion or exclusion criteria, from which we could deduce what they consider to be, or not to be, a SR.

Also, we found that journals that did not provide SR definition had a higher JIF than those with a definition, but this difference was not significant. This finding was not in line with our hypothesis, and it shows that in this respect journals with higher JIF did not have higher expectations from authors in terms of transparent reporting about what was considered to be a SR.

When extracting the elements of SR definitions, we tried to be as detailed as possible, to capture various terminology used in those definitions. We found many variations of similar concepts, but also many vague terms. Such vague terms were frequently reflected in the usage of the word systematic, such as: “systematic methods”, “systematic approach”, “systematic search”, “systematic synthesis”, “systematic analysis” and “systematic presentation”, without actually explaining what systematic means. We also found two expressions that were completely unclear about what the authors consider to be a SR, including “ Reviews were included if they were systematic ” and “ It was apparent in the text that a systematic review had been undertaken ”.

There were ten elements of a SR definition that used the type and number of sources that were searched in a SR, as an element of the SR definition. It has been suggested previously that a minimum number and types of sources should define SRs because searching only one database may not be universally considered a systematic search [ 16 ].

It could be argued that our categorisation was too detailed, as some of our categories of SR definition elements sound similar, for example, categories search, selection of studies, identification of studies and study eligibility. We left those categories as they were on purpose, because it may not be perfectly obvious what the difference between them is; for example, the term selection of studies in the Cochrane reviews is reserved for the description of the screening of abstracts and full texts, but it is unclear whether all authors use this term in the same context. Furthermore, it is unclear whether the identification of studies refers to searching, screening or eligibility, i.e. the inclusion/exclusion criteria. Because of this ambiguity, we chose to present more detailed categories.

The most commonly used individual five categories of the SR elements were also used as the most common combination of elements in the analysed sources of information, but only nine manuscripts used this combination of the five elements. Those five categories of elements are also included in the definition of SRs from the Cochrane Handbook [ 14 ].

In section 1.2.2 of the Cochrane Handbook, titled What is a systematic review?, the following definition can be found [quote]: “ A systematic review attempts to collate all empirical evidence that fits the pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimising bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993). The key characteristics of a systematic review are: a clearly stated set of objectives with pre-defined eligibility criteria for the studies; an explicit, reproducible methodology; a systematic search that attempts to identify all the studies that would meet the eligibility criteria; an assessment of the validity of the findings of the included studies, for example through the assessment of the risk of bias; and a systematic presentation, and synthesis, of the characteristics and findings of the included studies” [ 14 ].

Also, Cochrane was the most commonly mentioned organisation in the definitions of SRs; 13% of the manuscripts/textbooks mentioned Cochrane as a source of the SR definition. Therefore, one could argue that the Cochrane’s definition could be used as a formal definition of what a SR is. However, the Cochrane’s definition is also vague, as it is unclear what it means “ explicit, systematic methods” or “explicit, reproducible methodology”. Someone can explicitly describe the methodology that is not adequate. This inadequate methodology may also be reproducible, but that does not mean that it is good. Furthermore, the Cochrane definition of a SR repeatedly uses the adjective “systematic”, without explaining what the meaning of systematic is.

Two references used in Cochrane’s definition of a SR are those of Antman et al. [ 17 ] and Oxman et al. [ 18 ]. We also analysed which references were used to support the definitions of SRs in the manuscripts and textbooks; we found that the authors most commonly referred to the PRISMA statement [ 15 ] and Cochrane Handbook. However, the definition of SRs from the PRISMA statement manuscripts also uses vague terms such as clearly, systematic and explicit, without going into details of what they entail [ 15 ].

The research community would benefit from having a very specific definition of a SR. The five most commonly used SR definition elements that we identified could be used to create a more elaborate and unambiguous definition of a SR. We believe that the international research community should create an unambiguous SR definition; we hope that this study will be a starting point in that direction. As a first step, we suggest starting with the following template:

A systematic review is a review that reports or includes the following:

research question

sources that were searched, with a reproducible search strategy (naming of databases, naming of search platforms/engines, search date and complete search strategy)

inclusion and exclusion criteria

selection (screening) methods

critically appraises and reports the quality/risk of bias of the included studies

information about data analysis and synthesis that allows the reproducibility of the results

Some of those elements are mentioned in the SR definition from the Cochrane Handbook [ 14 ], as shown above, but the Cochrane’s definition still leaves a lot of ambiguity in several aspects. Those elements should be more specific in future. For example, which details should the clinical question report, how many databases/sources should be searched to be considered systematic, whether key methodological aspects (screening of titles and abstracts, screening of full texts, data extraction and risk of bias assessment) should be done by two authors independently or done by one author and verified by another. The naming of the databases is important for ensuring transparency and reproducibility, which should be features of a systematic approach. Those and other considerations should be taken into account in further efforts to clarify what exactly makes a SR.

Information presented in this manuscript could help inform a consensus meeting or a similar gathering where interested SR researchers could contribute to standardising a SR definition. A similar approach was recently suggested for the definition of a predatory journal. Cobey et al. have conducted a scoping review in which they summarised the literature on predatory journals, described its epidemiological characteristics and extracted empirical descriptions of the potential characteristics of predatory journals. In their conclusions, they informed readers that the results will be shared with attendees that will attend a stakeholder meeting seeking to develop a standardised definition for what constitutes a predatory journal [ 19 ].

One limitation of our study could be the use of information sources published within a certain period of time. However, this type of work, which relies on analysis of published literature, usually suffers from a time lag. Each new update of the search results in new literature sources to analyse, and time lag appears again by the time analysis is completed.

Furthermore, in our approach, we analysed both expressions that appeared to be definitions of SRs and the characteristics of SRs eligible for inclusion. It may be considered that the inclusion criteria for SRs are not eligible elements to define what a SR is. However, we considered that the eligibility and inclusion criteria which describe SRs would be useful in our analysis, as we seldom found explicit statements about the definition of a SR. We consider that the range of descriptors we found indicates a very rich vocabulary used by authors who are defining or searching for SRs and that our approach is an adequate starting point towards building a future consensus definition of a systematic review. Likewise, it could be argued that we are mixing a definition of a SR with measures of the quality of a SR. However, in the absence of an existing definition, we believe that we should assess all the descriptors used for SRs and report them explicitly and transparently, then readers can see for themselves that some of those may overlap with quality descriptors. Also, for searching Google Scholar we used a limited number of phrases. Google used to include details for searching the advanced interface, which is no longer available but this search information could be available from other sites (mostly libraries) which we did not utilize.

Our analysis is also limited by the fact that we only focused on the definitions, while we acknowledge that some relevant information might also be found in the explanatory text to the definition, if available.

The majority of manuscripts that include SRs actually do not provide a definition of what they consider to be a SR. The most commonly used reference sources of a SR definition use vague and ambiguous terms. We propose a new definition of a systematic review, which is open for further commenting and elaboration, with the aim of motivating the research community to create a more specific definition of this type of research.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Agency for Healthcare Research and Quality

Database of Reviews of Effect

Evidence-based medicine

Institute of Medicine

Journal Citation Reports

Journal Impact Factor

Mean and standard deviation

Meta-analysis

National Health Service

National Institute for Health and Care Excellence

Overview of systematic reviews

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols

QUality Of Reporting of Meta-analyses

Randomized controlled trial

• Systematic review

Sur RL, Dahm P. History of evidence-based medicine. Indian journal of urology : IJU : journal of the Urological Society of India. 2011;27(4):487–9.

Article   Google Scholar  

Group EW. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268(17):2420–5.

Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71–2.

Article   CAS   Google Scholar  

Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis? BMJ. 2014;348:g3725.

Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, Catala-Lopez F, Li L, Reid EK, Sarkis-Onofre R, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028.

Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.

Riva N, Puljak L, Moja L, Ageno W, Schunemann H, Magrini N, Squizzato A. Multiple overlapping systematic reviews facilitate the origin of disputes: the case of thrombolytic therapy for pulmonary embolism. J Clin Epidemiol. 2017.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic reviews. 2015;4:1.

Pieper D, Pollock M, Fernandes RM, Buchter RB, Hartling L. Epidemiology and reporting characteristics of overviews of reviews of healthcare interventions published 2012-2016: protocol for a systematic review. Systematic reviews. 2017;6(1):73.

Pieper D, Koensgen N, Breuing J, Ge L, Wegewitz U. How is AMSTAR applied by authors – a call for better reporting. Unpublished data. 2017.

Lunny C, McKenzie JE, McDonald S. Retrieval of overviews of systematic reviews in MEDLINE was improved by the development of an objectively derived and validated search strategy. J Clin Epidemiol. 2016;74:107–18.

Schwartz S, Campbell UB, Gatto NM, Gordon K. Toward a clarification of the taxonomy of "bias" in epidemiology textbooks. Epidemiology. 2015;26(2):216–22.

Aziz T, Compton S, Nassar U, Matthews D, Ansari K, Flores-Mir C. Methodological quality and descriptive characteristics of prosthodontic-related systematic reviews. J Oral Rehabil. 2013;40(4):263–78.

Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? Jama. 1998;280(3):280–2.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Puljak L. If there is only one author or only one database was searched, a study should not be called a systematic review. J Clin Epidemiol. 2017.

Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992;268(2):240–8.

Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci. 1993;703:125–33 discussion 133-124.

Cobey KD, Lalu MM, Skidmore B, Ahmadzai N, Grudniewicz A, Moher D. What is a predatory journal? A scoping review. F1000Research. 2018; 7 :1001.

Download references

Acknowledgments

We are grateful to Dr. Svjetlana Dosenovic for critical reading of the manuscript. The manuscript was revised by a native English speaker from the company Proof-Reading-Service.com Ltd., Hertfordshire, UK.

No external funding.

Author information

Authors and affiliations.

Department of Otorhinolaryngology, University Hospital Split, Split, Croatia

Marina Krnic Martinic

Institute for Research in Operative Medicine (IFOM), Witten/Herdecke University, Cologne, Germany

Dawid Pieper & Angelina Glatt

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

You can also search for this author in PubMed   Google Scholar

Contributions

Study design: LP, DP. Data extraction: MKM, AG. Data analysis and interpretation: MKM, AG, DP, LP. Writing the first draft of the manuscript: MKM, LP. Revisions of the manuscript for important intellectual content: MKM, AG, DP, LP. Final approval of the manuscript: MKM, AG, DP, LP. Agree to be accountable for all aspects of the work: MKM, AG, DP, LP. Guarantor: LP.

Corresponding author

Correspondence to Livia Puljak .

Ethics declarations

Ethics approval and consent to participate.

This study involved only analysis of data from published scientific literature; we did not collect any participant data.

Consent for publication

Not applicable.

Competing interests

Livia Puljak is the section editor of the BMC Medical Research Methodology, but she was not involved in handling of this manuscript in any way. The remaining authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: table s1..

Definitions of systematic reviews which explicitly quoted specific organizations/checklists/criteria. A table that contains definitions of systematic reviews, extracted from analyzed data sources, in which the authors hav explicitly quoted specific organizations, or checklists or criteria. Table S2. All references that were used to support definition of systematic review, or inclusion/exclusion criteria that could be used as a proxy for a definition. A list of references that autohrs of analyzed information sources have used in their manuscript to support either a definition of systematic review, or inclusion criteria, or exclusion criteria that could be used as a proxy indicator of a systematic review definition

Additional file 2: Table S3.

Definitions of a systematic review from nine manuscripts with the most commonly used combination of categories. A file containing verbatim extracted definitions that were used in nine manuscripts that had the most commonly used combination of categories of systematic review definitions

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Krnic Martinic, M., Pieper, D., Glatt, A. et al. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol 19 , 203 (2019). https://doi.org/10.1186/s12874-019-0855-0

Download citation

Received : 08 April 2019

Accepted : 18 October 2019

Published : 04 November 2019

DOI : https://doi.org/10.1186/s12874-019-0855-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research methodology

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Systematic Reviews: What is a systematic review?

What is a systematic review.

• Planning your review
• Additional resources

A systematic review is a tightly structured literature review that focuses on a topic with strict research parameters. The methodology used to collect research has to be consistent in order to reduce misinterpretation and misrepresentation of the data.

To help you understand and conduct your systematic review we have produce a number of posts to help you: 

• Getting started with your systematic review
• Introduction to Health Science databases
• Planning your systematic review
• Using frameworks to structure your search
• Advanced search: making use of Boolean operators
• Systematic review: organising your keywords and subject terms
• Making use of MeSH and Suggested Subject Terms

You can access these and more from the Specialist Library Support online resources page .

What is a Systematic literature search?

A systematic literature search is a literature review on a database (such as Medline) which demonstrates that you have compiled a list of appropriate search terms and includes the structure of your search history which provides the evidence on which your assignment is based.

This is a less rigorous process than a systematic review. A systematic review usually covers a wider scope; you would be expected to look at all the available research in the area in question. For example, you would be expected to visit the Library if articles were only held in hard copy format, and where necessary obtain articles not held by the Library via the Inter-Library Loan service .

You may be told that you need to conduct a systematic review when in fact you just need to perform a literature search in a systematic manner. 

If you are unsure about the differences between a systematic review and a literature review take a look at this guide:  What’s in a Name? The difference between a Systematic Review and a Literature Review and Why it Matters .

• << Previous: Home
• Next: Planning your review >>
• Last Updated: Feb 6, 2024 3:18 PM
• URL: https://subjects.library.manchester.ac.uk/systematic-reviews

Evidence Synthesis and Systematic Reviews

Systematic reviews, rapid reviews, scoping reviews.

• Other Review Types
• Resources for Reviews by Discipline and Type
• Tools for Evidence Synthesis
• Grey Literature

Definition : A systematic review is a summary of research results (evidence) that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and errors.

When to use : If you want to identify, appraise, and synthesize all available research that is relevant to a particular question with reproduceable search methods.

Limitations : It requires extensive time and a team

Resources :

• Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare
• The 8 stages of a systematic review
• Determining the scope of the review and the questions it will address
• Reporting the review

Definition : Rapid reviews are a form of evidence synthesis that may provide more timely information for decision making compared with standard systematic reviews

When to use : When you want to evaluate new or emerging research topics using some systematic review methods at a faster pace

Limitations : It is not as rigorous or as thorough as a systematic review and therefore may be more likely to be biased

• Cochrane guidance for rapid reviews
• Steps for conducting a rapid review
• Expediting systematic reviews: methods and implications of rapid reviews

Definition : Scoping reviews are often used to categorize or group existing literature in a given field in terms of its nature, features, and volume.

When to use : Label body of literature with relevance to time, location (e.g. country or context), source (e.g. peer-reviewed or grey literature), and origin (e.g. healthcare discipline or academic field) It also is used to clarify working definitions and conceptual boundaries of a topic or field or to identify gaps in existing literature/research

Limitations : More citations to screen and takes as long or longer than a systematic review.  Larger teams may be required because of the larger volumes of literature.  Different screening criteria and process than a systematic review

• PRISMA-ScR for scoping reviews
• JBI Updated methodological guidance for the conduct of scoping reviews
• JBI Manual: Scoping Reviews (2020)
• Equator Network-Current Best Practices for the Conduct of Scoping Reviews
• << Previous: Home Page
• Next: Other Review Types >>
• Last Updated: Feb 14, 2024 8:15 AM
• URL: https://guides.temple.edu/systematicreviews

Temple University

University libraries.

See all library locations

• Library Directory
• Locations and Directions
• Frequently Called Numbers

Need help? Email us at [email protected]

Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks

Affiliations.

• 1 Department of Otorhinolaryngology, University Hospital Split, Split, Croatia.
• 2 Institute for Research in Operative Medicine (IFOM), Witten/Herdecke University, Cologne, Germany.
• 3 Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia. [email protected].
• PMID: 31684874
• PMCID: PMC6829801
• DOI: 10.1186/s12874-019-0855-0

Background: A standard or consensus definition of a systematic review does not exist. Therefore, if there is no definition about a systematic review in secondary studies that analyse them or the definition is too broad, inappropriate studies might be included in such evidence synthesis. The aim of this study was to analyse the definition of a systematic review (SR) in health care literature, elements of the definitions that are used and to propose a starting point for an explicit and non-ambiguous SR definition.

Methods: We included overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks. We extracted the definitions of SRs, as well as the inclusion and exclusion criteria that could indicate which definition of a SR the authors used. We extracted individual elements of SR definitions, categorised and quantified them.

Results: Among the 535 analysed sources of information, 188 (35%) provided a definition of a SR. The most commonly used reference points for the definitions of SRs were Cochrane and the PRISMA statement. We found 188 different elements of SR definitions and divided them into 14 categories. The highest number of SR definition elements was found in categories related to searching (N = 51), analysis/synthesis (N = 23), overall methods (N = 22), quality/bias/appraisal/validity (N = 22) and aim/question (N = 13). The same five categories were also the most commonly used combination of categories in the SR definitions.

Conclusion: Currently used definitions of SRs are vague and ambiguous, often using terms such as clear, explicit and systematic, without further elaboration. In this manuscript we propose a more specific definition of a systematic review, with the ultimate aim of motivating the research community to establish a clear and unambiguous definition of this type of research.

Keywords: Definition; Research methodology; Systematic review.

• Biomedical Research / classification
• Biomedical Research / methods
• Epidemiologic Studies
• Evidence-Based Medicine / classification
• Evidence-Based Medicine / methods
• Meta-Analysis as Topic*
• Reproducibility of Results
• Review Literature as Topic*
• Systematic Reviews as Topic*
• Textbooks as Topic

• Open access
• Published: 29 April 2024

What is context in knowledge translation? Results of a systematic scoping review

• Tugce Schmitt   ORCID: orcid.org/0000-0001-6893-6428 1 ,
• Katarzyna Czabanowska 1 &
• Peter Schröder-Bäck 1  

Health Research Policy and Systems volume  22 , Article number:  52 ( 2024 ) Cite this article

27 Accesses

3 Altmetric

Metrics details

Knowledge Translation (KT) aims to convey novel ideas to relevant stakeholders, motivating their response or action to improve people’s health. Initially, the KT literature focused on evidence-based medicine, applying findings from laboratory and clinical research to disease diagnosis and treatment. Since the early 2000s, the scope of KT has expanded to include decision-making with health policy implications.

This systematic scoping review aims to assess the evolving knowledge-to-policy concepts, that is, macro-level KT theories, models and frameworks (KT TMFs). While significant attention has been devoted to transferring knowledge to healthcare settings (i.e. implementing health policies, programmes or measures at the meso-level), the definition of 'context' in the realm of health policymaking at the macro-level remains underexplored in the KT literature. This study aims to close the gap.

A total of 32 macro-level KT TMFs were identified, with only a limited subset of them offering detailed insights into contextual factors that matter in health policymaking. Notably, the majority of these studies prompt policy changes in low- and middle-income countries and received support from international organisations, the European Union, development agencies or philanthropic entities.

Peer Review reports

Few concepts are used by health researchers as vaguely and yet as widely as Knowledge Translation (KT), a catch-all term that accommodates a broad spectrum of ambitions. Arguably, to truly understand the role of context in KT, we first need to clarify what KT means. The World Health Organization (WHO) defines KT as ‘the synthesis, exchange and application of knowledge by relevant stakeholders to accelerate the benefits of global and local innovation in strengthening health systems and improving people’s health’ [ 1 ]. Here, particular attention should be paid to ‘innovation’, given that without unpacking this term, the meaning of KT would still remain ambiguous. Rogers’ seminal work ‘Diffusion of Innovations’ [ 2 ] defines innovation as an idea, practice or object that is perceived as novel by individuals or groups adopting it. In this context, he argues that the objective novelty of an idea in terms of the amount of time passed after its discovery holds little significance [ 2 ]. Rather, it is the subjective perception of newness by the individual that shapes their response [ 2 ]. In other words, if an idea seems novel to individuals, and thereby relevant stakeholders according to the aforementioned WHO definition, it qualifies as an innovation. From this perspective, it can be stated that a fundamental activity of KT is to communicate ideas that could be perceived as original to the targeted stakeholders, with the aim of motivating their response to improve health outcomes. This leaves us with the question of who exactly these stakeholders might be and what kind of actions would be required from them.

The scope of stakeholders in KT has evolved over time, along with their prompted responses. Initially, during the early phases of KT, the focus primarily revolved around healthcare providers and their clinical decisions, emphasising evidence-based medicine. Nearly 50 years ago, the first scientific article on KT was published, introducing Tier 1 KT, which concentrated on applying laboratory discoveries to disease diagnosis or treatment, also known as bench-to-bedside KT [ 3 ]. The primary motivation behind this initial conceptualisation of KT was to engage healthcare providers as the end-users of specific forms of knowledge, primarily related to randomised controlled trials of pharmaceuticals and evidence-based medicine [ 4 ]. In the early 2000s, the second phase of KT (Tier 2) emerged under the term ‘campus-to-clinic KT’ [ 3 ]. This facet, also known as translational research, was concerned with using evidence from health services research in healthcare provision, both in practice and policy [ 4 ]. Consequently, by including decision-makers as relevant end-users, KT scholars expanded the realm of research-to-action from the clinical environment to policy-relevant decision-making [ 5 ]. Following this trajectory, additional KT schemes (Tier 3–Tier 5) have been introduced into academic discourse, encompassing the dissemination, implementation and broader integration of knowledge into public policies [ 6 , 7 ]. Notably, the latest scheme (Tier 5) is becoming increasingly popular and represents the broadest approach, which describes the translation of knowledge to global communities and aims to involve fundamental, universal change in attitudes, policies and social systems [ 7 ].

In other words, a noticeable shift in KT has occurred with time towards macro-level interventions, named initially as evidence- based policymaking and later corrected to evidence- informed policymaking. In parallel with these significant developments, various alternative terms to KT have emerged, including ‘implementation science’, ‘knowledge transfer’, and ‘dissemination and research use’, often with considerable overlap [ 8 ]. Arguably, among the plethora of alternative terms proposed, implementation science stands out prominently. While initially centred on evidence-based medicine at the meso-level (e.g. implementing medical guidelines), it has since broadened its focus to ‘encompass all aspects of research relevant to the scientific study of methods to promote the uptake of research findings into routine settings in clinical, community and policy contexts’ [ 9 ], closely mirroring the definition to KT. Thus, KT, along with activities under different names that share the same objective, has evolved into an umbrella term over the years, encompassing a wide range of strategies aimed at enhancing the impact of research not only on clinical practice but also on public policies [ 10 ]. Following the adoption of such a comprehensive definition of KT, some researchers have asserted that using evidence in public policies is not merely commendable but essential [ 11 ].

In alignment with the evolution of KT from (bio-)medical sciences to public policies, an increasing number of scholars have offered explanations on how health policies should be developed [ 12 ], indicating a growing focus on exploring the mechanisms of health policymaking in the KT literature. However, unlike in the earlier phases of KT, which aimed to transfer knowledge from the laboratory to healthcare provision, decisions made for public policies may be less technical and more complex than those in clinical settings [ 3 , 13 , 14 ]. Indeed, social scientists point out that scholarly works on evidence use in health policies exhibit theoretical shortcomings as they lack engagement with political science and public administration theories and concepts [ 15 , 16 , 17 , 18 ]; only a few of these works employ policy theories and political concepts to guide data collection and make sense of their findings [ 19 ]. Similarly, contemporary literature that conceptualises KT as an umbrella term for both clinical and public policy decision-making, with calls for a generic ‘research-to-action’ [ 20 ], may fail to recognise the different types of actions required to change clinical practices and influence health policies. In many respects, such calls can even lead to a misconception that evidence-informed policymaking is simply a scaled-up version of evidence-based medicine [ 21 ].

In this study, we systematically review knowledge translation theories, models and frameworks (also known as KT TMFs) that were developed for health policies. Essentially, KT TMFs can be depicted as bridges that connect findings across diverse studies, as they establish a common language and standardise the measurement and assessment of desired policy changes [ 22 ]. This makes them essential for generalising implementation efforts and research findings [ 23 ]. While distinctions between a theory, a model or a framework are not always crystal-clear [ 24 ], the following definitions shed light on how they are interpreted in the context of KT. To start with, theory can be described as a set of analytical principles or statements crafted to structure our observations, enhance our understanding and explain the world [ 24 ]. Within implementation science, theories are encapsulated as either generalised models or frameworks. In other words, they are integrated into broader concepts, allowing researchers to form assumptions that help clarify phenomena and create hypotheses for testing [ 25 ].

Whereas theories in the KT literature are explanatory as well as descriptive, KT models are only descriptive with a more narrowly defined scope of explanation [ 24 ]; hence they have a more specific focus than theories [ 25 ]. KT models are created to facilitate the formulation of specific assumptions regarding a set of parameters or variables, which can subsequently be tested against outcomes using predetermined methods [ 25 ]. By offering simplified representations of complex situations, KT models can describe programme elements expected to produce desired results, or theoretical constructs believed to influence or moderate observed outcomes. In this way, they encompass theories related to change or explanation [ 22 ].

Lastly, frameworks in the KT language define a set of variables and the relations among them in a broad sense [ 25 ]. Frameworks, without the aim of providing explanations, solely describe empirical phenomena, representing a structure, overview, outline, system or plan consisting of various descriptive categories and the relations between them that are presumed to account for a phenomenon [ 24 ]. They portray loosely-structured constellations of theoretical constructs, without necessarily specifying their relationships; they can also offer practical methods for achieving implementation objectives [ 22 ]. Some scholars suggest sub-classifications and categorise a framework as ‘actionable’ if it has the potential to facilitate macro-level policy changes [ 11 ].

Context, which encompasses the entire environment in which policy decisions are made, is not peripheral but central to policymaking, playing a crucial role in its conceptualisation [ 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 ]. In the KT literature, the term ‘context’ is frequently employed, albeit often with a lack of precision [ 35 ]. It tends to serve as a broad term including various elements within a situation that are relevant to KT in some way but have not been explicitly identified [36]. However, there is a growing interest in delving deeper into what context refers to, as evidenced by increasing research attention [ 31 , 32 , 37 , 38 , 39 , 40 , 41 ]. While the definition of context in the transfer of knowledge to healthcare settings (i.e. implementing health policies, programmes or measures at the meso-level) has been systematically studied [ 36 , 37 , 42 , 43 ], the question of how KT scholars detail context in health policymaking remains unanswered. With our systematic scoping review, we aim to close this gap.

While KT TMFs, emerged from evidence-based medicine, have historically depicted the use of evidence from laboratories or healthcare organisations as the gold standard, we aimed to assess in this study whether and to what extent the evolving face of KT, addressing health policies, succeeded in foregrounding ‘context’. Our objective was thus not to evaluate the quality of these KT TMFs but rather to explore how scholars have incorporated contextual influences into their reasoning. We conducted a systematic scoping review to explore KT TMFs that are relevant to agenda-setting, policy formulation or policy adoption, in line with the aim of this study. Therefore, publications related to policy implementation in healthcare organisations or at the provincial level, as well as those addressing policy evaluation, did not meet our inclusion criteria. Consequently, given our focus on macro-level interventions, we excluded all articles that concentrate on translating clinical research into practice (meso-level interventions) and health knowledge to patients or citizens (micro-level interventions).

Prior systematic scoping reviews in the area of KT TMFs serve as a valuable foundation upon which to build further studies [ 44 , 45 ]. Using established methodologies may ensure a validated approach, allowing for a more nuanced understanding of KT TMFs in the context of existing scholarly work. Our review methodology employed a similar approach to that followed by Strifler et al. in 2018, who conducted a systematic scoping review of KT TMFs in the field of cancer prevention and management, as well as other chronic diseases [ 44 ]. Their search strategy was preferred over others for two primary reasons. First, Strifler et al. investigated KT TMFs altogether, systematically and comprehensively. Second, unlike many other review studies on KT, they focused on macro-level KT and included all relevant keywords useful for the purpose of our study in their Ovid/MEDLINE search query [ 44 ]. For our scoping review, we adapted their search query with the assistance of a specialist librarian. This process involved eliminating terms associated with cancer and chronic diseases, removing time limitation on the published papers, and including an additional language other than English due to authors’ proficiency in German. We included articles published in peer-reviewed journals until November 2022, excluding opinion papers, conference abstracts and study protocols, without any restriction on publication date or place. Our search query is presented in Table  1 .

Following a screening methodology similar to that employed by Votruba et al. [ 11 ], the first author conducted an initial screening of the titles and abstracts of 2918 unique citations. Full texts were selected and scrutinised if they appeared relevant to the topics of agenda-setting, policy formulation or policy adoption. Among these papers, the first author also identified those that conceptualised a KT TMF. Simultaneously, the last author independently screened 2918 titles and abstracts, randomly selecting 20% of them to identify studies related to macro-level KT. Regarding papers that conceptualised a KT TMF, all those initially selected by the first author underwent a thorough examination by the last author as well. In the papers reviewed by these two authors of this study, KT TMFs were typically presented as either Tables or Figures. In cases where these visual representations did not contain sufficient information about ‘context’, the main body of the study was carefully scrutinised by both reviewers to ensure no relevant information was missed. Any unclear cases were discussed and resolved to achieve 100% inter-rater agreement between the first and second reviewers. This strategy resulted in the inclusion of 32 relevant studies. The flow chart outlining our review process is provided in Fig.  1 .

Flow chart of the review process

According to the results of our systematic scoping review (Table  2 ), the first KT TMF developed for health policies dates back to 2003, confirming the emergence of a trend that expanded the meaning of the term Knowledge Translation to include policymakers as end-users of evidence during approximately the same period. In their study, Jacobson et al. [ 46 ] present a framework derived from a literature review to enhance understanding of user groups by organising existing knowledge, identifying gaps and emphasising the importance of learning about new contexts. However, despite acknowledging the significance of the user group context, the paper lacks a thorough explanation of the authors’ understanding of this term. The second study in our scoping review provides some details. Recognising a shift from evidence-based medicine to evidence-based health policymaking in the KT literature, the article by Dobrow et al. from 2004 [ 30 ] emphasises the importance of considering contextual factors. They present a conceptual framework for evidence-based decision-making, highlighting the influence of context in KT. Illustrated through examples from colorectal cancer screening policy development, their conceptual framework emphasises the significance of context in the introduction, interpretation and application of evidence. Third, Lehoux et al. [ 47 ] examine the field of Health Technology Assessment (HTA) and its role in informing decision and policymaking in Canada. By developing a conceptual framework for HTA dissemination and use, they touch on the institutional environment and briefly describe contextual factors.

Notably, the first three publications in our scoping review are authored by scholars affiliated with Canada, which is less of a coincidence, given the role of Canadian Institutes of Health Research (CIHR), the federal funding agency for health research: The CIHR Act (Bill C-13) mandates CIHR to ensure that the translation of health knowledge permeates every aspect of its work [ 48 ]. Moreover, it was CIHR that coined the term Knowledge Translation, defining KT as ‘a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically sound application of knowledge to improve health, provide more effective health services and products, and strengthen the health care system’ [ 49 ] . This comprehensive definition has since been adapted by international organisations (IOs), including WHO. The first document published by WHO that utilised KT to influence health policies dates back to 2005, entitled ‘Bridging the “know-do” gap: Meeting on knowledge translation in global health’, an initiative that was supported by the Canadian Coalition for Global Health Research, the Canadian International Development Agency, the German Agency for Technical Cooperation and the WHO Special Programme on Research and Training in Tropical Diseases [ 1 ]. Following this official recognition by WHO, studies in our scoping review after 2005 indicate a noticeable expansion of KT, encompassing a wider geographical area than Canada.

The article of Ashford et al. from 2006 [ 50 ] discusses the challenge of policy decisions in Kenya in the health field being disconnected from scientific evidence and presents a model for translating knowledge into policy actions through agenda-setting, coalition building and policy learning. However, the framework lacks explicit incorporation of contextual factors influencing health policies. Bauman et al. [ 51 ] propose a six-step framework for successful dissemination of physical activity evidence, illustrated through four case studies from three countries (Canada, USA and Brazil) and a global perspective. They interpret contextual factors as barriers and facilitators to physical activity and public health innovations. Focusing on the USA, Gold [ 52 ] explains factors, processes and actors that shape pathways between research and its use in a summary diagram, including a reference to ‘other influences in process’ for context. Green et al. [ 4 ] examine the gap between health research and its application in public health without focusing on a specific geographical area. Their study comprehensively reviews various concepts of diffusion, dissemination and implementation in public health, proposing ways to blend diffusion theory with other theories. Their ‘utilization-focused surveillance framework’ interprets context as social determinants as structures, economics, politics and culture.

Further, the article by Dhonukshe-Rutten et al. from 2010 [ 53 ] presents a general framework that outlines the process of translating nutritional requirements into policy applications from a European perspective. The framework incorporates scientific evidence, stakeholder interests and the socio-political context. The description of this socio-political context is rather brief, encompassing political and social priorities, legal context, ethical issues and economic implications. Ir et al. [ 54 ] analyse the use of knowledge in shaping policy on health equity funds in Cambodia, with the objective of understanding how KT contributes to the development of health policies that promote equity. Yet no information on context is available in the framework that they suggest. A notable exception among these early KT TMFs until 2010 is the conceptual framework for analysing integration of targeted health interventions into health systems by Atun et al. [ 55 ], in which the authors provide details about the factors that have an influence on the process of bringing evidence to health policies. Focusing on the adoption, diffusion and assimilation of health interventions, their conceptual framework provides a systematic approach for evaluating and informing policies in this field. Compared to the previous studies discussed above, their definition of context for this framework is comprehensive (Table  2 ). Overall, most of the studies containing macro-level KT TMFs published until 2010 either do not fully acknowledge contextual factors or provide generic terms such as cultural, political and economic for brief description (9 out of 10; 90%).

Studies published after 2010 demonstrate a notable geographical shift, with a greater emphasis on low- and middle-income countries (LMICs). By taking the adoption of the directly observed treatment, short-course (DOTS) strategy for tuberculosis control in Mexico as a case study, Bissell et al. [ 56 ] examine policy transfer to Mexico and its relevance to operational research efforts and suggest a model for analysis of health policy transfer. The model interprets context as health system, including political, economic, social, cultural and technological features. Focusing on HIV/AIDS in India, Tran et al. [ 57 ] explore KT by considering various forms of evidence beyond scientific evidence, such as best practices derived from programme experience and disseminated through personal communication. Their proposed framework aims to offer an analytical tool for understanding how evidence-based influence is exerted. In their framework, no information is available on context. Next, Bertone et al. [ 58 ] report on the effectiveness of Communities of Practice (CoPs) in African countries and present a conceptual framework for analysing and assessing transnational CoPs in health policy. The framework organises the key elements of CoPs, linking available resources, knowledge management activities, policy and practice changes, and improvements in health outcomes. Context is only briefly included in this framework.

Some other studies include both European and global perspectives. The publication from Timotijevic et al. from 2013 [ 59 ] introduces an epistemological framework that examines the considerations influencing the policy-making process, with a specific focus on micronutrient requirements in Europe. They present case studies from several European countries, highlighting the relevance of the framework in understanding the policy context related to micronutrients. Context is interpreted in this framework as global trends, data, media, broader consumer beliefs, ethical considerations, and wider social, legal, political, and economic environment. Next, funded by the European Union, the study by Onwujekwe et al. [ 60 ] examines the role of different types of evidence in health policy development in Nigeria. Although they cover the factors related to policy actors in their framework for assessing the role of evidence in policy development, they provide no information on context. Moreover, Redman et al. [ 61 ] present the SPIRIT Action Framework, which aims to enhance the use of research in policymaking. Context is interpreted in this framework as policy influences, i.e. public opinion, media, economic climate, legislative/policy infrastructure, political ideology and priorities, stakeholder interests, expert advice, and resources. From a global perspective, Spicer et al. [ 62 ] explore the contextual factors that influenced the scale-up of donor-funded maternal and newborn health innovations in Ethiopia, India and Nigeria, highlighting the importance of context in assessing and adapting innovations. Their suggested contextual factors influencing government decisions to accept, adopt and finance innovations at scale are relatively comprehensive (Table  2 ).

In terms of publication frequency, the pinnacle of reviewed KT studies was in 2017. Among six studies published in 2017, four lack details about context in their KT conceptualisations and one study touches on context very briefly. Bragge et al. [ 5 ] brought for their study an international terminology working group together to develop a simplified framework of interventions to integrate evidence into health practices, systems, and policies, named as the Aims, Ingredients, Mechanism, Delivery framework, albeit without providing details on contextual factors. Second, Mulvale et al. [ 63 ] present a conceptual framework that explores the impact of policy dialogues on policy development, illustrating how these dialogues can influence different stages of the policy cycle. Similar to the previous one, this study too, lacks information on context. In a systematic review, Sarkies et al. [ 64 ] evaluate the effectiveness of research implementation strategies in promoting evidence-informed policy decisions in healthcare. The study explores the factors associated with effective strategies and their inter-relationship, yet without further information on context. Fourth, Houngbo et al. [ 65 ] focus on the development of a strategy to implement a good governance model for health technology management in the public health sector, drawing from their experience in Benin. They outline a six-phase model that includes preparatory analysis, stakeholder identification and problem analysis, shared analysis and visioning, development of policy instruments for pilot testing, policy development and validation, and policy implementation and evaluation. They provide no information about context in their model. Fifth, Mwendera et al. [ 66 ] present a framework for improving the use of malaria research in policy development in Malawi, which was developed based on case studies exploring the policymaking process, the use of local malaria research, and assessing facilitators and barriers to research utilisation. Contextual setting is considered as Ministry of Health (MoH) with political set up, leadership system within the MoH, government policies and cultural set up. In contrast to these five studies, Ellen et al. [ 67 ] present a relatively comprehensive framework to support evidence-informed policymaking in ageing and health. The framework includes thought-provoking questions to discover contextual factors (Table  2 ).

Continuing the trend, studies published after 2017 focus increasingly on LMICs. In their embedded case study, Ongolo-Zogo et al. [ 68 ] examine the influence of two Knowledge Translation Platforms (KTPs) on policy decisions to achieve the health millennium development goals in Cameroon and Uganda. It explores how these KTPs influenced policy through interactions within policy issue networks, engagement with interest groups, and the promotion of evidence-supported ideas, ultimately shaping the overall policy climate for evidence-informed health system policymaking. Contextual factors are thereby interpreted as institutions (structures, legacies, policy networks), interests, ideas (values, research evidence) and external factors (reports, commitments). Focusing on the ‘Global South’, Plamondon et al. [ 69 ] suggest blending integrated knowledge translation with global health governance as an approach for strengthening leadership for health equity action. In terms of contextual factors, they include some information such as adapting knowledge to local context, consideration of the composition of non-traditional actors, such as civil society and private sector, in governance bodies and guidance for meaningful engagement between actors, particularly in shared governance models. Further, Vincenten et al. [ 70 ] propose a conceptual model to enhance understanding of interlinking factors that influence the evidence implementation process. Their evidence implementation model for public health systems refers to ‘context setting’, albeit without providing further detail.

Similarly, the study by Motani et al. from 2019 [ 71 ] assesses the outcomes and lessons learned from the EVIDENT partnership that focused on knowledge management for evidence-informed decision-making in nutrition and health in Africa. Although they mention ‘contextualising evidence’ in their conceptual framework, information about context is lacking. Focusing on Latin America and the Caribbean, Varallyay et al. [ 72 ] introduce a conceptual framework for evaluating embedded implementation research in various contexts. The framework outlines key stages of evidence-informed decision-making and provides guidance on assessing embeddedness and critical contextual factors. Compared to others, their conceptual framework provides a relatively comprehensive elaboration on contextual factors. In addition, among all the studies reviewed, Leonard et al. [ 73 ] present an exceptionally comprehensive analysis, where they identify the facilitators and barriers to the sustainable implementation of evidence-based health innovations in LMICs. Through a systematic literature review, they scrutinise 79 studies and categorise the identified barriers and facilitators into seven groups: context, innovation, relations and networks, institutions, knowledge, actors, and resources. The first one, context, contains rich information that could be seen in Table  2 .

Continuing from LMICs, Votruba et al. [ 74 ] present in their study the EVITA (EVIdence To Agenda setting) conceptual framework for mental health research-policy interrelationships in LMICs with some information about context, detailed as external influences and political context. In a follow-up study, they offer an updated framework for understanding evidence-based mental health policy agenda-setting [ 75 ]. In their revised framework, context is interpreted as external context and policy sphere, encompassing policy agenda, window of opportunity, political will and key individuals. Lastly, to develop a comprehensive monitoring and evaluation framework for evidence-to-policy networks, Kuchenmüller et al. [ 76 ] present the EVIPNet Europe Theory of Change and interpret contextual factors for evidence-informed policymaking as political, economic, logistic and administrative. Overall, it can be concluded that studies presenting macro-level KT TMFs from 2011 until 2022 focus mainly on LMICs (15 out of 22; close to 70%) and the majority of them were funded by international (development) organisations, the European Commission and global health donor agencies. An overwhelming number of studies among them (19 out of 22; close to 90%) provide either no information on contextual details or these were included only partly with some generic terms in KT TMFs.

Our systematic scoping review suggests that the approach of KT, which has evolved from evidence-based medicine to evidence-informed policymaking, tends to remain closely tied to its clinical origins when developing TMFs. In other words, macro-level KT TMFs place greater emphasis on the (public) health issue at hand rather than considering the broader decision-making context, a viewpoint shared by other scholars as well [ 30 ]. One reason could be that in the early stages of KT TMFs, the emphasis primarily focused on implementing evidence-based practices within clinical settings. At that time, the spotlight was mostly on content, including aspects like clinical studies, checklists and guidelines serving as the evidence base. In those meso-level KT TMFs, a detailed description of context, i.e. the overall environment in which these practices should be implemented, might have been deemed less necessary, given that healthcare organisations, such as hospitals to implement medical guidelines or surgical safety checklists, show similar characteristics globally.

However, as the scope of KT TMFs continues to expand to include the influence on health policies, a deeper understanding of context-specific factors within different jurisdictions and the dynamics of the policy process is becoming increasingly crucial. This is even more important for KT scholars aiming to conceptualise large-scale changes, as described in KT Tier 5, which necessitate a thorough understanding of targeted behaviours within societies. As the complexity of interventions increases due to the growing number of stakeholders either affecting or being affected by them, the interventions are surrounded by a more intricate web of attitudes, incentives, relationships, rules of engagement and spheres of influence [ 7 ]. The persisting emphasis on content over context in the evolving field of KT may oversimplify the complex process of using evidence in policymaking and understanding the society [ 77 ]. Some scholars argue that this common observation in public health can be attributed to the dominance of experts primarily from medical sciences [ 78 , 79 , 80 ]. Our study confirms the potential limitation of not incorporating insights from political science and public policy studies, which can lead to what is often termed a ‘naïve’ conceptualisation of evidence-to-policy schemes [ 15 , 16 , 17 ]. It is therefore strongly encouraged that the emerging macro-level KT concepts draw on political science and public administration if KT scholars intend to effectively communicate new ideas to policymakers, with the aim of prompting their action or response. We summarised our findings into three points.

Firstly, KT scholars may want to identify and pinpoint exactly where a change should occur within the policy process. The main confusion that we observed in the KT literature arises from a lack of understanding of how public policies are made. Notably, the term ‘evidence-informed policymaking’ can refer to any stage of the policy cycle, spanning from agenda-setting to policy formulation, adoption, implementation and evaluation. Understanding these steps will allow researchers to refine their language when advocating for policy changes across various jurisdictions; for instance, the word ‘implementation’ is often inappropriately used in KT literature. As commonly known, at the macro-level, public policies take the form of legislation, law-making and regulation, thereby shaping the practices or policies to be implemented at the meso- and micro-levels [ 81 ]. In other words, the process of using specific knowledge to influence health policies, however evidence-based it might be, falls mostly under the responsibility and jurisdiction of sovereign states. For this reason, macro-level KT TMFs should reflect the importance of understanding the policy context and the complexities associated with policymaking, rather than suggesting flawed or unrealistic top-down ‘implementation’ strategies in countries by foregrounding the content, or the (public) health issue at hand.

Our second observation from this systematic scoping review points towards a selective perception among researchers when reporting on policy interventions. Research on KT does not solely exist due to the perceived gap between scientific evidence and policy but also because of the pressures the organisations or researchers face in being accountable to their funding sources, ensuring the continuity of financial support for their activities and claiming output legitimacy to change public policies [ 8 ]. This situation indirectly compels researchers working to influence health policies in the field to provide ‘evidence-based’ feedback on the success of their projects to donors [ 82 ]. In doing so, researchers may overly emphasise the content of the policy intervention in their reporting to secure further funding, while they underemphasis the contextual factors. These factors, often perceived as a given, might actually be the primary facilitators of their success. Such a lack of transparency regarding the definition of context is particularly visible in the field of global health, where LMICs often rely on external donors. It is important to note that this statement is not intended as a negative critique of their missions or an evaluation of health outcomes in countries following such missions. Rather, it seeks to explain the underlying reason why researchers, particularly those reliant on donors in LMICs, prioritise promoting the concept of KT from a technical standpoint, giving less attention to contextual factors in their reasoning.

Lastly, and connected to the previous point, it is our observation that the majority of macro-level KT TMFs fail to give adequate consideration to both power dynamics in countries (internal vs. external influences) and the actual role that government plays in public policies. Notably, although good policymaking entails an honest effort to use the best available evidence, the belief that this will completely negate the role of power and politics in decision-making is a technocratic illusion [ 83 ]. Among the studies reviewed, the framework put forth by Leonard et al. [ 73 ] offers the most comprehensive understanding of context and includes a broad range of factors (such as political, social, and economic) discovered also in other reviewed studies. Moreover, the framework, developed through an extensive systematic review, offers a more in-depth exploration of these contextual factors than merely listing them as a set of keywords. Indeed, within the domains of political science and public policy, such factors shaping health policies have received considerable scholarly attention for decades. To define what context entails, Walt refers in her book ‘Health Policy: An Introduction to Process and Power’ [ 84 ] to the work of Leichter from 1979 [ 85 ], who provides a scheme for analysing public policy. This includes i) situational factors, which are transient, impermanent, or idiosyncratic; ii) structural factors, which are relatively unchanging elements of the society and polity; iii) cultural factors, which are value commitments of groups; and iv) environmental factors, which are events, structures and values that exist outside the boundaries of a political system and influence decisions within it. His detailed sub-categories for context can be found in Table  3 . This flexible public policy framework may offer KT researchers a valuable approach to understanding contextual factors and provide some guidance to define the keywords to focus on. Scholars can adapt this framework to suit a wide range of KT topics, creating more context-sensitive and comprehensive KT TMFs.

Admittedly, our study has certain limitations. Despite choosing one of the most comprehensive bibliographic databases for our systematic scoping review, which includes materials from biomedicine, allied health fields, biological and physical sciences, humanities, and information science in relation to medicine and healthcare, we acknowledge that we may have missed relevant articles indexed in other databases. Hence, exclusively using Ovid/MEDLINE due to resource constraints may have narrowed the scope and diversity of scholarly literature examined in this study. Second, our review was limited to peer-reviewed publications in English and German. Future studies could extend our findings by examining the extent to which contextual factors are detailed in macro-level KT TMFs published in grey literature and in different languages. Given the abundance of KT reports, working papers or policy briefs published by IOs and development agencies, such an endeavour could enrich our findings and either support or challenge our conclusions. Nonetheless, to our knowledge, this study represents the first systematic review and critical appraisal of emerging knowledge-to-policy concepts, also known as macro-level KT TMFs. It successfully blends insights from both biomedical and public policy disciplines, and could serve as a roadmap for future research.

The translation of knowledge to policymakers involves more than technical skills commonly associated with (bio-)medical sciences, such as creating evidence-based guidelines or clinical checklists. Instead, evidence-informed policymaking reflects an ambition to engage in the political dimensions of states. Therefore, the evolving KT concepts addressing health policies should be seen as a political decision-making process, rather than a purely analytical one, as is the case with evidence-based medicine. To better understand the influence of power dynamics and governance structures in policymaking, we suggest that future macro-level KT TMFs draw on insights from political science and public administration. Collaborative, interdisciplinary research initiatives could be undertaken to bridge the gap between these fields. Technocratic KT TMFs that overlook contextual factors risk propagating misconceptions in academic circles about how health policies are made, as they become increasingly influential over time. Research, the systematic pursuit of knowledge, is neither inherently good nor bad; it can be sought after, used or misused, like any other tool in policymaking. What is needed in the KT discourse is not another generic call for ‘research-to-action’ but rather an understanding of the dividing line between research-to- clinical -action and research-to- political -action.

Availability of data and materials

Available upon reasonable request.

WHO. Bridging the ‘Know-Do’ Gap: Meeting on Knowledge Translation in Global Health : 10–12 October 2005 World Health Organization Geneva, Switzerland [Internet]. 2005. https://www.measureevaluation.org/resources/training/capacity-building-resources/high-impact-research-training-curricula/bridging-the-know-do-gap.pdf

Rogers EM. Diffusion of innovations. 3rd ed. New York: Free Press; 1983.

Google Scholar  

Greenhalgh T, Wieringa S. Is it time to drop the ‘knowledge translation’ metaphor? A critical literature review. J R Soc Med. 2011;104(12):501–9.

Article   PubMed   PubMed Central   Google Scholar  

Green LW, Ottoson JM, García C, Hiatt RA. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009;30(1):151–74.

Article   PubMed   Google Scholar  

Bragge P, Grimshaw JM, Lokker C, Colquhoun H, Albrecht L, Baron J, et al. AIMD—a validated, simplified framework of interventions to promote and integrate evidence into health practices, systems, and policies. BMC Med Res Methodol. 2017;17(1):38.

Zarbin M. What Constitutes Translational Research? Implications for the Scope of Translational Vision Science and Technology. Transl Vis Sci Technol 2020;9(8).

Hassmiller Lich K, Frerichs L, Fishbein D, Bobashev G, Pentz MA. Translating research into prevention of high-risk behaviors in the presence of complex systems: definitions and systems frameworks. Transl Behav Med. 2016;6(1):17–31.

Tetroe JM, Graham ID, Foy R, Robinson N, Eccles MP, Wensing M, et al. Health research funding agencies’ support and promotion of knowledge translation: an international study. Milbank Q. 2008;86(1):125–55.

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1(1):1.

Article   PubMed Central   Google Scholar  

Rychetnik L, Bauman A, Laws R, King L, Rissel C, Nutbeam D, et al. Translating research for evidence-based public health: key concepts and future directions. J Epidemiol Community Health. 2012;66(12):1187–92.

Votruba N, Ziemann A, Grant J, Thornicroft G. A systematic review of frameworks for the interrelationships of mental health evidence and policy in low- and middle-income countries. Health Res Policy Syst. 2018;16(1):85.

Delnord M, Tille F, Abboud LA, Ivankovic D, Van Oyen H. How can we monitor the impact of national health information systems? Results from a scoping review. Eur J Public Health. 2020;30(4):648–59.

Malterud K, Bjelland AK, Elvbakken KT. Evidence-based medicine—an appropriate tool for evidence-based health policy? A case study from Norway. Health Res Policy Syst. 2016;14(1):15.

Borst RAJ, Kok MO, O’Shea AJ, Pokhrel S, Jones TH, Boaz A. Envisioning and shaping translation of knowledge into action: a comparative case-study of stakeholder engagement in the development of a European tobacco control tool. Health Policy. 2019;123(10):917–23.

Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy: a systematic review. PLoS ONE. 2013;8(10): e77404.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Cairney P. The politics of evidence-based policy making, 1st ed. London: Palgrave Macmillan UK: Imprint: Palgrave Pivot, Palgrave Macmillan; 2016.

Parkhurst J. The Politics of Evidence: From evidence-based policy to the good governance of evidence [Internet]. Routledge; 2016. https://www.taylorfrancis.com/books/9781315675008

Cairney P, Oliver K. Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy? Health Res Policy Syst. 2017;15(1):35.

Verboom B, Baumann A. Mapping the Qualitative Evidence Base on the Use of Research Evidence in Health Policy-Making: A Systematic Review. Int J Health Policy Manag. 2020;16.

Ward V, House A, Hamer S. Developing a framework for transferring knowledge into action: a thematic analysis of the literature. J Health Serv Res Policy. 2009;14(3):156–64.

Swinburn B, Gill T, Kumanyika S. Obesity prevention: a proposed framework for translating evidence into action. Obes Rev. 2005;6(1):23–33.

Article   CAS   PubMed   Google Scholar  

Damschroder LJ. Clarity out of chaos: Use of theory in implementation research. Psychiatry Res. 2020;283: 112461.

Birken SA, Rohweder CL, Powell BJ, Shea CM, Scott J, Leeman J, et al. T-CaST: an implementation theory comparison and selection tool. Implement Sci. 2018;13(1):143.

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

Rapport F, Clay-Williams R, Churruca K, Shih P, Hogden A, Braithwaite J. The struggle of translating science into action: foundational concepts of implementation science. J Eval Clin Pract. 2018;24(1):117–26.

Hagenaars LL, Jeurissen PPT, Klazinga NS. The taxation of unhealthy energy-dense foods (EDFs) and sugar-sweetened beverages (SSBs): An overview of patterns observed in the policy content and policy context of 13 case studies. Health Policy. 2017;121(8):887–94.

Sheikh K, Gilson L, Agyepong IA, Hanson K, Ssengooba F, Bennett S. Building the field of health policy and systems research: framing the questions. PLOS Med. 2011;8(8): e1001073.

Tran NT, Hyder AA, Kulanthayan S, Singh S, Umar RSR. Engaging policy makers in road safety research in Malaysia: a theoretical and contextual analysis. Health Policy. 2009;90(1):58–65.

Walt G, Gilson L. Reforming the health sector in developing countries: the central role of policy analysis. Health Policy Plan. 1994;9(4):353–70.

Dobrow MJ, Goel V, Upshur REG. Evidence-based health policy: context and utilisation. Soc Sci Med. 2004;58(1):207–17.

Barnfield A, Savolainen N, Lounamaa A. Health Promotion Interventions: Lessons from the Transfer of Good Practices in CHRODIS-PLUS. Int J Environ Res Public Health. 2020;17(4).

van de Goor I, Hämäläinen RM, Syed A, Juel Lau C, Sandu P, Spitters H, et al. Determinants of evidence use in public health policy making: results from a study across six EU countries. Health Policy Amst Neth. 2017;121(3):273–81.

Article   Google Scholar  

Ornstein JT, Hammond RA, Padek M, Mazzucca S, Brownson RC. Rugged landscapes: complexity and implementation science. Implement Sci. 2020;15(1):85.

Seward N, Hanlon C, Hinrichs-Kraples S, Lund C, Murdoch J, Taylor Salisbury T, et al. A guide to systems-level, participatory, theory-informed implementation research in global health. BMJ Glob Health. 2021;6(12): e005365.

Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implement Sci. 2017;12(1):21.

Rogers L, De Brún A, McAuliffe E. Defining and assessing context in healthcare implementation studies: a systematic review. BMC Health Serv Res. 2020;20(1):591.

Nilsen P, Bernhardsson S. Context matters in implementation science: a scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Serv Res. 2019;19(1):189.

Arksey H, O’Malley L, Baldwin S, Harris J, Mason A, Golder S. Literature review report: services to support carers of people with mental health problems. 2002;182.

Tabak RG, Khoong EC, Chambers D, Brownson RC. Bridging research and practice. Am J Prev Med. 2012;43(3):337–50.

O’Donovan MA, McCallion P, McCarron M, Lynch L, Mannan H, Byrne E. A narrative synthesis scoping review of life course domains within health service utilisation frameworks. HRB Open Res. 2019.

Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A, et al. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.

Bate P, Robert G, Fulop N, Øvretviet J, Dixon-Woods M. Perspectives on context: a collection of essays considering the role of context in successful quality improvement [Internet]. 2014. https://www.health.org.uk/sites/default/files/PerspectivesOnContext_fullversion.pdf

Ziemann A, Brown L, Sadler E, Ocloo J, Boaz A, Sandall J. Influence of external contextual factors on the implementation of health and social care interventions into practice within or across countries—a protocol for a ‘best fit’ framework synthesis. Syst Rev. 2019. https://doi.org/10.1186/s13643-019-1180-8 .

Strifler L, Cardoso R, McGowan J, Cogo E, Nincic V, Khan PA, et al. Scoping review identifies significant number of knowledge translation theories, models, and frameworks with limited use. J Clin Epidemiol. 2018;100:92–102.

Esmail R, Hanson HM, Holroyd-Leduc J, Brown S, Strifler L, Straus SE, et al. A scoping review of full-spectrum knowledge translation theories, models, and frameworks. Implement Sci. 2020;15(1):11.

Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy. 2003;8(2):94–9.

Lehoux P, Denis JL, Tailliez S, Hivon M. Dissemination of health technology assessments: identifying the visions guiding an evolving policy innovation in Canada. J Health Polit Policy Law. 2005;30(4):603–42.

Parliament of Canada. Government Bill (House of Commons) C-13 (36–2) - Royal Assent - Canadian Institutes of Health Research Act [Internet]. https://parl.ca/DocumentViewer/en/36-2/bill/C-13/royal-assent/page-31 . Accessed 1 Apr 2023.

Straus SE, Tetroe J, Graham I. Defining knowledge translation. CMAJ Can Med Assoc J. 2009;181(3–4):165–8.

Ashford L. Creating windows of opportunity for policy change: incorporating evidence into decentralized planning in Kenya. Bull World Health Organ. 2006;84(8):669–72.

Bauman AE, Nelson DE, Pratt M, Matsudo V, Schoeppe S. Dissemination of physical activity evidence, programs, policies, and surveillance in the international public health arena. Am J Prev Med. 2006;31(4):57–65.

Gold M. Pathways to the use of health services research in policy. Health Serv Res. 2009;44(4):1111–36.

Dhonukshe-Rutten RAM, Timotijevic L, Cavelaars AEJM, Raats MM, de Wit LS, Doets EL, et al. European micronutrient recommendations aligned: a general framework developed by EURRECA. Eur J Clin Nutr. 2010;64(2):S2-10.

Ir P, Bigdeli M, Meessen B, Van Damme W. Translating knowledge into policy and action to promote health equity: The Health Equity Fund policy process in Cambodia 2000–2008. Health Policy. 2010;96(3):200–9.

Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O. Integration of targeted health interventions into health systems: a conceptual framework for analysis. Health Policy Plan. 2010;25(2):104–11.

Bissell K, Lee K, Freeman R. Analysing policy transfer: perspectives for operational research. Int J Tuberc Lung Dis Off J Int Union Tuberc Lung Dis. 2011;15(9).

Tran NT, Bennett SC, Bishnu R, Singh S. Analyzing the sources and nature of influence: how the Avahan program used evidence to influence HIV/AIDS prevention policy in India. Implement Sci. 2013;8(1):44.

Bertone MP, Meessen B, Clarysse G, Hercot D, Kelley A, Kafando Y, et al. Assessing communities of practice in health policy: a conceptual framework as a first step towards empirical research. Health Res Policy Syst. 2013;11(1):39.

Timotijevic L, Brown KA, Lähteenmäki L, de Wit L, Sonne AM, Ruprich J, et al. EURRECA—a framework for considering evidence in public health nutrition policy development. Crit Rev Food Sci Nutr. 2013;53(10):1124–34.

Onwujekwe O, Uguru N, Russo G, Etiaba E, Mbachu C, Mirzoev T, et al. Role and use of evidence in policymaking: an analysis of case studies from the health sector in Nigeria. Health Res Policy Syst. 2015;13(1):46.

Redman S, Turner T, Davies H, Williamson A, Haynes A, Brennan S, et al. The SPIRIT action framework: a structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015;136–137:147–55.

Spicer N, Berhanu D, Bhattacharya D, Tilley-Gyado RD, Gautham M, Schellenberg J, et al. ‘The stars seem aligned’: a qualitative study to understand the effects of context on scale-up of maternal and newborn health innovations in Ethiopia, India and Nigeria. Glob Health. 2016;12(1):75.

Mulvale G, McRae SA, Milicic S. Teasing apart “the tangled web” of influence of policy dialogues: lessons from a case study of dialogues about healthcare reform options for Canada. Implement Sci IS. 2017;12.

Sarkies MN, Bowles KA, Skinner EH, Haas R, Lane H, Haines TP. The effectiveness of research implementation strategies for promoting evidence-informed policy and management decisions in healthcare: a systematic review. Implement Sci. 2017;12(1):132.

Houngbo PTh, Coleman HLS, Zweekhorst M, De Cock Buning TJ, Medenou D, Bunders JFG. A Model for Good Governance of Healthcare Technology Management in the Public Sector: Learning from Evidence-Informed Policy Development and Implementation in Benin. PLoS ONE. 2017;12(1):e0168842.

Mwendera C, de Jager C, Longwe H, Hongoro C, Phiri K, Mutero CM. Development of a framework to improve the utilisation of malaria research for policy development in Malawi. Health Res Policy Syst. 2017;15(1):97.

Ellen ME, Panisset U, de AraujoCarvalho I, Goodwin J, Beard J. A knowledge translation framework on ageing and health. Health Policy. 2017;121(3):282–91.

Ongolo-Zogo P, Lavis JN, Tomson G, Sewankambo NK. Assessing the influence of knowledge translation platforms on health system policy processes to achieve the health millennium development goals in Cameroon and Uganda: a comparative case study. Health Policy Plan. 2018;33(4):539–54.

Plamondon KM, Pemberton J. Blending integrated knowledge translation with global health governance: an approach for advancing action on a wicked problem. Health Res Policy Syst. 2019;17(1):24.

Vincenten J, MacKay JM, Schröder-Bäck P, Schloemer T, Brand H. Factors influencing implementation of evidence-based interventions in public health systems—a model. Cent Eur J Public Health. 2019;27(3):198–203.

Motani P, Van de Walle A, Aryeetey R, Verstraeten R. Lessons learned from Evidence-Informed Decision-Making in Nutrition & Health (EVIDENT) in Africa: a project evaluation. Health Res Policy Syst. 2019;17(1):12.

Varallyay NI, Langlois EV, Tran N, Elias V, Reveiz L. Health system decision-makers at the helm of implementation research: development of a framework to evaluate the processes and effectiveness of embedded approaches. Health Res Policy Syst. 2020;18(1):64.

Leonard E, de Kock I, Bam W. Barriers and facilitators to implementing evidence-based health innovations in low- and middle-income countries: a systematic literature review. Eval Program Plann. 2020;82: 101832.

Votruba N, Grant J, Thornicroft G. The EVITA framework for evidence-based mental health policy agenda setting in low- and middle-income countries. Health Policy Plan. 2020;35(4):424–39.

Votruba N, Grant J, Thornicroft G. EVITA 2.0, an updated framework for understanding evidence-based mental health policy agenda-setting: tested and informed by key informant interviews in a multilevel comparative case study. Health Res Policy Syst. 2021;19(1):35.

Kuchenmüller T, Chapman E, Takahashi R, Lester L, Reinap M, Ellen M, et al. A comprehensive monitoring and evaluation framework for evidence to policy networks. Eval Program Plann. 2022;91: 102053.

Ettelt S. The politics of evidence use in health policy making in Germany—the case of regulating hospital minimum volumes. J Health Polit Policy Law. 2017;42(3):513–38.

Greer SL, Bekker M, de Leeuw E, Wismar M, Helderman JK, Ribeiro S, et al. Policy, politics and public health. Eur J Public Health. 2017;27(suppl 4):40–3.

Fafard P, Cassola A. Public health and political science: challenges and opportunities for a productive partnership. Public Health. 2020;186:107–9.

Löblová O. Epistemic communities and experts in health policy-making. Eur J Public Health. 2018;28(suppl 3):7–10.

Maddalena V. Evidence-Based Decision-Making 8: Health Policy, a Primer for Researchers. In: Parfrey PS, Barrett BJ, editors. Clinical Epidemiology: Practice and Methods. New York, NY: Springer; 2015. (Methods in Molecular Biology).

Louis M, Maertens L. Why international organizations hate politics - Depoliticizing the world [Internet]. London and New York: Routledge; 2021. (Global Institutions). https://library.oapen.org/bitstream/handle/20.500.12657/47578/1/9780429883279.pdf

Hassel A, Wegrich K. How to do public policy. 1st ed. Oxford: Oxford University Press; 2022.

Book   Google Scholar  

Walt G. Health policy: an introduction to process and power. 7th ed. Johannesburg: Witwatersrand University Press; 2004.

Leichter HM. A comparative approach to policy analysis: health care policy in four nations. Cambridge: Cambridge University Press; 1979.

Download references

Acknowledgements

Not applicable.

Author information

Authors and affiliations.

Department of International Health, Care and Public Health Research Institute – CAPHRI, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands

Tugce Schmitt, Katarzyna Czabanowska & Peter Schröder-Bäck

You can also search for this author in PubMed   Google Scholar

Contributions

TS: Conceptualization, Methodology, Formal analysis, Investigation, Writing—Original Draft; KC: Writing—Review & Editing; PSB: Validation, Formal analysis, Writing—Review & Editing, Supervision.

Corresponding author

Correspondence to Tugce Schmitt .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Schmitt, T., Czabanowska, K. & Schröder-Bäck, P. What is context in knowledge translation? Results of a systematic scoping review. Health Res Policy Sys 22 , 52 (2024). https://doi.org/10.1186/s12961-024-01143-5

Download citation

Received : 26 June 2023

Accepted : 11 April 2024

Published : 29 April 2024

DOI : https://doi.org/10.1186/s12961-024-01143-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Knowledge Translation
• Evidence-informed policymaking
• Health systems

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 26 April 2024

Levels of autonomy in FDA-cleared surgical robots: a systematic review

• Audrey Lee   ORCID: orcid.org/0000-0002-4830-5781 1 , 2 ,
• Turner S. Baker 1 , 2 , 3 ,
• Joshua B. Bederson 1 , 2 &
• Benjamin I. Rapoport   ORCID: orcid.org/0000-0002-0049-6419 1 , 2  

npj Digital Medicine volume  7 , Article number:  103 ( 2024 ) Cite this article

491 Accesses

19 Altmetric

Metrics details

• Biotechnology
• Health care

The integration of robotics in surgery has increased over the past decade, and advances in the autonomous capabilities of surgical robots have paralleled that of assistive and industrial robots. However, classification and regulatory frameworks have not kept pace with the increasing autonomy of surgical robots. There is a need to modernize our classification to understand technological trends and prepare to regulate and streamline surgical practice around these robotic systems. We present a systematic review of all surgical robots cleared by the United States Food and Drug Administration (FDA) from 2015 to 2023, utilizing a classification system that we call Levels of Autonomy in Surgical Robotics (LASR) to categorize each robot’s decision-making and action-taking abilities from Level 1 (Robot Assistance) to Level 5 (Full Autonomy). We searched the 510(k), De Novo, and AccessGUDID databases in December 2023 and included all medical devices fitting our definition of a surgical robot. 37,981 records were screened to identify 49 surgical robots. Most surgical robots were at Level 1 (86%) and some reached Level 3 (Conditional Autonomy) (6%). 2 surgical robots were recognized by the FDA to have machine learning-enabled capabilities, while more were reported to have these capabilities in their marketing materials. Most surgical robots were introduced via the 510(k) pathway, but a growing number via the De Novo pathway. This review highlights trends toward greater autonomy in surgical robotics. Implementing regulatory frameworks that acknowledge varying levels of autonomy in surgical robots may help ensure their safe and effective integration into surgical practice.

Similar content being viewed by others

The IDEAL framework for surgical robotics: development, comparative evaluation and long-term monitoring

The rise of robots in surgical environments during COVID-19

Next-generation robotics in gastrointestinal surgery

Introduction.

When the first surgical robots entered medical practice, they were truly robotically-assisted systems with no independent decision-making and action-taking abilities 1 . Yet, in the popular imagination, there has always been an anticipated future with autonomous systems performing complex procedures with minimal surgeon intervention. Today, advancements in automation and the growth of artificial intelligence and machine learning have brought this imagined future closer to reality. Adjacent fields such as assistive robotics and industrial robotics are already seeing examples of robots with increasing autonomy that can work with and around humans. While the same degree of autonomy is not yet available in surgery, it is a perceived inevitability that requires careful planning.

The surgical robotics field has changed significantly since the Automated Endoscopic System for Optimal Positioning (AESOP) became the first FDA-cleared surgical robot in 1993. The FDA cleared AESOP through the 510(k) Premarket Notification pathway as a Class II (moderate risk) device, which set a precedent for the regulatory evaluation of surgical robots 1 . Since then, surgical robots have progressed from minor supporting roles to more complex autonomous systems. For instance, robots in current surgical practice range from leader-follower systems like the da Vinci Surgical System (Intuitive Surgical, USA), where the robot does not perform tasks automatically but is entirely controlled by the surgeon, to systems like the TSolution One (Think Surgical, USA), where the robot generates patient-specific operative plans and automatically performs bone milling while the surgeon watches. However, the taxonomic tools to describe and regulate these robotic systems have remained static and narrow.

The prevailing classification of surgical robots utilizes organizing frameworks and definitions that carry legacy constructs from industrial robotics and autonomous motor vehicles and do not do justice to surgical robotics today. We provide commonly used definitions and standards for machinery used in surgery in Supplementary Table 1 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 . For example, the IEC/TR 60601-4-1 Medical Electrical Equipment Technical Report defines a taxonomy for degrees of autonomy in medical robotics. However, it focuses solely on technical metrics while failing to discuss practical developmental benchmarks and human-robot interactions critical for ensuring patient and surgeon safety as the field advances. Since 2015, the FDA has advocated for the term “robotically-assisted surgical devices” instead of “surgical robots” to emphasize that all cleared systems have no robotic autonomy, as they require the surgeon’s direct and continuous control to move and activate surgical instruments 10 , 11 , 12 . By this definition, the surgeon is entirely responsible for the safety of the procedure and is expected to maintain proper training across different models of robotically-assisted surgical devices 2 , 12 .

The growing integration of automation and machine learning into patient-specific surgical planning and task execution now challenges the presumption that surgical robots lack autonomy. This development also makes it increasingly difficult to regulate and streamline surgical workflows around such technologies. The lack of adequate tools to capture these trends toward increasing robotic autonomy complicates the roles and responsibilities of surgeons and manufacturers and raises many potential legal and ethical considerations. For instance, questions such as who is legally responsible for procedural safety as surgical tasks are increasingly automated, what additional technical competencies would surgeon training programs require, and who ensures that the machine learning models in these systems continue to perform adequately over time highlight the need for further clarity on regulatory paradigms. For surgical robots to live up to their technological potential, standards organizations, regulatory agencies, and medical societies need a unifying framework tailored to surgical robotics. Such a framework would enable the development of regulatory standards and surgical practice parameters to provide reasonable assurance of the safety and effectiveness of modern surgical robots.

We analyzed all Class II-risked surgical robots cleared by the FDA since 2015 through the lens of a Levels of Autonomy in Surgical Robotics (LASR) taxonomy to identify trends in the regulatory process for surgical robotics and automation. This review is intended to highlight key considerations for the development of regulation and surgical practice parameters around increasingly autonomous robotic systems.

After duplicate removal, we manually screened 37,981 database records, from which we reviewed 6445 full-text reports. 1620 reports were grouped to identify unique surgical robots (Fig. 1 ). Each surgical robot was then classified using the LASR scale (Fig. 2 ).

Study selection process.

Characteristics of each level of autonomy.

We identified 49 unique surgical robots with our search strategy. We considered most robotic systems as Level 1 (Robot Assistance) (42 systems [86%]) which operate under continuous surgeon control (Fig. 3a ). We also considered 4 systems (8%) as Level 2 (Task Autonomy) surgical robots that could execute preprogrammed, automated actions for a specific surgical task. The most advanced surgical robots cleared by the FDA reached Level 3 (Conditional Autonomy) (3 systems [6%]), which could generate patient-specific strategies for a surgical procedure. There were no examples of Level 4 and Level 5 surgical robots.

a Current total surgical robots. b Cumulative count of new surgical robots over time by the year of first FDA clearance. c Cumulative count of FDA regulatory pathway taken by new surgical robots over time by the year of first FDA clearance.

After the surgical robots were organized by the year of their first FDA clearance, we observed that 15 robotic systems were first cleared prior to 2015 but obtained addendum clearances for their expanded capabilities within our search period (Fig. 3b ). 34 entirely new surgical robots were introduced for Level 1 through Level 3 since 2015. Since 2017, there has been a gradual shift towards increased task automation with the introduction of new Level 2 surgical robots. Only one additional Level 3 surgical robot was cleared by the FDA in 2015. Most of the surgical robots were cleared through the FDA’s 510(k) pathway (44 systems [90%]). A smaller but growing number of systems were introduced through the De Novo pathway (5 systems [10%]) (Fig. 3c ). Only 19 of the surgical robots (39%) had accompanying clinical testing data. These included all 3 of the Level 3 surgical robots (100%), 3 of the Level 2 surgical robots (75%), and 13 of the Level 1 systems (31%). 2 of the surgical robots were reported to have machine learning-enabled software features in their submissions to the FDA. However, 3 additional surgical robots were marketed to have machine learning-enabled capabilities on their product websites that were not in their FDA summary documents.

Nearly all robotic systems were designed to accommodate a variety of procedures across different specialties (73%) (Fig. 4a ). Orthopedic surgery was the fastest-growing specialty in surgical robotics, with 33% of all new robotic systems introduced since 2015 intended for spine, knee, and hip surgeries (Fig. 4b ). The number of surgical robots for urology also expanded with 11 new or improved robotic systems, followed by general surgery (10 systems), thoracic surgery (9 systems), and neurosurgery (9 systems). There were fewer robotic systems for other specialties, including otolaryngology (ENT)/head and neck surgery, interventional radiology, and plastic surgery.

a By specialty. b By subspecialty.

Most specialties have only Level 1 surgical robots (Fig. 5 ). The three most advanced Level 3 robotic systems were intended for autonomously generating and executing patient-specific plans for bone milling in orthopedic surgery, prostate biopsy in urology, and hair follicle extraction in plastic surgery. Orthopedic surgery, urology, general surgery, gynecology, and interventional radiology were the only specialties with Level 2 surgical robots.

Organized by surgical specialty.

An abbreviated summary of the Level 2 and Level 3 robotic systems is provided in Table 1 . All surgical robots and their data collected in this study are presented in Supplementary Data 1 .

Since 2015, the FDA has cleared nearly 50 surgical robots, including new systems and existing systems with expanded capabilities. Research developments at the intersection of automation, machine learning, and robotics continue to advance the levels of autonomy embodied by surgical robots and challenge traditional paradigms. However, the current frameworks for classifying and regulating these robotic systems have not kept pace with these advancements, as they emphasize only technical capabilities and are not specific to surgical robots. There is a need for a clear framework that captures the roles of surgeons during procedures with these systems while allowing room for growth as the field evolves. Such a framework would help to establish a common baseline to develop regulatory standards and practice parameters that promote procedural safety and liability management.

We utilized a Levels of Autonomy in Surgical Robotics (LASR) classification system to capture the technological and regulatory trends in surgical robotics involving automation and robotic autonomy. By categorizing robotic systems with LASR, we can appreciate that the field has progressed with features of higher levels of robotic autonomy, like patient-specific surgical plan generation and task automation. We observed that the most advanced surgical robots cleared for clinical use reached Level 3 capabilities. LASR may help guide the development of more focused regulatory standards and practice parameters for surgical robotics.

The FDA currently regulates all surgical robots as Class II (moderate risk) devices, which follows the precedent established by the clearance of prior Level 1 robotic systems 11 . The FDA considered most surgical robots through the 510(k) Premarket Notification pathway, which requires a demonstration of substantial equivalence to a legally marketed device or “predicate” through non-clinical testing 14 . Five robotic systems—the AquaBeam Robotic System (Procept BioRobotics, USA), Anovo Surgical System (Momentis Surgical, Israel), MARS Surgical System (Levita Magnetics, USA), Iotasoft Insertion System (iotaMotion, USA), Galen ES (Galen Robotics, USA)—were introduced via the De Novo pathway. Devices progressing through the De Novo pathway do not have predicates and instead undergo a risk-based classification. If the FDA grants the De Novo request for a device, the device may serve as a predicate for future iterations to pursue the 510(k) pathway 15 . Current Level 2 and Level 3 surgical robots mitigate risk by requiring the surgeon to review and decide what automated actions or strategies the robotic system should execute. While this approach of placing the burden of responsibility on the surgeon as the ultimate decision-maker may be adequate for the clearance of current surgical robots, it may not be sufficient for future robotic systems with higher levels of autonomy. Practical developmental benchmarks based on LASR are needed to delineate transitions in levels of autonomy and recognize diverse modes of human-robot interactions in surgical robotics. Predicate creep—a repetitive cycle of technology changes between 510(k) clearances that may result in the sudden introduction of devices with high levels of complexity—has been identified in surgical robotics 16 , 17 . To ensure that future robotic systems with higher levels of autonomy are appropriately introduced with actual clinical evidence of safety, organizing frameworks that expand on LASR may be used to guide the definition of substantial equivalence requirements for more advanced surgical robots. Alternatively, there has been speculation that future Level 4 and Level 5 surgical robots may be deemed Class III (high risk) devices that require Premarket Approval (PMA) 18 . PMA is the most stringent regulatory pathway for medical devices that often involve new concepts not found in existing devices. This pathway may ensure rigorous evaluation of these technologies, thereby increasing confidence in their safety and effectiveness. However, it may simultaneously create bottlenecks in the innovation of Level 4 and Level 5 surgical robots. The regulatory approval of Class III devices often requires significantly more time and cost investment than that of Class II devices 5 , 14 , 18 . As advancements in surgical robotics are primarily driven by key players in the market, this can potentially impose barriers to entry for newer and smaller surgical robotics companies. Medical specialties with comparatively smaller device markets may fall further behind in surgical robotics innovation. Moreover, there is concern that surgical robots at these levels may be considered systems that practice medicine because of their independent decision-making capabilities 18 . Since regulating the practice of medicine is more in the realm of medical societies rather than the FDA, these societies need to work alongside regulatory agencies and engineers to determine how to evaluate these surgical robots.

At present, most instances of surgical robot autonomy involve closed-loop control paradigms. However, a growing number of surgical robot manufacturers reported machine learning-enabled software features in their systems. Consistent with the FDA’s publicly available list of artificial intelligence and machine learning-enabled devices 19 , we identified two surgical robots—the Senhance Surgical System (Asensus Surgical, USA) and Cirq Robotics (Brainlab, Germany)—with reported machine learning-enabled capabilities. While these capabilities are currently limited to aiding in patient registration, automated endoscope camera control, and digital 3D measurements, it is evident that machine learning plays a growing role in advancing robotic autonomy. In addition, we identified three surgical robots—the Artas iX System (Venus Concept, Canada), CORI Surgical System (Smith and Nephew, USA), Artemis (Eigen, USA)—marketed to have artificial intelligence and machine learning-enabled capabilities not mentioned in their FDA summary documents. While it is beyond the scope of this review to understand this discrepancy, it underscores the need for careful consideration of how surgical robot software components, particularly those integrating machine learning, are evaluated along with the system’s hardware components. Machine learning algorithms may propagate biases that exist within training data 20 . When these biases are not rectified, they may lead to unintended consequences. In the context of surgical robots, this could pose a significant risk, especially if these machine learning algorithms are part of automated tasks. Questions such as “Should clinical data be required in these cases?” and “Is it safe to declare purely software medical devices or non-machine learning-enabled surgical robots as predicates?” must be clarified. Hence, machine learning as a means of advancing surgical robot autonomy may need to be scrutinized more, if not the same, as other means of robot automation.

Technological progress in surgical robotics requires the parallel evolution of regulatory frameworks and surgical practice workflows specific to each level of autonomy. Recognizing the different levels of autonomy of surgical robots is essential to their safe and effective integration into surgical practice. Nevertheless, levels of autonomy do not provide an all-encompassing classification of risk, as risk is ultimately procedure-specific. Future work can address these limitations by incorporating surgical context into the concept of levels of autonomy 21 . While the regulation of increasingly autonomous surgical robots is yet to be determined, establishing practice parameters for each level of autonomy will be needed regardless.

Surgical robotics is evolving with new modes of surgeon-robot interactions, the integration of machine learning, and the potential for higher levels of robotic autonomy. How surgical robots are categorized and regulated must keep up with these technological advancements. The Levels of Autonomy in Surgical Robotics scale helped to reflect the modern state of the field with an understanding that increasing robotic autonomy is seemingly inevitable given progress in adjacent fields. Recognizing these trends in regulatory frameworks will be essential to ensuring patient and surgeon safety in this developing area of medical technology.

This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guidelines 22 .

Data collection

All records from the FDA 510(k) and De Novo databases with a decision date since January 1, 2015 were downloaded. Duplicate records by 510(k) Number and De Novo Number were automatically removed. AccessGUDID (Global Unique Device Identification Database) records were also collected using the online portal system with search query ((robot*) AND (surg*)) OR ((robot*) AND (intervention*)). No restriction on time was applied to the AccessGUDID search, as it was not a filter option. Duplicate AccessGUDID records by Public Device Record Key were automatically removed. Record collection for all databases occurred on March 2, 2023 and on December 11, 2023 using the same search method.

Eligibility criteria and screening process

One researcher (AL) independently and manually reviewed the database fields of all records and performed duplicate checking of all records on separate days. We excluded records with Device Classification Names (510(k) and De Novo), and Global Medical Device Nomenclature Terms (AccessGUDID) that were not directly related to electromechanical systems used for surgery. Examples of names and terms that we deemed unrelated were “polymer patient examination glove”, “wheelchair, powered”, and “general surgical procedure kit”. Examples of names and terms that we deemed related included “system, surgical, computer-controlled instrument”, “robotic surgical arm system”, and “robotic surgical navigation system”.

Reports—the full-text summary and statement documents accompanying submissions to the FDA, or device-specific pages on AccessGUDID—were retrieved and manually assessed by one researcher (AL) alongside manufacturer and distributor websites to identify eligible studies or surgical robots. We defined “surgical robot” according to the International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC) definitions in ISO 8373, IEC 80601-2-77, and IEC 60601-4-1 as medical electrical equipment with a degree of autonomy that incorporates a computer-controlled electromechanical component intended to actuate, position, orient, or manipulate a surgical instrument—an invasive device with an applied part that could administer energy or invade into the patient’s body through an incision on the skin or inner surface of a natural orifice 8 . By this definition, we excluded robotic radiation therapy systems and robotic high-intensity focused ultrasound systems which do not require incisions, and surgical navigation systems that lack a computer-controlled electromechanical component. We also excluded stand-alone surgical instruments and software devices that were not intended for use with a surgical robot. Conversely, we included surgical instruments, software, and accessories that were intended for use with a surgical robot.

Since the FDA requires a new 510(k) submission for changes or modifications to existing devices and AccessGUDID documents all versions or models of devices, there were multiple reports corresponding to each unique surgical robot. One review author (AL) manually assessed all eligible reports alongside corresponding manufacturer and distributor websites to identify unique devices ( Supplementary Methods ). Older-generation surgical robots from the same company were grouped with their newest versions.

Levels of Autonomy in Surgical Robotics (LASR) taxonomy

In concordance with prior research and standardization efforts, we defined a Levels of Autonomy in Surgical Robotics (LASR) scale that clarifies the division of roles between surgeons and robotic systems during surgery 2 , 3 , 5 , 8 , 9 , 18 , 23 , 24 . Specifically, we used concepts from the framework on levels of autonomy for medical robotics originally proposed by Yang et al. 18 and further tailored to surgical robotics by Fosch-Villaronga et al. 23 ., Haidegger et al. 5 , 24 , and Attanasio et al. 2 . based on the emerging ISO and IEC standards 3 , 8 , 9 . Building upon these, we introduced additional clarifications informed by surgeon feedback to define the division of roles between surgeons and robots during surgery and human-robot interactions. Thus, the LASR taxonomy classifies each surgical robot by its highest level of autonomy capabilities from 0 (No Autonomy) to 5 (Full Autonomy) (Fig. 2 ).

Level 0—no autonomy

Devices without robotic equipment. The surgeon generates, selects, executes, and monitors all surgical actions, and the device provides no aid in such actions. Surgeries performed with these devices are considered identical to non-robotic manual cases.

By our definition of a “surgical robot”, we excluded Level 0 devices in this review.

Level 1—robot assistance

Surgical robots that require the surgeon to control all movements of the system and activation of its surgical instruments directly and continuously. The surgeon generates, selects, executes, and monitors all surgical actions, and the surgical robot aids the surgeon in the execution and monitoring of such actions either with passive support or active guidance. In passive support, the surgeon maintains a free range of motion while the surgical robot provides minor assistance that does not grossly interfere with the surgeon’s intended motion trajectories. Examples include teleoperation, tremor filtration, and tool tracking. In active guidance, the surgical robot provides mechanical support such as haptic feedback or motion constraints to influence the surgeon’s physical actions.

Level 2—task autonomy

Surgical robots that can execute and monitor preprogrammed, automated actions for a specific task when selected by the surgeon, without requiring the surgeon’s continuous direct control over the movements and instrument activations. The surgical robot is unable to independently define parameters to generate plans, so the surgeon needs to provide the information required to perform the action. These actions are predictable and designed to reduce variability across procedures. The preprogrammed actions may automate either a few discrete surgical gestures—the smallest meaningful interaction of a surgical instrument with human tissue—or the complete task, which involves a coordinated sequence of multiple surgical gestures 25 , 26 .

Level 3—conditional autonomy

Surgical robots that can propose various patient-specific strategies for surgical tasks or procedures that the surgeon may select from or revise, and then automatically execute and monitor the actions of the surgeon-approved plan. The robotic system extracts parameters from uploaded data streams such as preoperative patient scans to autonomously generate potential strategies for a task, and constantly monitors the surgical environment via methods like real-time intraoperative imaging to update the strategies.

Level 4—high-level autonomy

Surgical robots that can generate and proactively select the optimal patient-specific surgical plan and autonomously execute and monitor the plan upon surgeon approval. These robotic systems constantly monitor the surgical environment and autonomously make minor updates to the procedural plan as needed. If extreme changes occur intraoperatively such that the surgical robot’s uncertainty exceeds the limits for guaranteed safety, the robotic system may request the surgeon to safely intervene through methods such as temporarily handing over control to the surgeon or requesting additional inputs. Of note, the surgeon is only required to approve the plan and supervise the procedure; although the surgeon has the option to intervene when they see fit or when requested, the robotic system shall be able to complete the procedure even without surgeon intervention.

Level 5—full autonomy

Surgical robots that can independently make decisions regarding the whole surgical procedure, including preoperative workflows. These systems can generate and select the optimal patient-specific surgical plan without prior surgeon approval, and autonomously execute and monitor the plan. Although the surgeon has the option to safely intervene, these robotic systems shall be able to independently handle all environmental and adverse conditions without requesting or needing surgeon intervention.

The surgeon may supervise the procedure at any level.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. All FDA 510(k) summary files used in this research are available at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm . All FDA De Novo summary files used in this research are available at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMN/denovo.cfm . All AccessGUDID records used in this research are available at https://accessgudid.nlm.nih.gov/ .

George, E. I., Brand, T. C., LaPorta, A., Marescaux, J. & Satava, R. M. Origins of Robotic Surgery: From Skepticism to Standard of Care. JSLS 22 https://doi.org/10.4293/JSLS.2018.00039 (2018).

Attanasio, A., Scaglioni, B., Momi, E. D., Fiorini, P. & Valdastri, P. Autonomy in Surgical Robotics. Annu. Rev. Control, Robot., Autonomous Syst. 4 , 651–679 (2021).

Article   Google Scholar  

Chinzei, K. Safety of Surgical Robots and IEC 80601-2-77: The First International Standard for Surgical Robots. Acta Polytechnica Hungarica (2019).

Fiorini, P., Goldberg, K. Y., Liu, Y. & Taylor, R. H. Concepts and Trends n Autonomy for Robot-Assisted Surgery. Proc. IEEE Inst. Electr. Electron Eng. 110 , 993–1011 (2022).

Article   PubMed   PubMed Central   Google Scholar  

Haidegger, T. Autonomy for Surgical Robots: Concepts and Paradigms. IEEE Trans. Med. Robot. Bionics 1 , 65–76 (2019).

Herron, D. M., Marohn, M. & The, S.-M. R. S. C. G. A consensus document on robotic surgery. Surg. Endosc. 22 , 313–325 (2008).

Article   CAS   PubMed   Google Scholar  

ISO - International Organization for Standardization. in Terms and definitions — General Vol. ISO 8373:2021 (2021).

ISO - International Organization for Standardization. Vol. IEC 80601-2-77:2019 (2019).

ISO - International Organization for Standardization. Vol. IEC 60601-4-1:2017 (2017).

United States Food & Drug Administration. (ed United States Department of Health and Human Services) (2020).

United States Food & Drug Administration. (ed United States Department of Health and Human Services) (2015).

United States Food & Drug Administration. Computer-Assisted Surgical Systems , < https://www.fda.gov/medical-devices/surgery-devices/computer-assisted-surgical-systems > (2022).

Siciliano, B. & Khatib, O. in Springer Handbook of Robotics (eds Siciliano B. & Khatib O.) 1-6 (Springer International Publishing, 2016).

United States Food & Drug Administration. How to Study and Market Your Device , < https://www.fda.gov/medical-devices/device-advice-comprehensive-regulatory-assistance/how-study-and-market-your-device > (2022).

United States Food & Drug Administration. De Novo Classification Request , < https://www.fda.gov/medical-devices/premarket-submissions-selecting-and-preparing-correct-submission/de-novo-classification-request > (2022).

Lefkovich, C. & Rothenberg, S. Identification of predicate creep under the 510(k) process: A case study of a robotic surgical device. PLoS One 18 , e0283442 (2023).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Freeman, A. Predicate creep: the danger of multiple predicate devices. Ann. Health Law 23 , 127–139 (2014).

Google Scholar  

Yang, G.-Z. et al. Medical robotics - Regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci. Robot. 2 , eaam8638 (2017).

Article   PubMed   Google Scholar  

United States Food & Drug Administration. Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices , < https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices > (2023).

Mittermaier, M., Raza, M. M. & Kvedar, J. C. Bias in AI-based models for medical applications: challenges and mitigation strategies. npj Digital Med. 6 , 113 (2023).

Nagy, T. D. & Haidegger, T. Performance and Capability Assessment in Surgical Subtask Automation. Sensors 22 , 2501 (2022).

Matthew, J. P. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372 , n71 (2021).

Fosch-Villaronga, E., Khanna, P., Drukarch, H. & Custers, B. The Role of Humans in Surgery Automation. Int. J. Soc. Robot. 15 , 563–580 (2023).

Haidegger, T., Speidel, S., Stoyanov, D. & Satava, R. M. Robot-Assisted Minimally Invasive Surgery—Surgical Robotics in the Data Age. Proc. IEEE 110 , 835–846 (2022).

Lin, H. C. Structure in surgical motion . (The Johns Hopkins University, 2010).

Ma, R. et al. Surgical gestures as a method to quantify surgical performance and predict patient outcomes. NPJ Digit. Med. 5 , 187 (2022).

Download references

Acknowledgements

Funding/Support: Author A.L. is supported by the Medical Scientist Training Program at the Icahn School of Medicine at Mount Sinai (NIH 5T32GM007280). Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Author information

Authors and affiliations.

Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, New York, USA

Audrey Lee, Turner S. Baker, Joshua B. Bederson & Benjamin I. Rapoport

Sinai BioDesign, Icahn School of Medicine at Mount Sinai, New York, New York, USA

Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, New York, USA

Turner S. Baker

You can also search for this author in PubMed   Google Scholar

Contributions

All authors meet the four criteria for authorship in line with the International Committee of Medical Journal Editors Recommendations. All authors reviewed, commented on, and approved the manuscript. A.L. had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Corresponding author

Correspondence to Benjamin I. Rapoport .

Ethics declarations

Competing interests.

Author B.I.R. is the Chief Science Officer at Precision Neuroscience Corporation. The subject matter of this manuscript is unrelated to the work of that company and the work presented here was not funded by or connected with the work of that company. The other authors declare no financial or non-financial competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, supplementary data 1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Lee, A., Baker, T.S., Bederson, J.B. et al. Levels of autonomy in FDA-cleared surgical robots: a systematic review. npj Digit. Med. 7 , 103 (2024). https://doi.org/10.1038/s41746-024-01102-y

Download citation

Received : 13 October 2023

Accepted : 04 April 2024

Published : 26 April 2024

DOI : https://doi.org/10.1038/s41746-024-01102-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Flood resilience: a review of evolving definitions

• Original Paper
• Open access
• Published: 30 April 2024

Cite this article

You have full access to this open access article

• Sophie Laidlaw   ORCID: orcid.org/0009-0004-1681-4057 1 &
• Sarah Percival 1  

Flooding is one of the most complicated and prolific natural hazards that communities face. Added to this, more people will be affected by this hazard than any other in the future. Within recent years, there has been a notable shift in flood risk management from risk-based approaches to resilience-based. Considered a novel and necessary approach, yet no single definition of flood resilience exists. Leading to confusion surrounding the applicability of the concept. A systematic review of flood resilience definitions was hence conducted, covering a 5-year period from 2017 to 2021, resulting in 65 papers, supplemented by a narrative review (to include papers outside of the scope of the study), which added a further 11 papers. Results indicated that whilst there is no singular definition for flood resilience, there are similarities between definitions through the use of synonymous language. Whilst there is evidence of these definitions evolving over time, there is still confusion over the definition. Further research is required to further comprehend the definitions of resilience, helping to develop the use of resilience within flood sciences and corresponding flood risk management practices.

Avoid common mistakes on your manuscript.

1 Introduction

The severity of hydrological hazards is ever increasing across the globe, heavily impacting the livelihood of communities worldwide (Kundzewicz and Matczak 2015 ), with approximately 34.2 million people being affected between 1990 and 2020 (Salas 2023 ). It is expected by 2050 that 70% of the world’s population will live in urban areas vulnerable to flooding (da Silva et al. 2012 ). Increasing the likelihood of flood-related disasters due to sheer community exposure. This risk is further compounded by ever increasing climatological changes and population pressures, with flooding expected to affect more people in the future than any other natural hazard (Hallegatte et al. 2017 ). Furthermore, current levels of flood adaption are considered inadequate, especially within the UK (Committee on Climate Change 2016 ; Percival et al. 2019 ), where 1 in 6 houses are at risk from flooding (Environment Agency 2023 ). Hence, there is an urgent need for resilient flood risk management and the research to drive it. Ensuring vulnerable communities are prepared for flooding and understand the risks they potentially face. Leading to a reduction in flood impacts including the mental and economic burdens they can have.

Flooding is a complicated and prolific hazard and one the UK for example has experienced many times over the years resulting in varying levels of impact. This includes coincident flood events, which is a combination of several flood types at once, adding further layers of complexity to an already very complicated problem (Thorne 2014 ). This was especially the case in the UK 2007 floods, where a combination of heavy rainfall and high-water levels caused by unusual weather (Environment Agency 2007 ), led to around 48,000 households being affected, 13 deaths (Cabinet Office 2008 ), and an economic cost of around £3.2 billion (Penning-Rosswell, 2014 ). This was then followed in 2013/14, where a combination of pluvial, fluvial, coastal and groundwater flooding caused significant damage to the South West of the UK, costing the economy a further £1.3 billion (Environment Agency 2016 ). Worldwide, there have also been several major and complicated flood events, particularly in the last 2 years, including the 2021 floods in Germany and Belgium (Copernicus 2021 ), the devastating 2023 Pakistan floods that affected over 30 million people (The Guardian 2023 ), and finally the 2023 Greek and Libyan floods where a combination of extreme rainfall and multiple dam collapse caused more than 6000 deaths (Flemming 2023 ; UNICEF 2023 ). Alas, these types of flood disasters (complex, costly and life-changing) are expected to increase drastically in the future, and a real shift in flood management from risk- based to resilience-based approaches, is vitally needed (Aven 2019 ). This shift is crucial to ensure management of our complex systems and reduce vulnerability within areas most at risk to flooding (Morrison et al. 2018 ). It is widely accepted that floods cannot be stopped from occurring, therefore learning from previous experiences to help reduce hardship and community vulnerability (resilience) is essential to help us deliver effective flood risk management (Kuang and Liao 2020 ).

Whilst considered a novel approach in natural hazards and flooding, resilience is widely used in other disciplines, such as psychology, ecology, and medicine. Holling ( 1973 ) first introduced the term of resilience into ecology, providing a definition referring to the persistence of systems, their ability to absorb change whilst maintaining the same relationships, similar to an equilibrium. This has provided a backbone for other fields to build on, however, within flood resilience, it has created ambiguity surrounding a definitive application of the concept, with no single definition available (Adedeji et al 2018 ; McClymont et al 2020a , b ; Disse et al 2020 ), and with different branches within flood resilience (i.e., community flood resilience, urban flood resilience, climate resilience) produced, all utilising different definitions. Understanding the definitions of flood resilience is important in creating clarity within the field, which is currently lacking in several aspects of the term and the corresponding management (McClymont et al. 2020b ). Whilst risk-based approaches can consider resilience within vulnerability measurements (IPCC, 2014 ; Percival and Teeuw 2019 ; Biswas 2023 ), it may lead to generalisation or marginalisation of resilience, and therefore reduces its significance within flood risk analyses and the measures based on them. This creates irregularities that can lead to questions regarding the dependability of the measurements and the flood risk management established due to them.

Hence this paper explored the evolution of resilience definitions within the flooding sector, discussing how differences within the definitions may be influenced by the direction of research, as well as the field in which the research is based (i.e., disaster resilience, urban resilience, flood resilience). The aim of this review being to comprehend the current definitions of flood resilience within the sector in order to enhance the applicability of this desired outcome within flood risk management and associated policies.

2 Methodology

A systematic review was conducted over a 5-year period, followed up by a supplementary narrative review, including key papers known by the authors, that may fall outside of the systematic review margins. A total of 76 papers were reviewed, outlined in Fig.  1 .

Flow diagram of the search criteria used to identify papers including definitions of flood resilience

A systematic review was first deployed using a pre-determined eligibility criterion, including a key word search, using the terms “Disaster Resilience” and “Flood Resilience” on Scopus and Google Scholar. This provided an opportunity to review papers not previously known by the authors, providing a comprehensive cover of the data set (Petticrew and Roberts 2008 ). The review was conducted over a 5-year time frame which provided a data set of n  = 827 and resulted in a search from 1st January 2017 to 31st December 2021.

Initially during the screening stage, the key word searches provided over 200 results. A more enhanced search was then deployed to further hone these results. This limited the results to presence of key words in the title, abstract, key words, and highlights of search results, including:

Papers published within the Environmental or Social Sciences sector.

Papers and articles published in English.

Open access journals.

Article or conference papers.

Whilst these criteria potentially could have limited the search field, by limiting results to this specific criterion, it ensured results were focused within natural hazards, specifically resilience to flooding. A full text of each paper that met the criteria was then obtained and a final criterion was introduced (Fig.  1 ); contain a definition of resilience ( n  =  65 ). This was used to ensure that each of the papers were focused on resilience, and the definitions could be coded and categorised, dependent on the focus of the paper, the type of resilience discussed, and the focus of the definition.

To enhance the review dataset even further, a narrative review (Fig.  1 ) was conducted to provide further depth and integrity. This included papers previously known to the authors, that were outside of the search scope (i.e., outside of the 5-year timeframe) yet were seen as pioneering ideas within the resilience sector. This added an additional 11 papers to the review.

3 Results and discussion

Initial analysis of the papers indicated research within flood resilience is increasing; this is potentially due to shifts from risk-based approaches to more pragmatic resilience-based ways of thinking, with further increases expected in the future (Fig.  2 ). This change in perspective has also translated in the definitions of flood resilience, with over 30 definitions uncovered within this review. The previously mentioned definition provided by Holling ( 1973 ), is still assumed to be the pioneering definition, and used within multiple papers included within this review (Manyena 2006 ; Cutter et al. 2008 ). Creating a foundation for further definitions to be built upon, which is evident in many of the flood resilience definitions highlighted within this study.

Resilience papers published including a definition of flood resilience, 2017–2021 (n = 65)

A frequency analysis was also conducted during the review and showed similar language is used throughout the definitions of resilience within the dataset. The most common words were ‘absorb’ ( n  =  25 ), ‘recover’ ( n  =  25), and ‘adapt’ ( n  =  22) (Fig.  3 ). This language is synonymous within not only flood resilience but also flood risk and disaster risk in general. The majority of the dataset provided differing definitions of flood resilience, yet, the most common was stated in 6 papers (Atreya and Kunreuther 2016 ; Keating et al. 2017 ; Campbell et al. 2019 ; Rezende et al. 2019 ; Laurien et al 2020 ; Hochrainer-Stigler et al. 2020 ) and was provided by Keating et al., ( 2017 ) as “ the ability of a system, community, or society to pursue its social, ecological, and economic development and growth objectives, while managing its disaster risk over time, in a mutually reinforcing way ”. Suggesting there are different elements of flood resilience that need to be considered including aspects of our systems, our communities, and society. This definition is one of the only ones that reflects reality, as all aspects of life are considered within, this is not always the case in other definitions. Whilst some studies create diverse and well-rounded definitions, which are not specific to a singular source, this is not always the case. Haque and Doberstein’s ( 2021 ) definition of community flood resilience simplifies the term, only referencing a community’s ability to withstand external factors, with minimal support. Even though complex definitions (such as Keating et al., ( 2017 )), provide an in depth understanding of flood resilience, simple definitions usually provide a foundation that complex definitions can be built upon, which is vital for the evolution of definitions such as flood resilience.

TagCloud of key words observed in flood resilience definitions included within the study

Early definitions of flood resilience appear to encompass a broader concept of resilience, for example Wildavsky ( 1991 ), referred to it as ‘bouncing back’ after unanticipated dangers, which overtime, becomes more focused and branches into several disciplines within flood resilience, including community, socio-economic and systems resilience, with overlap between the disciplines, as shown in Fig.  4 . Papers were categorised by the focus of their definitions (Fig.  4 ), with systems equalling physical based approaches, community equating to definitions that consider how communities react to flooding, and socio-ecological definitions reflecting on the relationships between society and ecosystems. The ‘other’ category encompasses more generalised definitions, such as Xu et al ( 2021 ) ‘ The ability to prepare and plan for, absorb, recover from, and more successfully adapt to adverse event ’. These types of definitions are recorded throughout the review and are generally tautological.

Aspects of resilience included within flood resilience definitions from the review

McDonald-Harker et al. ( 2021 ) focused their research on resilience amongst children and youth in disasters and used a socio-economic definition of disaster resilience within the study: “ capacity to navigate to health-enhancing resources that nurture individual, relational and community assets, as well as the capacity of individuals to negotiate with others for these resources to be provided to them in culturally meaningful ways ", which also encompasses community resilience. However, He et al., ( 2021 ), who published just a month later, provided a definition of flood resilience predominantly relating to physical systems, their capacities, and the ability to function the same. This can be considered a systems-based definition. Even though this study is also based on social flood resilience, it uses a definition that is more engineering-based, to encompass multiple aspects of resilience. Highlighting the direction of the research/project affects how the definitions are established.

Further evidence of dichotomy can be observed between the different fields of research. Within this review, 57% of the papers were flood resilience based, with the remaining 43% split relatively equally between disaster resilience (10%), climate resilience (6%), community flood resilience (11%), urban flood resilience (9%) and other, uncategorised resilience (6%) (Fig.  5 ). Definitions of flood resilience between these categories differ, with some overlap between the fields. For example, there are differences between urban flood resilience and flood resilience, however, there is overlap when it comes to the socio-economic focus of the two. Wardekker et al., ( 2020 ) defines urban flood resilience as “ the ability of a city or urban system to withstand a wide array of shocks and stresses ”. A second definition was also provided which had a greater socio-economic base, referring to how communities, businesses and systems adapt and grow after a disaster. Within urban flood resilience, there is usually a single definition provided, sometimes combining physical and socio-economic definitions, that are not specific to just flooding, but also other natural hazards. For example, Agrawal et al., ( 2020 ), who defined resilience as “ the ability of a system, community or society exposed to hazards to resist, absorb, accommodate, adapt to, transform and recover from the effects of a hazard in a timely and efficient manner, including through the preservation and restoration of its essential basic structures and functions through risk management. ” Highlighting, whilst there are differences between the definitions, urban flood resilience appears to be defined in a much simpler way than flood resilience, whilst at the same time conveying a similar message. This indicates that a complex definition of flood resilience may not be required for all aspects of flood resilience, and potentially a simple one, depending on the context, could be enough.

Papers included within the review split into resilience types (n = 109)

As a relatively new concept, there is a level of evolution expected within the term flood resilience, from simplistic definitions to more advanced and specific definitions, focusing on the different aspects of flood resilience. With increasing diversity over time, it is increasingly more difficult to define flood resilience. This has been widely observed within other fields, causing a lack of convergence (Monte et al. 2021 ), creating confusion in emerging topics or associated management. For example, Perry ( 2018 ) discussed the evolution of definitions within disaster resilience. Whilst there is a level of evolution expected within disaster resilience, since it is a relatively new concept, this increasing diversification of the definitions is making it increasingly difficult to define, however, this could be seen as a more accurate reflection of reality.

Due to the several branches of flood resilience, (i.e. urban flood resilience, community flood resilience, coastal flood resilience) the evolution of definitions is increasingly complex, and branches in many directions. However, there is evidence of evolvement, within all sectors of flooding. For example, Murdock et al., ( 2018 ), who focused on flood resilience, provided a simple definition, referring to coping with disturbances. This was then built on by Hemmati et al., ( 2020 ), who also focused on flood resilience, by suggesting that resilience is not only the ability to cope, but also to recover and adapt to any adverse effects. Whilst these are both systems based definitions, the evolution and addition to Murdock et al., ( 2018 ) definition has provided a richer definition, that not only focuses on coping, but also recovery, which is now viewed as a key part to flood resilience. This evolution suggests that as research develops, and a greater understanding of flood resilience is developed, the definitions become increasingly diverse, and include more key features (included in Fig.  3 ). Whilst this is expected within novel approaches, the continuation of divergence and inclusion of differing defining features can cause complications when trying to advance science (Quarentelli, 1995) and any management and/or policies related to that science.

There is further evidence of definitions becoming more well-rounded and inclusive of all aspects of flood resilience. Whilst earlier definitions (Klein et al. ( 2003 ), Pelling ( 2003 ) and Cutter et al., ( 2008 )) tend to focus on one aspect of flood resilience (i.e. systems or community), there is a notable increase of inclusion of 2 or more aspects of flood resilience, as early as 2005, with the UNISDR defining flood resilience as ‘The capacity of a system, community or society potentially exposed to hazards to adapt, by resisting or changing in order to reach and maintain an acceptable level of functioning and structure. This is determined by the degree to which the social system is capable of organising itself to increase this capacity for learning from past disasters for better future protection and to improve risk reduction measures’ (UNISDR, 2005 ; Manyena 2006 ). Whilst these definitions are still limited, it indicates that researchers are focusing more on the holistic element of flood resilience, with it not only being considered a social or physical component, but multi-dimensional. This idea has been adopted in other fields of resilience, such as Walker et al., ( 2002 ), who suggested that resilience could be defined using three key attributes: 1. maintenance of structure and function in the face of disturbance, 2. the ability to self-organise in response or anticipation to disturbance, 3. capacity for learning and adaptation (Bohensky and Leitch 2014 ). Whilst this incorporates different aspects of resilience, it could be considered as reductionist when defining flood resilience, due to the complexity/reality of its nature.

Within the UK public sector, there are also variations in the definitions, if one is even provided. Within the HM Government ( 2016 ) National Flood Resilience Review, the focus is still very much risk-based, focusing on infrastructure and defences. However, the Flood and Coastal Erosion Risk Management (FCERM) Scheme, set out by the Environment Agency, defines resilience in terms of flooding and coastal change, referring to the capacity of not only people but also places. Whilst it still refers to ‘protecting’ people and places, it also incorporates recovery and adaptation to coastal changes and climate changes (Environment Agency 2023 ). This shows there are further considerations of flood resilience within the governmental sector, and the understanding of the concept is developing, but not at the same rate as wider research.

Overall, whilst research into flood resilience is increasing and diversifying, there is still no one consensus on the definition of flood resilience. Due to the many branches of flood resilience, many of the definitions provided focus on different aspects of flooding, however, the flood resilience definitions can be grouped into community, socio-economic and systems resilience. These groupings provide organisation for the flood resilience definitions, with many encompassing more than one grouping (i.e. Heinzlef et al. ( 2019 ); Hochrainer-Stigler et al ( 2020 ); Slavíková, Hartmann, and Thaler, ( 2021 )). Whilst there is evident of diversification of flood resilience definitions over time, they appear to use synonymous language (Fig.  3 ), yet there is very few repeated throughout the review. Hence, whilst the language used is very similar between definitions, the focus of the paper it originated from influences the definition provided, increasing the complexity of the definitions. Although this is expected, it may not be necessary for future definitions. Furthermore, though complex definitions of flood resilience can provide a deeper understanding, a simple baseline definition may be enough and more effective, especially when the definitions are being provided to the general public or other non-expert stakeholders. This definition can then be built on depending on the direction of the study/project and the stakeholders involved.

Finally, whilst this study has indicated there is a level of evolution within definitions of flood resilience, there are limitations to this study. These include the scope of the systematic review. Conducted over a span of 5 years, only more recent publications were included within the analysis. This may have created a bias towards how the term has evolved within the field due to only including more recent research. Whilst this was supplemented with a narrative review, the research’s scope was still small. Whilst the focus of this review was directed towards flood resilience, there are many other fields within natural hazards that utilise the concept of resilience, which may provide a more comprehensive definition. By expanding the criteria and timespan of the research a more comprehensive review of resilience could take place. Not only furthering our understanding of flood and disaster resilience, but also resilience to other natural hazards. Utilising previous resilience research can aid in the creation of a base definition that can be built upon within multiple hazard-related fields. Leading to a more thorough understanding of the patterns of the use of the term resilience and the definitions provided. This will allow future research to further understand previous uses of the definition, with the aim to progress definitions of resilience, to help comprehend the idea further and provide a basis for new research and effective risk management.

4 Conclusion

The frequency and intensity of natural hazards, especially flooding, are expected to increase over the coming years. Previously, research has focused on risk, however, there is a need to shift to resilience-based approaches. However, resilience is widely used within the field of social sciences and psychology; and this has created ambiguity surrounding the definition of resilience particularly in the field of natural hazards. Hence a systematic and narrative review of flood resilience via 65 papers was presented in this article and found that whilst the frequency of flood resilience research is increasing, there is still no single definition for flood resilience, creating confusion, complexity and potentially misuse of the term. Currently, flood resilience definitions are split between several fields, however they can mostly be grouped into community, socio-economic and systems resilience, with many incorporating two or more of these. The language used between the definitions is similar, however very few definitions were repeated directly. Over time, the definition has evolved, with earlier definitions being considered simple and later ones increasing in complexity. Furthermore, overtime, there is also an element of dichotomy that has influenced the definition used.

Overall, whilst there currently is no single definition for flood resilience, many of them utilise similar language, and portray similar messages. The differences are expected to be due to the novel nature of the term within the field and has been witnessed in other fields. However, for the term to become more widely used, it needs to become more definitive and ideally with a general overarching definition, that will ultimately help understanding and application of the term. Providing a foundation for resilient flood risk management protocols to be built upon, leading to sustainable and resilient responses to flooding.

Adedeji TJ, Proverbs DG, Xiao H, Oladokun VO (2018) Towards a conceptual framework for property level flood resilience. Int J Saf Secur Eng. https://doi.org/10.2495/SAFE-V8-N4-493-504

Article   Google Scholar  

Agrawal N, Elliott M, Simonovic SP (2020) Risk and resilience: a case of perception versus reality in flood management. Water. https://doi.org/10.3390/w12051254

Atreya, A. and Kunreuther, H., (2016). Measuring community resilience: the role of the community rating system (CRS). https://doi.org/10.2139/ssrn.2788230

Aven T (2019) The call for a shift from risk to resilience: What does it mean? Risk Anal. https://doi.org/10.1111/risa.13247

Biswas S (2023) A review of socio-economic vulnerability: the emergence of its theoretical concepts, models and methodologies. Nat Hazards Res. https://doi.org/10.1016/j.nhres.2023.05.005

Bohensky EL, Leitch AM (2014) Framing the flood: a media analysis of themes of resilience in the 2011 Brisbane flood. Reg Environ Change. https://doi.org/10.1007/s10113-013-0438-2

Campbell KA, Laurien F, Czajkowski J, Keating A, Hochrainer-Stigler S, Montgomery M (2019) First insights from the flood resilience measurement tool: a large-scale community flood resilience analysis. Int J Disaster Risk Reduct. https://doi.org/10.1016/j.ijdrr.2019.101257

Committee on Climate Change (2016) UK Climate Change Risk Assessment 2017 – Synthesis Report: Priorities for the Next Five Years

Copernicus (2021) Flooding in Europe. [Online] Available at: https://climate.copernicus.eu/esotc/2021/flooding-july#:~:text=Extreme%20flooding%20occurred%20in%20parts,was%20the%20highest%20on%20record . [Accessed: 21/04/2023]

Cutter SL, Barnes L, Berry M, Burton C, Evans E, Tate E, Webb J (2008) A place-based model for understanding community resilience to natural disasters. Glob Environ Chang. https://doi.org/10.1016/j.gloenvcha.2008.07.013

Da Silva J, Kernaghan S, Luque A (2012) A systems approach to meeting the challenges of urban climate change. Int J Urb Sustain Dev. https://doi.org/10.1080/19463138.2012.718279

Disse M, Johnson TG, Leandro J, Hartmann T (2020) Exploring the relation between flood risk management and flood resilience. Water Secur. https://doi.org/10.1016/j.wasec.2020.100059

Environment Agency (2007) Review of 2007 Summer Floods [Online] Available at: https://assets.publishing.service.gov.uk/media/5a7c9a55ed915d6969f460bd/geho1107bnmi-e-e.pdf [Accessed: 20/4/2023]

Environment Agency (2016) The costs and impacts of the winter 2013 to 2014 floods- Summary [Online] Available at: https://assets.publishing.service.gov.uk/media/60354956d3bf7f0ab6856b56/The_costs_and_impacts_of_the_winter_2013_to_2014_floods_-_summary.pdf [Accessed: 11/09/23]

Environment Agency (2023) Building back better and mainstream property flood resilience. [Online] Available at: https://environmentagency.blog.gov.uk/2023/05/22/building-back-better-and-mainstreaming-property-flood-resilience/ [Accessed: 06/04/2023]

Flemming, L., (2023) Libya Floods: Entire neighbourhoods dragged into the sea. BBC News [Online] 12th September 2023. Available at: https://www.bbc.co.uk/news/world-africa-66785466 [Accessed: 30/09/2023]

Guardian, The. (2023) At least 50 dead in Pakistan monsoon floods since end of June. The Guardian [Online]. 7 th July 2023. Available at: https://www.theguardian.com/world/2023/jul/07/pakistan-monsoon-floods-punjab-province-deaths?CMP=share_btn_url

Hallegatte, S., Vogt-Schilb, A., Bangalore, M. & Rozenberg, J. (2017)  Unbreakable: Building the Resilience of the Poor in the Face of Natural Disasters  Ch. 3, 63–77.

Haque CE, Doberstein B (2021) Adaptive governance and community resilience to cyclones in coastal Bangladesh: addressing the problem of fit, social learning, and institutional collaboration. Environ Sci Policy. https://doi.org/10.1016/j.envsci.2021.08.007

He H, Delang CO, Zhou J, Li Y, He W (2021) Simulation of social resilience affected by extreme events in ancient China. Clim Change. https://doi.org/10.1007/s10584-021-03134-9

Heinzlef C, Becue V, Serre D (2019) Operationalizing urban resilience to floods in embanked territories–application in Avignon. Provence Alpes Côte D’azur Reg Saf Sci. https://doi.org/10.1016/j.ssci.2019.05.003

Hemmati M, Ellingwood BR, Mahmoud HN (2020) The role of urban growth in resilience of communities under flood risk. Earth’s Future. https://doi.org/10.1029/2019EF001382

HM Government (2016) National Flood Resilience Review [Online] Available at: https://assets.publishing.service.gov.uk/media/5a7f973de5274a2e8ab4d1af/national-flood-resilience-review.pdf [Accessed: 23/08/2023]

Hochrainer-Stigler S, Laurien F, Velev S, Keating A, Mechler R (2020) Standardized disaster and climate resilience grading: a global scale empirical analysis of community flood resilience. J Environ Manage. https://doi.org/10.1016/j.jenvman.2020.111332

Holling CS (1973) Resilience and stability of ecological systems. Annu Rev Ecol Syst 4(1):1–23. https://doi.org/10.1146/annurev.es.04.110173.000245

IPCC (2014) Summary for policymakers. In: Climate change 2014: impacts, adaptation, and vulnerability. Part A: global and sectoral aspects. Contribution of working group II to the fifth assessment report of the intergovernmental panel on climate change [Field CB, Barros VR, Dokken DJ, Mach KJ, Mastrandrea MD, Bilir TE, Chatterjee M, Ebi KL, Estrada YO, Genova RC, Girma B, Kissel ES, Levy AN, MacCracken S, Mastrandrea PR, White LL (eds)] Cambridge University Press, Cambridge, New York, NY.

Keating A, Campbell K, Mechler R, Magnuszewski P, Mochizuki J, Liu W, Szoenyi M, McQuistan C (2017) Disaster resilience: what it is and how it can engender a meaningful change in development policy. Dev Policy Rev. https://doi.org/10.1111/dpr.12201

Klein RJ, Nicholls RJ, Thomalla F (2003) Resilience to natural hazards: How useful is this concept? Glob Environ Change Part b: Environ Hazards. https://doi.org/10.1016/j.hazards.2004.02.001

Kuang D, Liao KH (2020) Learning from floods: linking flood experience and flood resilience. J Environ Manage. https://doi.org/10.1016/j.jenvman.2020.111025

Kundzewicz ZW, Matczak PIOTR (2015) Hydrological extremes and security. Proc Int Assoc Hydrol Sci. https://doi.org/10.5194/piahs-366-44-2015

Laurien F, Hochrainer-Stigler S, Keating A, Campbell K, Mechler R, Czajkowski J (2020) A typology of community flood resilience. Reg Environ Change. https://doi.org/10.1007/s10113-020-01593-x

Manyena SB (2006) The concept of resilience revisited. Disasters. https://doi.org/10.1111/j.0361-3666.2006.00331.x

McClymont K, Cunha DGF, Maidment C, Ashagre B, Vasconcelos AF, de Macedo MB, Dos Santos MFN, Júnior MNG, Mendiondo EM, Barbassa AP, Rajendran L (2020a) Towards urban resilience through Sustainable Drainage Systems: A multi-objective optimisation problem. J Environ Manage. https://doi.org/10.1016/j.jenvman.2020.111173

McClymont K, Morrison D, Beevers L, Carmen E (2020b) Flood resilience: a systematic review. J Environ Planning Manage. https://doi.org/10.1080/09640568.2019.1641474

McDonald-Harker C, Drolet J, Sehgal A (2021) A strength-based approach to exploring factors that contribute to resilience among children and youth impacted by disaster. Br J Soc Work. https://doi.org/10.1093/bjsw/bcab109

Monte BEO, Goldenfum JA, Michel GP, de Albuquerque Cavalcanti JR (2021) Terminology of natural hazards and disasters: a review and the case of Brazil. Int J Disaster Risk Reduct. https://doi.org/10.1016/j.ijdrr.2020.101970

Morrison A, Westbrook CJ, Noble BF (2018) A review of the flood risk management governance and resilience literature. J Flood Risk Manag. https://doi.org/10.1111/jfr3.12315

Murdock HJ, De Bruijn KM, Gersonius B (2018) Assessment of critical infrastructure resilience to flooding using a response curve approach. Sustainability. https://doi.org/10.3390/su10103470

Office C (2008) The pitt review: lessons learned from the 2007 floods. Cabinet Office, London

Google Scholar  

Pelling M (2003) The Vulnerability of cities: natural disasters and social resilience. Earthscan, London

Penning-Rowsell EC (2014) A realistic assessment of fluvial and coastal flood risk in England and Wales. Trans Inst Br Geogr. https://doi.org/10.1111/tran.12053

Percival S, Gaterell M, Teeuw R (2019) Urban neighbourhood flood vulnerability and risk assessments at different diurnal levels. J Flood Risk Manag. https://doi.org/10.1111/jfr3.12466

Percival S, Teeuw R (2019) A methodology for urban micro-scale coastal flood vulnerability and risk assessment and mapping. Nat Hazards. https://doi.org/10.1007/s11069-019-03648-7

Perry RW (2018) Defining disaster: an evolving concept. In Handbook of Disaster Research. https://doi.org/10.1007/978-3-319-63254-4_1

Petticrew, M. and Roberts, H., (2008).  Systematic reviews in the social sciences: A practical guide .

Rezende OM, Miranda FM, Haddad AN, Miguez MG (2019) A framework to evaluate urban flood resilience of design alternatives for flood defence considering future adverse scenarios. Water. https://doi.org/10.3390/w11071485

Salas, E.B., (2023) Global number of people affected by floods 1990–2022 . [Online] Available at: https://www.statista.com/statistics/1293353/global-number-of-people-affected-by-floods/ [Accessed: 03/04/2023]

Slavíková, L., Hartmann, T. and Thaler, T., 2021. Paradoxes of financial schemes for resilient flood recovery of households.  Wiley Interdisciplinary Reviews: Water . https://doi.org/10.1002/wat2.1497

Thorne C (2014) Geographies of UK flooding in 2013/4. Geogr J. https://doi.org/10.1111/geoj.12122

UNICEF (2023) Devastating Floods in Libya [Online] Available at: https://www.unicef.org/emergencies/devastating-flooding-libya [Accessed: 30/09/2023]

UNISDR (United Nations International Strategy for Disaster Risk Reduction) (2005) Hyogo Framework for 2005–2015: Building the Resilience of Nations and Communities to Disasters. [ Online]. Available at: https://www.unisdr.org/2005/wcdr/intergover/official-doc/L-docs/Hyogo-framework-for-action-english.pdf [Accessed: 6/12/2022]

Walker B, Carpenter S, Anderies J, Abel N, Cumming G, Janssen M, Lebel L, Norberg J, Peterson GD, Pritchard R (2002) Resilience management in social-ecological systems: a working hypothesis for a participatory approach. Conserv Ecol. https://doi.org/10.5751/ES-00356-060114

Wardekker A, Wilk B, Brown V, Uittenbroek C, Mees H, Driessen P, Wassen M, Molenaar A, Walda J, Runhaar H (2020) A diagnostic tool for supporting policymaking on urban resilience. Cities. https://doi.org/10.1016/j.cities.2020.102691

Wildavsky A.(1991). Searching for Safety. Transaction Books.

Xu W, Cong J, Proverbs D, Zhang L (2021) An evaluation of urban resilience to flooding. Water. https://doi.org/10.3390/w13152022

Download references

The authors have not disclosed any funding.

Author information

Authors and affiliations.

School of Biological and Environmental Sciences, Liverpool John Moores University, Liverpool, UK

Sophie Laidlaw & Sarah Percival

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sophie Laidlaw .

Ethics declarations

Conflict of interest.

The authors have not disclosed any competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Laidlaw, S., Percival, S. Flood resilience: a review of evolving definitions. Nat Hazards (2024). https://doi.org/10.1007/s11069-024-06627-9

Download citation

Received : 09 October 2023

Accepted : 12 April 2024

Published : 30 April 2024

DOI : https://doi.org/10.1007/s11069-024-06627-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Flood resilience
• Flood risk management
• Community flood resilience
• Systematic review
• Find a journal
• Publish with us
• Track your research

• Introduction
• Article Information

This figure addresses the 4 failures, any of which may constitute device abandonment.

eAppendix. Supplementary Data

eReferences.

Data Sharing Statement

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn
• CME & MOC

Okun MS , Marjenin T , Ekanayake J, et al. Definition of Implanted Neurological Device Abandonment : A Systematic Review and Consensus Statement . JAMA Netw Open. 2024;7(4):e248654. doi:10.1001/jamanetworkopen.2024.8654

Manage citations:

© 2024

• Permissions

Definition of Implanted Neurological Device Abandonment : A Systematic Review and Consensus Statement

• 1 Department of Neurology, Norman Fixel Institute for Neurological Diseases, Gainesville, Florida
• 2 Department of Neurosurgery, Norman Fixel Institute for Neurological Diseases, Gainesville, Florida
• 3 Musculoskeletal Clinical Regulatory Advisers, Washington, District of Columbia
• 4 Department of Neurosurgery, National Guard Hospital, Riyadh, Saudia Arabia
• 5 Department of Electronic Engineering, Imperial College London, United Kingdom
• 6 Quetz Ltd, Chelmsford, England
• 7 University of Tasmania, Tasmania, Australia
• 8 Department of Medical Physics and Biomedical Engineering, University College London, London, England
• 9 Amber Therapeutics Limited, London, England
• 10 The Royal Society, London, England
• 11 Neurotech Network, St Petersburg, Florida
• 12 Center for Neuro-Restoration, Cleveland Clinic, Cleveland, Ohio
• 13 Center for Bioethics, Massachusetts General Hospital, Harvard Medical School, Boston
• 14 Department of Psychiatry, Massachusetts General Hospital, Harvard Medical School, Boston
• 15 Medical Research Council Brain Network Dynamics Unit, Departments of Engineering Sciences and Clinical Neurosciences, University of Oxford, Oxford, England
• 16 Department of Neurology, Georgetown University Medical Center, Washington, District of Columbia
• 17 Department of Biochemistry, Georgetown University Medical Center, Washington, District of Columbia
• 18 Neuroethics Studies Program, Georgetown University Medical Center, Washington, District of Columbia
• 19 Defense Medical Ethics Center, Uniformed Services University of the Health Sciences, Bethesda, Maryland
• 20 Department of Psychiatry, Uniformed Services University of the Health Sciences, Bethesda, Maryland

Question   What definition for neurological device abandonment can be developed through consensus?

Findings   This systematic review and consensus statement reviewed 734 articles published in the professional literature and found that 7 were relevant to or addressed the issue of neurological device abandonment. A multistakeholder group developed a consensus definition for neurological device abandonment inclusive of devices used in deep brain stimulation, vagal nerve stimulation, and spinal cord stimulation, including failures related to patient consent, support before the end of the device's lifespan, and safety concerns.

Meaning   This study established a formal definition of neurological device abandonment, which may be important for development of guidelines, policies, and laws that collectively have the potential to reduce or prevent such abandonment.

Importance   Establishing a formal definition for neurological device abandonment has the potential to reduce or to prevent the occurrence of this abandonment.

Objective   To perform a systematic review of the literature and develop an expert consensus definition for neurological device abandonment.

Evidence Review   After a Royal Society Summit on Neural Interfaces (September 13-14, 2023), a systematic English language review using PubMed was undertaken to investigate extant definitions of neurological device abandonment. Articles were reviewed for relevance to neurological device abandonment in the setting of deep brain, vagal nerve, and spinal cord stimulation. This review was followed by the convening of an expert consensus group of physicians, scientists, ethicists, and stakeholders. The group summarized findings, added subject matter experience, and applied relevant ethics concepts to propose a current operational definition of neurological device abandonment. Data collection, study, and consensus development were done between September 13, 2023, and February 1, 2024.

Findings   The PubMed search revealed 734 total articles, and after review, 7 articles were found to address neurological device abandonment. The expert consensus group addressed findings as germane to neurological device abandonment and added personal experience and additional relevant peer-reviewed articles, addressed stakeholders’ respective responsibilities, and operationally defined abandonment in the context of implantable neurotechnological devices. The group further addressed whether clinical trial failure or shelving of devices would constitute or be associated with abandonment as defined. Referential to these domains and dimensions, the group proposed a standardized definition for abandonment of active implantable neurotechnological devices.

Conclusions and Relevance   This study’s consensus statement suggests that the definition for neurological device abandonment should entail failure to provide fundamental aspects of patient consent; fulfill reasonable responsibility for medical, technical, or financial support prior to the end of the device’s labeled lifetime; and address any or all immediate needs that may result in safety concerns or device ineffectiveness and that the definition of abandonment associated with the failure of a research trial should be contingent on specific circumstances.

Patients who have received implanted neurological devices, such as deep brain, vagal nerve, and spinal cord stimulation, will be increasingly abandoned. 1 , 2 This phenomenon of device abandonment will increase coincidently with neurotechnology market growth as increasing types and sophistication of implantable devices are made commercially available, older iterations of neurotechnology become obsolete or more difficult to maintain, and health care insurance coverage fails to keep pace with these realities. The topic and definition of abandonment was recently debated at the Royal Society Summit on Neural Interfaces (September 13-14, 2023) and resulting therefrom, we reviewed the literature and developed a preliminary definition for implantable neurological device abandonment based on the existing data and experience of experts in the field.

Considering the expanding device abandonment phenomenon, we suggest that it will be critical to define shareholder and stakeholder groups and their respective needs and priorities within the expanding current and proposed environments of implantable neurotechnology use. As strongly advocated by the disability movement, the adage of “nothing about us without us” aptly characterizes active roles that shareholders and stakeholders 3 should play in clinical trials conducted to generate evidence of safety and efficacy, as well as processes, guidelines, and laws required for sound commercialization, provision, access, monitoring, and economic support of extant and emerging devices.

The most important stakeholders are patients receiving these neurotechnology implants. This is because while the involvement of other shareholders and stakeholders will likely wax and wane over the utility lifetime of a device, the relationship of the patient with the device is perdurable; namely, it provides the patient with a means toward sustaining personal agency. 4 Thus, although these devices are not generally considered to be life-sustaining or life-supporting in the absolute sense, we argue that their value in qualitative life sustenance and support cannot and should not be denied, neglected, or abandoned. In this study, we refer to patients and participants interchangeably. The authors recognize that these terms refer to the people living with neurological conditions and that there are many roles within the health ecosystem. The context of this study is for specific roles that people with lived experience have within the clinical and research environment present during the time of implant and management of their neurological device.

In this study, we sought to more clearly define involved stakeholders, their respective roles and responsibilities, and circumstances and premises that constitute abandonment of patients who have active implantable technologies that are intended to diagnose, treat, or otherwise mitigate neuropsychiatric diseases, injury, and conditions; therefrom, we sought to offer a standardized definition of abandonment of active implantable neurotechnological devices. Throughout, we use the term abandonment to mean a failure to actively support medical needs of patients who, through no fault of their own, do not possess the medical, technical, or financial capabilities to maintain the safe and effective use of a durable implanted neurotechnological device.

Following a Royal Society Summit on Neural Interfaces, a systematic review of articles in English using the PubMed search engine was undertaken to investigate extant definitions of neurological device abandonment ( Figure 1 ). Articles were reviewed for relevance to neurological device abandonment in the setting of deep brain, vagal nerve, and spinal cord stimulation. An expert review group was convened to summarize findings, add subject matter experience, and apply relevant ethics concepts and any missing literature. The group proposed a current, operational definition for neurological device abandonment. The group also addressed device durability and insolvency of device companies. Data collection, study, and consensus development were conducted between September 13 to 14, 2023, and February 1, 2024. The Preferred Reporting Items for Systematic Reviews and Meta-analyses ( PRISMA ) reporting guideline was used. Our PubMed review used the search terms abandonment and deep brain stimulation , abandonment and neuromodulation , abandonment and neurological devices , retention and deep brain stimulation , device malfunction and deep brain stimulation , device removal and deep brain stimulation , abandonment and vagal nerve stimulation , and abandonment and spinal cord stimulation .

The expert consensus group consisted of 3 neuroethicists (F.G., G.L.M., and J.G.), 2 neuroscientists with experience in device engineering (S.P.D. and T.D.), 2 patients with implanted devices (S.P.D. and J.F.), 1 neurologist (M.S.O.), 1 neuropsychologist (C.K.), 1 neurosurgeon (who also founded a device company; J.E.), 1 neurological device regulatory specialist (T.M.), and 1 policy representative from the Royal Society (J.P.). One member of the group (S.P.D.) was counted as both a neuroscientist and a patient with a neurological device implant. Figure 1 summarizes the search strategy, which revealed that of 734 articles identified, 7 articles 3 , 5 - 10 were related to or addressed neurological device abandonment.

The consensus group discussed findings and contributed additional professional and personal experience and other relevant peer-reviewed ethics constructs and articles to propose a preliminary comprehensive definition for neurological device abandonment. The group addressed stakeholders and their respective responsibilities and operationally defined the context of abandonment, whether clinical trial failure constituted abandonment, and if and to what extent shelving of devices impacts abandonment, as defined. Finally, based on the literature, discussion, and expert experience, the group proposed a standardized definition for abandonment of active implantable neurotechnological devices.

In addition to the patient, key stakeholders include clinician-scientists, family members, and device manufacturers. All presumably share a common goal of improving patients’ lives, yet various stakeholders may have additional incentives and aims that may not completely support or sustain patient benefit. For example, although the clinician’s primary fiduciary responsibility is to the patient, clinician-scientists can have 2 fiduciary responsibilities: patient care and contributing to scientific knowledge, and these may be in tension if not frank conflict. Feinsinger and colleagues 11 - 13 have argued that clinician-scientists’ primary responsibility is always to patients while contribution to scientific inquiry and knowledge is secondary. However, there is some ambiguity in defining if and how the pursuit of scientific inquiry may result in direct benefit to patients in a clinical trial, and it should be appreciated that negative trials may also be associated with potential benefits. 14 - 16 Beyond the clinical and research encounter, it is important to acknowledge that device manufacturers have fiduciary responsibility qua fiscal responsibility to their boards and shareholders given that considerable resources have been invested in the development and funding of clinical trials. 17 , 18 Finally, it should be recognized that device manufacturing companies also have a responsibility to ensure their own credibility and reputation.

Such variation in stakeholder fiduciary responsibilities can lead to situations in which patients have received a medical device that may be beneficial but ongoing access to the device and the expertise and finances required to manage the device may not be guaranteed after implant. We contend that this is especially problematic in the context of active implanted neurotechnology for several reasons. First, the severity of signs and symptoms of patients enrolled in clinical trials may render these individuals at somewhat more risk. Second, there are potentially greater risks associated with neurosurgical intervention and possible effects of neurostimulation on cognition, emotion, and behavior, which would require ongoing monitoring and intervention (eg, adjustment of device performance parameters). Third, failure to monitor and maintain the implanted technology could lead to recidivistic and perhaps rebound signs, symptoms, and effects in such patients, which may create additional burden and harms. Fourth, and as an undergirding ethical construct, longitudinal evaluation and maintenance of implanted devices are essential to the intended purpose of the trial (ie, to assess the safety, effectiveness, and relative efficiency of the technology, 14 overarching goals of science via the acquisition of knowledge with intent to advancing public good, and essence of medicine: to provide right and good care of patients who are the subject of clinician moral and technical regard). 19 , 20 More information on defining and sustaining fiduciary responsibility and country specificity can be found in the eAppendix in Supplement 1 .

In general, medical abandonment is formally defined as an abrogation of clinical responsibility as incurred by a clinician’s unilateral termination of their treatment of a patient in need absent provision of adequate notice to or support for the patient to obtain substitutional care. However, as it relates to abandonment of care in circumstances wherein a patient receives an implant of an active neurotechnological device, a standardized definition that fully and granularly captures and obtains the specifics of such dissolution of responsibility has not been established, to our knowledge. While issues described in this study may also be applicable to noninvasive neurological technologies, the nonindwelling nature of such devices fails to evoke many of the same concerns. Existing notions of what constitutes device abandonment may depend on the relative perspective and values of the clinician, patient, family member, device manufacturer, and insurance company. The Royal Society Summit on Neural Interfaces meeting (September 13-14, 2023) highlighted the need for an improved definition of implantable neurological device abandonment.

Patient experience has established several factors associated with abandonment, including lack of payer support for device maintenance and replacement, the paucity or complete absence of plans for continued provision, and the use of other investigational devices when companies dissolve or cease manufacturing or providing services for a particular product. These challenges emphasize a need for technology-related guidelines and policies to ensure services to sustain patient involvement and accommodate long-term patient needs. 21 Furthermore, ethical concerns about neurotechnological device abandonment arise, at least in part, because neural systems are relatively functionally and to some extent structurally plastic. Thus, the introduction of device hardware (eg, electrodes) into the nervous system parenchyma and the actual modulatory effect of such instruments can create alterations in neurological node and network activity, which may manifest as alterations in cognitive, emotive, or behavioral domains. Simple discontinuation of the function of the device can and has been noted to evoke changes in the pathology treated and aspects of individual capacity and agency. 22

Ensuring patient and participant awareness of these outcomes and the contingencies of continued care is paramount to the probity of obtaining their consent to participate in a clinical trial or agreement to receive an implanted device. 23 , 24 Indeed, to uphold the ethical probity of any treatment or trial of such neurotechnology, genuine informed consent must address potential benefits, burdens, and risks associated with the specific device and patient understanding of associated outcomes that could arise. 23 , 25 , 26

An important consideration in developing a realistic definition of device abandonment is that clinical trials often fail to achieve their desired outcomes. To be clear, trial failure is not abandonment. While the guiding maxim for clinical care is benevolence (ie, a desire to maximize the good), the undergirding principle of clinical research in reality is nonmaleficence (ie, nonharm), given that the intended idiosyncratic and more generalized goods of any research investigation remain uncertain through the course of the study. 23 , 27 - 29 Therefore, overarching responsibility and measures to avoid harm afford a sound moral keel for any research enterprise despite the omnipresent chance of failure to achieve good ends as desired by intention and design. Trial failure can arise from safety concerns or lack of efficacy or effect, and hence discontinuation represents responsible action to avoid undue burden and harm.

However, for trial termination to remain contrary to abandonment and axiomatically nonmaleficent, it is essential for 3 things to occur. First, study participants should be informed about the possibility of discontinuance owing to such concerns about safety and inefficacy, as well as their relative assignment to treatment or control arms of the investigation. Although this information is important, patients may have difficulty understanding or retaining it. This can lead to possible therapeutic misconception and misperception by the patient of clinical abandonment. 3 Second, participants should be notified if and when the trial is being terminated. Finally, researchers in charge of the study should provide participating patients resources and vectors for other therapeutics that meet accepted standards of care. To be sure, any definition of abandonment must specify these distinctions of trial failure vs abandonment.

It is critical to disaggregate and disambiguate a failed clinical trial from a failed potential therapy. Clinical trials of active implantable neurotechnologies offer unprecedented opportunities not only to afford possible benefits rendered by successful outcomes, but also to more thoroughly investigate mechanisms of devices in question and neural structures and functions they affect. This information can lead to foundational knowledge about brain-behavior relationships that may afford viable targets to alleviate research participant and subsequent patient suffering and debility. Accomplishing these goals depends on the trial design, including choice of outcome measures, modulation parameters, surgical site, definition of benefit, timeline to assess outcomes, power analyses, variability in research participant characteristics and sign or symptom presentation, differences in surgical approach, and relevant neurophysiology. 14 Variables that may contribute to trial failure are provided in the eAppendix in Supplement 1 .

A more complex issue can arise when a particular implantable neurotechnological device is demonstrated to have efficacy in a clinical trial but then fails to translate to use in practice owing to stakeholder agendas. We refer to this circumstance as shelving. It can occur when an interventional approach is deemed to be implementable, safe, and effective but is prevented from being used in clinical care owing to ongoing issues, tensions, or conflicts in corporate intellectual property control or other licensing agreements. This can occur when companies have breakdowns in relations with a clinician-inventor or when a change in commercial strategic direction for funding to support clinical translation leads to intentional buy and block impediment of further treatment. Although this may be explicitly contrary to fundamental ethical principles guiding humanitarian considerations, it is legal as a matter of fact. At present, there is no explicit pull mechanism to ensure rollout and provision of a proven therapy after a successful clinical trial. Thus, there is potential for abandonment for non–therapeutic or health economic reasons. See the eAppendix in Supplement 1 for more information on shelving of devices.

Given that these are new technologies, it is important to address the durability of any implanted device. Durability of a neurotechnology refers to the time that the device or system remains functional and effective without requiring excessive maintenance or repair throughout its span of use. This includes the device as a single entity and as a levelled iteration (eg, versions 1.0, 2.0, and beyond) or category (eg, unipolar deep brain stimulation electrodes vs multipolar electrodes) of a therapeutic tool. Given the rapid pace of development and progress in neuroscience and technological applications in research and clinical care, what works and may be considered as cutting edge or at least a viable standard of care today may not be regarded as state of the field or even adequately effective tomorrow. 27 , 30 Patients should be informed of these possibilities and realities as an element of obtaining their consent so as to afford insight and judgment about future considerations of acquiring care as may be required and, thereby, avoiding abandonment, as mentioned previously.

Finally, there are numerous examples of neurotechnology companies becoming insolvent. For example, the commercial entity Neurovista (date of insolvency, August 2013), which was developing a first-in-human brain implant, declared bankruptcy, and patients who received implants with the technology felt betrayed. The sentiment was fortified by patient therapeutic expectations and by the perception that an unsettling break in trust had occurred. Recent reports provide evidence that 1 patient who was part of the trial compared the experience to a sense of loss or theft, stating, “They took away that part of me,” which the individual felt compromised their agency and in this way left them abandoned to an absence of care. 2 , 31 It is important to bear in mind that such devices are regarded as enabling technologies, 23 , 27 , 32 - 35 and therefore, it is vital to consider and respect the degree to which some patients may identify with these devices as constituent to their identities and personalities. 31 , 36 - 42 The distress they experience may in some cases be directly proportional to the effectiveness of the technology and their subjective relationship with it.

In cases of device maintenance or replacement (with repaired or newer versions), payers will surely play a role in determining sustainability of resources and services that can be provided to patients. We posit that any genuine discussion and actions toward defining and preventing neurotechnological device abandonment must address the value of payer conjoinment to the enterprise in ways that are supportive and facilitative to positive, beneficent ends. Failure of this sector participation would render any such efforts toward these goals problematic at least, if not impossible in reality. Lessons learned from prior and current experience with the payer sector may serve as key pediments toward bridging extant gaps in the regnant system and conduct of health care support. 25

It should be noted that when explantation or removal of a device is necessary, it will be important to address challenges of who will pay for expenses incurred. To be sure, future efforts will need to clarify the status of abandoned devices (eg, defective devices, those no longer functioning after battery depletion, or functional devices providing waning benefit). Therefore, the safety and ethics of device removal will need to be determined for each case, with special considerations afforded to whether a future upgrade in the software or change in management strategy could convert a nonfunctioning device to a functioning device.

Apropos to the previously mentioned facts, factors, considerations, and concerns provided in this systematic review and consensus statement, we propose the adoption of a standard definition of abandonment of active implantable neurotechnological devices , which constitutes 1 of the following ( Figure 2 ):

1. Failure to provide information relevant to (the existence or absence of) plans for medical, technical, and/or financial responsibility as fundamental aspects of patient consent during and after a clinical trial.

2. Failure to fulfill reasonable responsibility for medical, technical, and/or financial support prior to the end of an implantable device’s labeled lifetime.

3. Failure to address any immediate needs (eg, infection or device programming) of the individual using the implanted device, which may result in safety concerns and/or the deterioration of device effectiveness.

4. Failure of a clinical research trial if or when (1) informed consent has failed to address ongoing access to and management of the implanted device (per 1) and/or such other devices that may be demonstrated as having equal or greater therapeutic value in the future and (2) individuals responsible for the trial have not made a reasonable effort to facilitate continued access to device and support for patients who benefit from the device.

This study has several important limitations. First, because the field currently lacks a formal, accepted definition of device abandonment, it is possible that the literature review and expert group could have missed relevant aspects of abandonment. Second, the literature was sparse on this topic, and thus it will be likely that as more publications become available, these works could help refine future definitions. Third, our review did not examine similar abandonment challenges in cardiac pacemaker and related technologies. However, we performed a review of 232 additional articles using the search terms abandonment and pacemaker , which revealed 41 relevant articles that afforded comparative illustration of abandonment challenges that were similar in cardiac and neural technology implant cases. These challenges included magnetic resonance imaging–induced heating of partially abandoned devices, infections, broken lead fragments, and capping of a disconnected device. We anticipate that challenges similar to those noted for cardiac pacemaker use would increase in number as more neurological devices are implanted. Hence, we posit that definitions and issues of device abandonment will continue to evolve and therefore will require ongoing attention as neurotechnologies are further developed and in the contexts of current practices.

In this systematic review and consensus statement, a comprehensive literature review on neurological device abandonment revealed that this ethical issue was largely buried within case reports, case series, and clinical trials. Dialogue like that recently conducted at the Royal Society, with the convergence of stakeholders and combined with experience has the potential to yield a more functional definition of neurological device abandonment. We opine that these tenets previously listed may afford a working basis for further consideration, discourse, and dialogue toward establishing a formal definition of abandonment of active implantable neurotechnological devices and guidelines, policies, and laws to prevent its occurrence. We encourage such discussion and welcome participation to advance such ends, especially as devices expand into neuropsychiatric indications.

Accepted for Publication: February 27, 2024.

Published: April 30, 2024. doi:10.1001/jamanetworkopen.2024.8654

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 Okun MS et al. JAMA Network Open .

Corresponding Author: Michael S. Okun, MD, Department of Neurology, Norman Fixel Institute for Neurological Diseases, 3409 SW Williston Rd, Gainesville, FL 32607 ( [email protected] ).

Author Contributions: Drs Okun and Giordano had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: Okun, Ekanayake, Kubu.

Drafting of the manuscript: Okun, Marjenin, Ekanayake, Gilbert, Doherty, Kubu, Lázaro-Muñoz, Giordano.

Critical review of the manuscript for important intellectual content: Okun, Marjenin, Ekanayake, Doherty, Pilkington, French, Kubu, Lázaro-Muñoz, Denison.

Administrative, technical, or material support: Ekanayake, Pilkington, Denison.

Supervision: Okun, Ekanayake.

Conflict of Interest Disclosures: Dr Okun reported serving as a medical advisor to the Parkinson’s Foundation; receiving research grants from the National Institutes of Health (NIH), Parkinson’s Foundation, Michael J. Fox Foundation, Parkinson Alliance, Smallwood Foundation, Bachmann-Strauss Foundation, Tourette Syndrome Association, and University of Florida Foundation; serving as principal investigator of an NIH Training Grant; receiving royalties for publications with Hachette Book Group, Demos, Manson, Amazon, Smashwords, Books4Patients, Perseus, Robert Rose, Oxford University Press, and Cambridge University Press; serving as an associate editor for the New England Journal of Medicine Journal Watch Neurology and JAMA Neurology ; participating in continuing medical education and educational activities in the past 12 to 24 months on movement disorders sponsored by WebMD/Medscape, RMEI Medical Education, the American Academy of Neurology, the Movement Disorders Society, Mediflix, and Vanderbilt University; that grants from industry were received by the University of Florida and not Dr Okun; participating as a site principal investigator or co-investigator for several NIH-, foundation-, and industry-sponsored trials without receiving honoraria; and that research projects at the University of Florida receive device and drug donations. Dr Gilbert reported receiving a Royal Society bursary award to attend the Neural Interfaces Summit 2023 and grants from the University of Tasmania EthicsLab during the conduct of the study. Dr Doherty reported receiving devices for research studies from Innocon Medical and grants from Brain Research UK, the Inspire Foundation, and Innovate UK outside the submitted work and owning less than 1% of shares in Amber Therapeutics Ltd, London, which has subsidiaries Bioinduction Ltd (maker of the Picostim and Picostim DyNeuMo, Bristol, UK, in several first-in-human studies) and Finetech Medical Ltd (manufacturer of the Sacral Anterior Root Stimulator). Dr Kubu reported receiving grants from the National Institute of Mental Health (NIMH) during the conduct of the study and having a patent issued. Dr Lázaro-Muñoz reported receiving grants from the NIH. Dr Denison reported receiving supply devices for research from Amber Therapeutics during the conduct of the study and serving as nonexecutive chairman of Mint Neuro, which makes circuits for implants, and a consultant for Cortec, which develops neurotechnology. Dr Giordano reported receiving award UL1TR001409 from the NIH National Center for Advancing Translational Sciences through the Clinical and Translational Science Awards Program, a trademark of the Department of Health and Human Services, part of the Roadmap Initiative Re-Engineering the Clinical Research Enterprise, and National Sciences Foundation Award 2113811-Amendment ID 001 and support from the Henry Jackson Foundation for Military Medicine; Strategic Multilayer Assessment Branch of the Joint Staff, J-39, US Strategic Command, Pentagon; Asklepios Biosciences; and Leadership Initiatives. No other disclosures were reported.

Data Sharing Statement: See Supplement 2 .

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts