literature review of empirical research

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Module 2 Chapter 3: What is Empirical Literature & Where can it be Found?

In Module 1, you read about the problem of pseudoscience. Here, we revisit the issue in addressing how to locate and assess scientific or empirical literature . In this chapter you will read about:

• distinguishing between what IS and IS NOT empirical literature
• how and where to locate empirical literature for understanding diverse populations, social work problems, and social phenomena.

Probably the most important take-home lesson from this chapter is that one source is not sufficient to being well-informed on a topic. It is important to locate multiple sources of information and to critically appraise the points of convergence and divergence in the information acquired from different sources. This is especially true in emerging and poorly understood topics, as well as in answering complex questions.

What Is Empirical Literature

Social workers often need to locate valid, reliable information concerning the dimensions of a population group or subgroup, a social work problem, or social phenomenon. They might also seek information about the way specific problems or resources are distributed among the populations encountered in professional practice. Or, social workers might be interested in finding out about the way that certain people experience an event or phenomenon. Empirical literature resources may provide answers to many of these types of social work questions. In addition, resources containing data regarding social indicators may also prove helpful. Social indicators are the “facts and figures” statistics that describe the social, economic, and psychological factors that have an impact on the well-being of a community or other population group.The United Nations (UN) and the World Health Organization (WHO) are examples of organizations that monitor social indicators at a global level: dimensions of population trends (size, composition, growth/loss), health status (physical, mental, behavioral, life expectancy, maternal and infant mortality, fertility/child-bearing, and diseases like HIV/AIDS), housing and quality of sanitation (water supply, waste disposal), education and literacy, and work/income/unemployment/economics, for example.

Three characteristics stand out in empirical literature compared to other types of information available on a topic of interest: systematic observation and methodology, objectivity, and transparency/replicability/reproducibility. Let’s look a little more closely at these three features.

Systematic Observation and Methodology. The hallmark of empiricism is “repeated or reinforced observation of the facts or phenomena” (Holosko, 2006, p. 6). In empirical literature, established research methodologies and procedures are systematically applied to answer the questions of interest.

Objectivity. Gathering “facts,” whatever they may be, drives the search for empirical evidence (Holosko, 2006). Authors of empirical literature are expected to report the facts as observed, whether or not these facts support the investigators’ original hypotheses. Research integrity demands that the information be provided in an objective manner, reducing sources of investigator bias to the greatest possible extent.

Transparency and Replicability/Reproducibility.   Empirical literature is reported in such a manner that other investigators understand precisely what was done and what was found in a particular research study—to the extent that they could replicate the study to determine whether the findings are reproduced when repeated. The outcomes of an original and replication study may differ, but a reader could easily interpret the methods and procedures leading to each study’s findings.

What is NOT Empirical Literature

By now, it is probably obvious to you that literature based on “evidence” that is not developed in a systematic, objective, transparent manner is not empirical literature. On one hand, non-empirical types of professional literature may have great significance to social workers. For example, social work scholars may produce articles that are clearly identified as describing a new intervention or program without evaluative evidence, critiquing a policy or practice, or offering a tentative, untested theory about a phenomenon. These resources are useful in educating ourselves about possible issues or concerns. But, even if they are informed by evidence, they are not empirical literature. Here is a list of several sources of information that do not meet the standard of being called empirical literature:

• your course instructor’s lectures
• political statements
• advertisements
• newspapers & magazines (journalism)
• television news reports & analyses (journalism)
• many websites, Facebook postings, Twitter tweets, and blog postings
• the introductory literature review in an empirical article

You may be surprised to see the last two included in this list. Like the other sources of information listed, these sources also might lead you to look for evidence. But, they are not themselves sources of evidence. They may summarize existing evidence, but in the process of summarizing (like your instructor’s lectures), information is transformed, modified, reduced, condensed, and otherwise manipulated in such a manner that you may not see the entire, objective story. These are called secondary sources, as opposed to the original, primary source of evidence. In relying solely on secondary sources, you sacrifice your own critical appraisal and thinking about the original work—you are “buying” someone else’s interpretation and opinion about the original work, rather than developing your own interpretation and opinion. What if they got it wrong? How would you know if you did not examine the primary source for yourself? Consider the following as an example of “getting it wrong” being perpetuated.

Example: Bullying and School Shootings . One result of the heavily publicized April 1999 school shooting incident at Columbine High School (Colorado), was a heavy emphasis placed on bullying as a causal factor in these incidents (Mears, Moon, & Thielo, 2017), “creating a powerful master narrative about school shootings” (Raitanen, Sandberg, & Oksanen, 2017, p. 3). Naturally, with an identified cause, a great deal of effort was devoted to anti-bullying campaigns and interventions for enhancing resilience among youth who experience bullying.  However important these strategies might be for promoting positive mental health, preventing poor mental health, and possibly preventing suicide among school-aged children and youth, it is a mistaken belief that this can prevent school shootings (Mears, Moon, & Thielo, 2017). Many times the accounts of the perpetrators having been bullied come from potentially inaccurate third-party accounts, rather than the perpetrators themselves; bullying was not involved in all instances of school shooting; a perpetrator’s perception of being bullied/persecuted are not necessarily accurate; many who experience severe bullying do not perpetrate these incidents; bullies are the least targeted shooting victims; perpetrators of the shooting incidents were often bullying others; and, bullying is only one of many important factors associated with perpetrating such an incident (Ioannou, Hammond, & Simpson, 2015; Mears, Moon, & Thielo, 2017; Newman &Fox, 2009; Raitanen, Sandberg, & Oksanen, 2017). While mass media reports deliver bullying as a means of explaining the inexplicable, the reality is not so simple: “The connection between bullying and school shootings is elusive” (Langman, 2014), and “the relationship between bullying and school shooting is, at best, tenuous” (Mears, Moon, & Thielo, 2017, p. 940). The point is, when a narrative becomes this publicly accepted, it is difficult to sort out truth and reality without going back to original sources of information and evidence.

What May or May Not Be Empirical Literature: Literature Reviews

Investigators typically engage in a review of existing literature as they develop their own research studies. The review informs them about where knowledge gaps exist, methods previously employed by other scholars, limitations of prior work, and previous scholars’ recommendations for directing future research. These reviews may appear as a published article, without new study data being reported (see Fields, Anderson, & Dabelko-Schoeny, 2014 for example). Or, the literature review may appear in the introduction to their own empirical study report. These literature reviews are not considered to be empirical evidence sources themselves, although they may be based on empirical evidence sources. One reason is that the authors of a literature review may or may not have engaged in a systematic search process, identifying a full, rich, multi-sided pool of evidence reports.

There is, however, a type of review that applies systematic methods and is, therefore, considered to be more strongly rooted in evidence: the systematic review .

Systematic review of literature. A systematic reviewis a type of literature report where established methods have been systematically applied, objectively, in locating and synthesizing a body of literature. The systematic review report is characterized by a great deal of transparency about the methods used and the decisions made in the review process, and are replicable. Thus, it meets the criteria for empirical literature: systematic observation and methodology, objectivity, and transparency/reproducibility. We will work a great deal more with systematic reviews in the second course, SWK 3402, since they are important tools for understanding interventions. They are somewhat less common, but not unheard of, in helping us understand diverse populations, social work problems, and social phenomena.

Locating Empirical Evidence

Social workers have available a wide array of tools and resources for locating empirical evidence in the literature. These can be organized into four general categories.

Journal Articles. A number of professional journals publish articles where investigators report on the results of their empirical studies. However, it is important to know how to distinguish between empirical and non-empirical manuscripts in these journals. A key indicator, though not the only one, involves a peer review process . Many professional journals require that manuscripts undergo a process of peer review before they are accepted for publication. This means that the authors’ work is shared with scholars who provide feedback to the journal editor as to the quality of the submitted manuscript. The editor then makes a decision based on the reviewers’ feedback:

• Accept as is
• Accept with minor revisions
• Request that a revision be resubmitted (no assurance of acceptance)

When a “revise and resubmit” decision is made, the piece will go back through the review process to determine if it is now acceptable for publication and that all of the reviewers’ concerns have been adequately addressed. Editors may also reject a manuscript because it is a poor fit for the journal, based on its mission and audience, rather than sending it for review consideration.

Indicators of journal relevance. Various journals are not equally relevant to every type of question being asked of the literature. Journals may overlap to a great extent in terms of the topics they might cover; in other words, a topic might appear in multiple different journals, depending on how the topic was being addressed. For example, articles that might help answer a question about the relationship between community poverty and violence exposure might appear in several different journals, some with a focus on poverty, others with a focus on violence, and still others on community development or public health. Journal titles are sometimes a good starting point but may not give a broad enough picture of what they cover in their contents.

In focusing a literature search, it also helps to review a journal’s mission and target audience. For example, at least four different journals focus specifically on poverty:

• Journal of Children & Poverty
• Journal of Poverty
• Journal of Poverty and Social Justice
• Poverty & Public Policy

Let’s look at an example using the Journal of Poverty and Social Justice . Information about this journal is located on the journal’s webpage: http://policy.bristoluniversitypress.co.uk/journals/journal-of-poverty-and-social-justice . In the section headed “About the Journal” you can see that it is an internationally focused research journal, and that it addresses social justice issues in addition to poverty alone. The research articles are peer-reviewed (there appear to be non-empirical discussions published, as well). These descriptions about a journal are almost always available, sometimes listed as “scope” or “mission.” These descriptions also indicate the sponsorship of the journal—sponsorship may be institutional (a particular university or agency, such as Smith College Studies in Social Work ), a professional organization, such as the Council on Social Work Education (CSWE) or the National Association of Social Work (NASW), or a publishing company (e.g., Taylor & Frances, Wiley, or Sage).

Indicators of journal caliber.  Despite engaging in a peer review process, not all journals are equally rigorous. Some journals have very high rejection rates, meaning that many submitted manuscripts are rejected; others have fairly high acceptance rates, meaning that relatively few manuscripts are rejected. This is not necessarily the best indicator of quality, however, since newer journals may not be sufficiently familiar to authors with high quality manuscripts and some journals are very specific in terms of what they publish. Another index that is sometimes used is the journal’s impact factor . Impact factor is a quantitative number indicative of how often articles published in the journal are cited in the reference list of other journal articles—the statistic is calculated as the number of times on average each article published in a particular year were cited divided by the number of articles published (the number that could be cited). For example, the impact factor for the Journal of Poverty and Social Justice in our list above was 0.70 in 2017, and for the Journal of Poverty was 0.30. These are relatively low figures compared to a journal like the New England Journal of Medicine with an impact factor of 59.56! This means that articles published in that journal were, on average, cited more than 59 times in the next year or two.

Impact factors are not necessarily the best indicator of caliber, however, since many strong journals are geared toward practitioners rather than scholars, so they are less likely to be cited by other scholars but may have a large impact on a large readership. This may be the case for a journal like the one titled Social Work, the official journal of the National Association of Social Workers. It is distributed free to all members: over 120,000 practitioners, educators, and students of social work world-wide. The journal has a recent impact factor of.790. The journals with social work relevant content have impact factors in the range of 1.0 to 3.0 according to Scimago Journal & Country Rank (SJR), particularly when they are interdisciplinary journals (for example, Child Development , Journal of Marriage and Family , Child Abuse and Neglect , Child Maltreatmen t, Social Service Review , and British Journal of Social Work ). Once upon a time, a reader could locate different indexes comparing the “quality” of social work-related journals. However, the concept of “quality” is difficult to systematically define. These indexes have mostly been replaced by impact ratings, which are not necessarily the best, most robust indicators on which to rely in assessing journal quality. For example, new journals addressing cutting edge topics have not been around long enough to have been evaluated using this particular tool, and it takes a few years for articles to begin to be cited in other, later publications.

Beware of pseudo-, illegitimate, misleading, deceptive, and suspicious journals . Another side effect of living in the Age of Information is that almost anyone can circulate almost anything and call it whatever they wish. This goes for “journal” publications, as well. With the advent of open-access publishing in recent years (electronic resources available without subscription), we have seen an explosion of what are called predatory or junk journals . These are publications calling themselves journals, often with titles very similar to legitimate publications and often with fake editorial boards. These “publications” lack the integrity of legitimate journals. This caution is reminiscent of the discussions earlier in the course about pseudoscience and “snake oil” sales. The predatory nature of many apparent information dissemination outlets has to do with how scientists and scholars may be fooled into submitting their work, often paying to have their work peer-reviewed and published. There exists a “thriving black-market economy of publishing scams,” and at least two “journal blacklists” exist to help identify and avoid these scam journals (Anderson, 2017).

This issue is important to information consumers, because it creates a challenge in terms of identifying legitimate sources and publications. The challenge is particularly important to address when information from on-line, open-access journals is being considered. Open-access is not necessarily a poor choice—legitimate scientists may pay sizeable fees to legitimate publishers to make their work freely available and accessible as open-access resources. On-line access is also not necessarily a poor choice—legitimate publishers often make articles available on-line to provide timely access to the content, especially when publishing the article in hard copy will be delayed by months or even a year or more. On the other hand, stating that a journal engages in a peer-review process is no guarantee of quality—this claim may or may not be truthful. Pseudo- and junk journals may engage in some quality control practices, but may lack attention to important quality control processes, such as managing conflict of interest, reviewing content for objectivity or quality of the research conducted, or otherwise failing to adhere to industry standards (Laine & Winker, 2017).

One resource designed to assist with the process of deciphering legitimacy is the Directory of Open Access Journals (DOAJ). The DOAJ is not a comprehensive listing of all possible legitimate open-access journals, and does not guarantee quality, but it does help identify legitimate sources of information that are openly accessible and meet basic legitimacy criteria. It also is about open-access journals, not the many journals published in hard copy.

An additional caution: Search for article corrections. Despite all of the careful manuscript review and editing, sometimes an error appears in a published article. Most journals have a practice of publishing corrections in future issues. When you locate an article, it is helpful to also search for updates. Here is an example where data presented in an article’s original tables were erroneous, and a correction appeared in a later issue.

• Marchant, A., Hawton, K., Stewart A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K., & John, A. (2017). A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS One, 12(8): e0181722. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5558917/
• Marchant, A., Hawton, K., Stewart A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K., & John, A. (2018).Correction—A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS One, 13(3): e0193937.  http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0193937

Search Tools. In this age of information, it is all too easy to find items—the problem lies in sifting, sorting, and managing the vast numbers of items that can be found. For example, a simple Google® search for the topic “community poverty and violence” resulted in about 15,600,000 results! As a means of simplifying the process of searching for journal articles on a specific topic, a variety of helpful tools have emerged. One type of search tool has previously applied a filtering process for you: abstracting and indexing databases . These resources provide the user with the results of a search to which records have already passed through one or more filters. For example, PsycINFO is managed by the American Psychological Association and is devoted to peer-reviewed literature in behavioral science. It contains almost 4.5 million records and is growing every month. However, it may not be available to users who are not affiliated with a university library. Conducting a basic search for our topic of “community poverty and violence” in PsychINFO returned 1,119 articles. Still a large number, but far more manageable. Additional filters can be applied, such as limiting the range in publication dates, selecting only peer reviewed items, limiting the language of the published piece (English only, for example), and specified types of documents (either chapters, dissertations, or journal articles only, for example). Adding the filters for English, peer-reviewed journal articles published between 2010 and 2017 resulted in 346 documents being identified.

Just as was the case with journals, not all abstracting and indexing databases are equivalent. There may be overlap between them, but none is guaranteed to identify all relevant pieces of literature. Here are some examples to consider, depending on the nature of the questions asked of the literature:

• Academic Search Complete—multidisciplinary index of 9,300 peer-reviewed journals
• AgeLine—multidisciplinary index of aging-related content for over 600 journals
• Campbell Collaboration—systematic reviews in education, crime and justice, social welfare, international development
• Google Scholar—broad search tool for scholarly literature across many disciplines
• MEDLINE/ PubMed—National Library of medicine, access to over 15 million citations
• Oxford Bibliographies—annotated bibliographies, each is discipline specific (e.g., psychology, childhood studies, criminology, social work, sociology)
• PsycINFO/PsycLIT—international literature on material relevant to psychology and related disciplines
• SocINDEX—publications in sociology
• Social Sciences Abstracts—multiple disciplines
• Social Work Abstracts—many areas of social work are covered
• Web of Science—a “meta” search tool that searches other search tools, multiple disciplines

Placing our search for information about “community violence and poverty” into the Social Work Abstracts tool with no additional filters resulted in a manageable 54-item list. Finally, abstracting and indexing databases are another way to determine journal legitimacy: if a journal is indexed in a one of these systems, it is likely a legitimate journal. However, the converse is not necessarily true: if a journal is not indexed does not mean it is an illegitimate or pseudo-journal.

Government Sources. A great deal of information is gathered, analyzed, and disseminated by various governmental branches at the international, national, state, regional, county, and city level. Searching websites that end in.gov is one way to identify this type of information, often presented in articles, news briefs, and statistical reports. These government sources gather information in two ways: they fund external investigations through grants and contracts and they conduct research internally, through their own investigators. Here are some examples to consider, depending on the nature of the topic for which information is sought:

• Agency for Healthcare Research and Quality (AHRQ) at https://www.ahrq.gov/
• Bureau of Justice Statistics (BJS) at https://www.bjs.gov/
• Census Bureau at https://www.census.gov
• Morbidity and Mortality Weekly Report of the CDC (MMWR-CDC) at https://www.cdc.gov/mmwr/index.html
• Child Welfare Information Gateway at https://www.childwelfare.gov
• Children’s Bureau/Administration for Children & Families at https://www.acf.hhs.gov
• Forum on Child and Family Statistics at https://www.childstats.gov
• National Institutes of Health (NIH) at https://www.nih.gov , including (not limited to):
• National Institute on Aging (NIA at https://www.nia.nih.gov
• National Institute on Alcohol Abuse and Alcoholism (NIAAA) at https://www.niaaa.nih.gov
• National Institute of Child Health and Human Development (NICHD) at https://www.nichd.nih.gov
• National Institute on Drug Abuse (NIDA) at https://www.nida.nih.gov
• National Institute of Environmental Health Sciences at https://www.niehs.nih.gov
• National Institute of Mental Health (NIMH) at https://www.nimh.nih.gov
• National Institute on Minority Health and Health Disparities at https://www.nimhd.nih.gov
• National Institute of Justice (NIJ) at https://www.nij.gov
• Substance Abuse and Mental Health Services Administration (SAMHSA) at https://www.samhsa.gov/
• United States Agency for International Development at https://usaid.gov

Each state and many counties or cities have similar data sources and analysis reports available, such as Ohio Department of Health at https://www.odh.ohio.gov/healthstats/dataandstats.aspx and Franklin County at https://statisticalatlas.com/county/Ohio/Franklin-County/Overview . Data are available from international/global resources (e.g., United Nations and World Health Organization), as well.

Other Sources. The Health and Medicine Division (HMD) of the National Academies—previously the Institute of Medicine (IOM)—is a nonprofit institution that aims to provide government and private sector policy and other decision makers with objective analysis and advice for making informed health decisions. For example, in 2018 they produced reports on topics in substance use and mental health concerning the intersection of opioid use disorder and infectious disease,  the legal implications of emerging neurotechnologies, and a global agenda concerning the identification and prevention of violence (see http://www.nationalacademies.org/hmd/Global/Topics/Substance-Abuse-Mental-Health.aspx ). The exciting aspect of this resource is that it addresses many topics that are current concerns because they are hoping to help inform emerging policy. The caution to consider with this resource is the evidence is often still emerging, as well.

Numerous “think tank” organizations exist, each with a specific mission. For example, the Rand Corporation is a nonprofit organization offering research and analysis to address global issues since 1948. The institution’s mission is to help improve policy and decision making “to help individuals, families, and communities throughout the world be safer and more secure, healthier and more prosperous,” addressing issues of energy, education, health care, justice, the environment, international affairs, and national security (https://www.rand.org/about/history.html). And, for example, the Robert Woods Johnson Foundation is a philanthropic organization supporting research and research dissemination concerning health issues facing the United States. The foundation works to build a culture of health across systems of care (not only medical care) and communities (https://www.rwjf.org).

While many of these have a great deal of helpful evidence to share, they also may have a strong political bias. Objectivity is often lacking in what information these organizations provide: they provide evidence to support certain points of view. That is their purpose—to provide ideas on specific problems, many of which have a political component. Think tanks “are constantly researching solutions to a variety of the world’s problems, and arguing, advocating, and lobbying for policy changes at local, state, and federal levels” (quoted from https://thebestschools.org/features/most-influential-think-tanks/ ). Helpful information about what this one source identified as the 50 most influential U.S. think tanks includes identifying each think tank’s political orientation. For example, The Heritage Foundation is identified as conservative, whereas Human Rights Watch is identified as liberal.

While not the same as think tanks, many mission-driven organizations also sponsor or report on research, as well. For example, the National Association for Children of Alcoholics (NACOA) in the United States is a registered nonprofit organization. Its mission, along with other partnering organizations, private-sector groups, and federal agencies, is to promote policy and program development in research, prevention and treatment to provide information to, for, and about children of alcoholics (of all ages). Based on this mission, the organization supports knowledge development and information gathering on the topic and disseminates information that serves the needs of this population. While this is a worthwhile mission, there is no guarantee that the information meets the criteria for evidence with which we have been working. Evidence reported by think tank and mission-driven sources must be utilized with a great deal of caution and critical analysis!

In many instances an empirical report has not appeared in the published literature, but in the form of a technical or final report to the agency or program providing the funding for the research that was conducted. One such example is presented by a team of investigators funded by the National Institute of Justice to evaluate a program for training professionals to collect strong forensic evidence in instances of sexual assault (Patterson, Resko, Pierce-Weeks, & Campbell, 2014): https://www.ncjrs.gov/pdffiles1/nij/grants/247081.pdf . Investigators may serve in the capacity of consultant to agencies, programs, or institutions, and provide empirical evidence to inform activities and planning. One such example is presented by Maguire-Jack (2014) as a report to a state’s child maltreatment prevention board: https://preventionboard.wi.gov/Documents/InvestmentInPreventionPrograming_Final.pdf .

When Direct Answers to Questions Cannot Be Found. Sometimes social workers are interested in finding answers to complex questions or questions related to an emerging, not-yet-understood topic. This does not mean giving up on empirical literature. Instead, it requires a bit of creativity in approaching the literature. A Venn diagram might help explain this process. Consider a scenario where a social worker wishes to locate literature to answer a question concerning issues of intersectionality. Intersectionality is a social justice term applied to situations where multiple categorizations or classifications come together to create overlapping, interconnected, or multiplied disadvantage. For example, women with a substance use disorder and who have been incarcerated face a triple threat in terms of successful treatment for a substance use disorder: intersectionality exists between being a woman, having a substance use disorder, and having been in jail or prison. After searching the literature, little or no empirical evidence might have been located on this specific triple-threat topic. Instead, the social worker will need to seek literature on each of the threats individually, and possibly will find literature on pairs of topics (see Figure 3-1). There exists some literature about women’s outcomes for treatment of a substance use disorder (a), some literature about women during and following incarceration (b), and some literature about substance use disorders and incarceration (c). Despite not having a direct line on the center of the intersecting spheres of literature (d), the social worker can develop at least a partial picture based on the overlapping literatures.

Figure 3-1. Venn diagram of intersecting literature sets.

Take a moment to complete the following activity. For each statement about empirical literature, decide if it is true or false.

Social Work 3401 Coursebook Copyright © by Dr. Audrey Begun is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License , except where otherwise noted.

Share This Book

• Meriam Library

SWRK 330 - Social Work Research Methods

• Literature Reviews and Empirical Research
• Databases and Search Tips
• Article Citations
• Scholarly Journal Evaulation
• Statistical Sources
• Books and eBooks

What is a Literature Review?

Empirical research.

• Annotated Bibliographies

A literature review  summarizes and discusses previous publications  on a topic.

It should also:

explore past research and its strengths and weaknesses.

be used to validate the target and methods you have chosen for your proposed research.

consist of books and scholarly journals that provide research examples of populations or settings similar to your own, as well as community resources to document the need for your proposed research.

The literature review does not present new  primary  scholarship. 

be completed in the correct citation format requested by your professor  (see the  C itations Tab)

Access Purdue  OWL's Social Work Literature Review Guidelines here .  

Empirical Research  is  research  that is based on experimentation or observation, i.e. Evidence. Such  research  is often conducted to answer a specific question or to test a hypothesis (educated guess).

How do you know if a study is empirical? Read the subheadings within the article, book, or report and look for a description of the research "methodology."  Ask yourself: Could I recreate this study and test these results?

These are some key features to look for when identifying empirical research.

NOTE:  Not all of these features will be in every empirical research article, some may be excluded, use this only as a guide.

• Statement of methodology
• Research questions are clear and measurable
• Individuals, group, subjects which are being studied are identified/defined
• Data is presented regarding the findings
• Controls or instruments such as surveys or tests were conducted
• There is a literature review
• There is discussion of the results included
• Citations/references are included

See also Empirical Research Guide

• << Previous: Citations
• Next: Annotated Bibliographies >>
• Last Updated: Feb 6, 2024 8:38 AM
• URL: https://libguides.csuchico.edu/SWRK330

Meriam Library | CSU, Chico

A systematic literature review of empirical research on quality requirements

• Original Article
• Open access
• Published: 08 February 2022
• Volume 27 , pages 249–271, ( 2022 )

Cite this article

You have full access to this open access article

• Thomas Olsson   ORCID: orcid.org/0000-0002-2933-1925 1 ,
• Séverine Sentilles   ORCID: orcid.org/0000-0003-0165-3743 2 &
• Efi Papatheocharous   ORCID: orcid.org/0000-0002-5157-8131 1  

5245 Accesses

10 Citations

3 Altmetric

Explore all metrics

Quality requirements deal with how well a product should perform the intended functionality, such as start-up time and learnability. Researchers argue they are important and at the same time studies indicate there are deficiencies in practice. Our goal is to review the state of evidence for quality requirements. We want to understand the empirical research on quality requirements topics as well as evaluations of quality requirements solutions. We used a hybrid method for our systematic literature review. We defined a start set based on two literature reviews combined with a keyword-based search from selected publication venues. We snowballed based on the start set. We screened 530 papers and included 84 papers in our review. Case study method is the most common (43), followed by surveys (15) and tests (13). We found no replication studies. The two most commonly studied themes are (1) differentiating characteristics of quality requirements compared to other types of requirements, (2) the importance and prevalence of quality requirements. Quality models, QUPER, and the NFR method are evaluated in several studies, with positive indications. Goal modeling is the only modeling approach evaluated. However, all studies are small scale and long-term costs and impact are not studied. We conclude that more research is needed as empirical research on quality requirements is not increasing at the same rate as software engineering research in general. We see a gap between research and practice. The solutions proposed are usually evaluated in an academic context and surveys on quality requirements in industry indicate unsystematic handling of quality requirements.

Similar content being viewed by others

Requirements quality research: a harmonized theory, evaluation, and roadmap

Julian Frattini, Lloyd Montgomery, … Michael Unterkalmsteiner

Empirical research on requirements quality: a systematic mapping study

Lloyd Montgomery, Davide Fucci, … Walid Maalej

Quality in model-driven engineering: a tertiary study

Miguel Goulão, Vasco Amaral & Marjan Mernik

Avoid common mistakes on your manuscript.

1 Introduction

Quality requirements—also known as non-functional requirements—are requirements related to how well a product or service is supposed to perform the intended functionality [ 48 ]. Examples are start-up time, access control, and learnability [ 56 ]. Researchers have long argued the importance of quality requirements [ 39 , 68 , 82 ]. However, to what extent have problems and challenges with quality requirements been studied empirically? A recent systematic mapping study identified quality requirements as one of the emergent areas of empirical research [ 7 ]. There are several proposals over the years for how to deal with quality requirements, e.g., the NFR method [ 78 ], QUPER [ 87 ], quality models [ 37 ], and i* [ 107 ]. However, to what extent have they been empirically validated? We present a systematic literature review of empirical studies on problems and challenges as well as validated techniques and methods for quality requirements engineering.

Ambreen et al. conducted a systematic mapping study on empirical research in requirements engineering [ 7 ], published in 2018. They found 270 primary studies where 36 papers were categorized as research on quality requirements. They concluded that empirical research on quality requirements is an emerging area within requirements engineering. Berntsson Svensson et al. carried out a systematic mapping study on empirical studies on quality requirements [ 15 ], published in 2010. They found 18 primary empirical studies on quality requirements. They concluded that there is a lack of unified view and reliable empirical evidence, for example, through replications and that there is a lack of empirical work on prioritization in particular. In our study, we follow up on the systematic mapping study om Ambreen et al. [ 7 ] by performing a systematic literature review in one of the highlighted areas. Our study complements Berntsson Svensson et al. study from 2010 by performing a similar systematic literature review 10 years later and by methodologically also using a snowball approach.

There exist several definitions of quality requirements as well as names [ 48 ]. Glinz defines a non-functional requirement as an attribute (such as performance or security) or a constraint on the system. The two prevalent terms are quality requirements and non-functional requirements. Both are used roughly as much and usually mean approximately the same thing. The ISO25010 defines quality in use as to whether the solution fulfills the goals with effectiveness, efficiency, freedom from risk, and satisfaction [ 56 ]. Eckhardt et al. analyzed 530 quality requirements and found that they described a behavior—essentially a function [ 40 ]. Hence, the term non-functional might be counter-intuitive. We use the term quality requirements in this paper. In layman’s terms, we mean a quality requirement expresses how well a solution should execute an intended function, as opposed to functional requirements which express what the solution should perform. Furthermore, conceptually, we use the definition from Glinz [ 48 ] and the sub-characteristics of ISO25010 as the main refinement of quality requirements [ 56 ].

We want to understand from primary studies (1) what are the problems and challenges with quality requirements as identified through empirical studies, and (2) which quality requirements solutions have been empirically validated. We are motivated by addressing problems with quality requirements in practice and understanding why quality requirements is still, after decades of research, often reported as a troublesome area of software engineering in practice. Hence, we study which are the direct observations and experience with quality requirements. We define the following research questions for our systematic literature review:

Which empirical methods are used to study quality requirements?

What are the problems and challenges for quality requirements identified by empirical studies?

Which quality requirements solution proposals have been empirically validated?

We study quality requirements in general and therefore exclude papers focusing on specific aspects, e.g., on safety or user experience.

We summarize the related literature reviews in Sect.  2 . We describe the hybrid method we used for our systematic literature review in Sect.  3 . Section  4 elaborates on the findings from screening of 530 papers to finally include 84 papers from the years 1995 to 2019. We discuss the results and threats to validity in Sect.  5 ; empirical studies on quality requirements are—in relative terms—less common than other types of requirements engineering papers, there is a lack of longitudinal studies of quality requirements topics, we found very few replications. We conclude the paper in Sect.  6 with a reflection that there seems to be a divide between solutions proposed in an academic setting and the challenges and needs of practitioners.

2 Related work

A recent systematic mapping study on empirical studies on requirements engineering states that quality requirements are “by far the most active among these emerging research areas” [ 7 ]. They classified 36 papers of the 270 they included as papers in the quality requirements area. In their mapping, they identify security and usability as the most common topics. These results are similar to that of Ouhbi et al. systematic mapping study from 2013 [ 81 ]. However, they had slightly different keywords in their search, including also studies on quality in the requirements engineering area, which is not necessarily the same as quality requirements. A systematic mapping study is suggested for a broader area whereas a systematic literature review for a narrower area which is studies in more depth [ 63 ]. To our knowledge, there are no recent systematic literature reviews on quality requirements.

Berntsson Svensson et al. performed a systematic literature review on empirical studies on managing quality requirements in 2010 [ 15 ]. They identified 18 primary studies. They classified 12 out of the 18 primary studies as case studies, three as experiments, two as surveys, and one as a mix of survey and experiment. They classified only four of the 18 studies as properly handling validity threats systematically. Their results indicate that there is a lack of replications and multiple studies on the same or similar phenomena. However, they identify a dichotomy between two views; those who argue that quality requirements need special treatment and others who argue quality requirements need to be handled at the same time as other requirements. Furthermore, they identify a lack of studies on prioritization of quality requirements. Berntsson Svensson et al. limited their systematic literature review to studies containing the keyword “software,” whereas we did not in our study. Furthermore, Berntsson Svensson et al. performed a keyword-based literature search with a number of keywords required to be present in the search set. We used a hybrid approach and relied on snowballing instead of strict keywords. Lastly, we used Ivarsson and Gorschek [ 58 ] for rigor, which entailed stricter inclusion criteria, i.e., as a result we did not include all studies from Berntsson Svensson et al. This, in combination with performing the study 10 years afterward, means we complement Berntsson Svensson both in terms of the method as well as studied period.

Alsaqaf et al. could not find any empirical studies on quality requirements in their 2017 systematic literature review on quality requirements in large-scale agile projects [ 5 ]. They included studies on agile practices and requirements in general. Hence, their scope does not overlap significantly with ours. They found, however, 12 challenges to quality requirements in an agile context. For example, a focus on delivering functionality at the expense of architecture flexibility, difficulties in documenting quality requirements in user stories, and late validation of quality requirements. We do not explicitly focus on agile practices. Hence, there is a small overlap between their study and ours.

We designed a systematic literature review using a hybrid method [ 77 ]. The hybrid method combines a keyword-based search, typical of a systematic literature review [ 63 ], to define a start set and a snowball method [ 105 ] to systematically find relevant papers. We base our study on two literature reviews [ 7 , 15 ], which we complement in a systematic way. The overall process is found in Fig.  1 .

3.1 Approach

We decided to use a hybrid approach for our literature review [ 77 ]. A standard keyword-based systematic literature review [ 63 ] can result in a very large set of papers to review if keywords are not restrictive. On the other hand, having too restrictive keywords can result in a too-small set of papers. A snowball approach [ 105 ], on the other hand, is sensitive to the start set. If the studies are published in different communities not referencing each other, there is a risk of not finding relevant papers if the start set is limited to one community. Hence, we used a hybrid method where we combine the results from a systematic mapping study and a systematic literature review to give us one start set with a keyword-based search in the publication venues of the papers from the two review papers.

We used two different approaches to create the start sets: Start set I is based on two other literature reviews, and Start set II is created through a keyword-based search in relevant publication venues. The two start sets are combined and snowballed on to arrive at the final set of included papers. The numbers between the steps in each set are the number of references within that set. The numbers between the sets are the total number of references included in the final set

Start set I

We defined Start set I for our systematic literature review by using a systematic mapping study on empirical evidence for requirements engineering in general [ 7 ] from 2018 and a systematic literature review from 2010 [ 15 ] with similar research questions as in our paper.

The systematic literature review from 2010 by Berntsson Svensson et al. includes 18 primary studies [ 15 ]. However, we have different inclusion criteria (see Sect.  3.2 ). Hence, not all the references are included. In our final set, we included 10 of the 18 studies.

The systematic mapping by Ambreem et al. from 2018 looks at empirical evidence in general for requirements engineering [ 7 ]. They included 270 primary studies. However, there are some duplicates in their list. They classified 36 papers to be in the quality requirements area. However, there is an overlap with the Berntsson Svensson et al. review [ 15 ]. When we remove the already included papers from Berntsson Svensson et al., we reviewed 24 from Ambreem et al. and in the end included 4 of them.

Start set II

To complement start set I, we also performed a keyword-based search. We have slightly different research questions than the two papers in the Start set I. Therefore, our search string is slightly different than that of Ambreen et al. and Berntsson Svensson et al. Also, the most recent references in Start set I are from 2014, i.e., five years before we performed our search. Hence, we also fill the gap of the papers published since 2014. We include all studies, not just studies from 2014 and onward, as our method and research questions are slightly different.

We used the most frequent and highly ranked publication venues from Start set I to limit our search but still have a relevant scope. Table  1 summarizes the included conferences and journals. Even though the publication venues included are not an exhaustive list of applicable venues, we believe they are representative venues that are likely to include most communities and thereby reducing the risk with a snowball approach of missing relevant publications, as intended with the hybrid method.

We used Scopus to search. The search was performed in September 2019. Table  2 outlines the components of the search string. The title and abstract were included in the search and only papers that include both the keyword for quality requirements and the keywords for empirical research.

Snowballing

The last step in our hybrid systematic review [ 77 ] is the snowballing of the start set papers. We snowballed on the extended start set—the combination of Start set I and Start set II—to get to our final set of papers (cf. Fig.  1 ). In a snowball approach, both references in the paper (backward references) and papers referring to the paper (forward references) were screened [ 105 ]. We used Google Scholar to find forward references.

3.2 Planning

We arrived at the following inclusion criteria, in a discussion among the researchers and based on related work:

The paper should be on quality requirements or have quality requirements as a central result.

There should be empirical results with a well-defined method and validity section, not just an example or anecdotal experience.

Papers should be written in English.

The papers should be peer-reviewed.

The papers should be primary studies.

Conference or journal should be listed in reference ranking such as SJR.

Similarly, we defined our exclusion criteria as:

Literature reviews, meta-studies, etc.,—secondary studies—are excluded.

If a conference paper is extended into a journal version, we only include the journal version.

We include papers only once, i.e., duplicates are removed throughout the process.

Papers focusing on only specific aspect(s) (security, sustainability, etc.) are excluded.

All researchers were involved in the screening and classification process, even though the primary researcher performed the bulk of the work. The screening and classification were performed as follows:

Screen based on title and/or abstract.

We performed a full read when at least one researcher wanted to include the paper from the screening step.

Papers were classified according to the review protocol, see Sect.  3.3 . This was performed by the primary researcher and validated by another researcher.

To ensure reliability in the inclusion of papers and coding, the process was performed iteratively according to the sets.

For Start set I, all references from the systematic literature review [ 15 ] and systematic mapping study [ 7 ] were screened by two or three researchers. We only used the title in the screening step for Start set I. Full read and classification were performed by two or three researchers.

For Start set II, the screening was primarily performed by the primary researcher but with frequent alignment with at least one more researcher to ensure consistent screening—both on title and abstract. Similarly for the full read and classification of the papers. Specifically, we paid extra attention to which papers to exclude to ensure we did not exclude relevant papers.

The primary researcher performed the snowballing. We screened on title only for backward and forward snowballing. We included borderline cases, to ensure we did not miss any relevant references.

The full read and classification were primarily performed by the primary researcher for Start set II and the Snowballing set. A sample of the papers was read by another researcher to improve validity in addition to those cases already reviewed by more than one researcher.

The combined number of primary studies from the systematic literature review [ 15 ] and systematic mapping study [ 7 ] are 288 in Start set I. However, there is an overlap between the two studies and there are some duplicates in the Ambreen et al. paper [ 7 ]. In the end, we had 274 unique papers in Start set I. After the screening, 41 papers remained. After the full read, additional papers were excluded resulting in 14 papers in the Start set I.

Our search in Start Set II resulted in 190 papers. A total of 173 papers remained after removing duplicate papers and papers already included in Start set I. After the screening and full read, the final Start set II was 23. Hence, the extended start set (combining Start set I and Start set II) together resulted in the screening of 447 papers and the inclusion of 37 papers.

The snowball process was repeated until no new papers are found. We iterated 2 times—denoted I1 and I2 in Fig.  1 . In iteration 1, we reviewed 77 papers and included 43. In iteration 2, we reviewed 6 papers and included 4. This resulted in a total of 84 papers included and 530 papers reviewed.

3.3 Classification and review protocol

We developed the review protocol based on the systematic literature review [ 15 ] and systematic mapping study [ 7 ], the methodology papers [ 63 , 105 ], and our research questions. The main items in our protocol are:

Type of empirical study according to Wieringa et al. [ 104 ]. As we are focusing on empirical studies, we use the evaluation type—investigations of quality requirements practices in a real setting—,validation type—investigations of solution proposals before they are implemented in a real setting—,or experience type—studies where the researchers are taking a more part in the study, not just observing.

Method used in the papers. We found the following primary methods used: Experiment, test, case study, survey, and action research.

Analysis of rigor according to Ivarsson and Gorschek [ 58 ].

Thematic analysis of the papers—in an initial analysis based on the author keyword and in later iterations further refined and grouped during the analysis process.

We used a spreadsheet for documentation of the classification and review notes. The classification scheme evolved iteratively (see Sect.  3.2 ) as we included more papers. The initial themes were documented in the review process. In the analysis phase, the initial themes were used for an initial grouping of the papers. The themes were aligned and grouped in the analysis process of the papers, which included a number of meetings and iterative reviews of the results. The final themes which we used for the papers are the results of the iterative analysis process, primarily performed by the first and third researcher.

3.4 Validity

All cases where there were uncertainties whether to include a paper—both in the screening step and the full read step—or on the classification were reviewed by at least two researchers. Furthermore, to ensure consistent use of the inclusion and exclusion criterion as well as the classification we also sampled and reviewed papers that had only been screened or reviewed by only one researcher.

We used Scopus for Start set II. We confirmed that all journals and conferences selected from Start set I were found in Scopus. However, REFSQ was only indexed from 2005 and onward. However, we do not see this as a problem as we are snowballing and the papers that are missing from the Scopus search due to this, should appear in the results through the snowballing process.

We used Google Scholar in the snowballing. This is recommended [ 105 ] and usually gives the most complete results.

A hybrid search strategy can be sensitive to starting conditions, as pointed out by Mourão et al. [ 77 ]. However, their results indicate that the strategy can produce similar results as a standard systematic literature review. We carefully selected the systematic literature review and the systematic mapping study as Start set I and extended it with a keyword-based search for selected forums in Start set II. Hence, we believe the extended start set on which we snowballed is likely to be sufficient to ensure a good result when complemented with the snowball approach.

4 Analysis and results

The screening and reading of the papers in Start set I was performed in August and September 2019. The keyword-based search for Start set II was performed in September 2019. The snowballing was subsequently performed in October and November. In total, 530 papers are screened, of which 194 papers are read in full. This resulted in including 84 papers, from 1995 to 2019—see Fig.  2 .

An overview of papers included—publication type and year of publication

4.1 RQ1 Which empirical methods are used to study quality requirements?

The type of studies performed is found in Table  3 —categorized according to Wieringa et al. [ 104 ]. We differentiate between two types of validations: experiments involving human subjects and tests of algorithms on a data set. For the latter, the authors either report experiment or case study, whereas we call them test. The evaluations we found are either performed as case studies or surveys. Lastly, we found three papers that used action research—categorized as experience in Table  3 . It should be noted that the authors of the action research papers did not themselves explicitly say they performed an action research study. However, when we classified the papers, it is quite clear that, according to Wieringa et al. [ 104 ], they are in the action research category.

Case studies in an industry setting are the most common (35 of 84), followed by surveys in industry (14 of 84) and test in academic settings (13 of 84). This indicates that research on quality requirements is applied and evidence is primarily of individual case studies rather than through validation in laboratory settings. Case studies seem to have similar popularity over time, see Fig.  3 . We speculate that since requirements engineering in general as well as quality requirements in particular is a human-intensive activity, there are not so many clear cause–effect relationships to study in a rigorous experiment. Rather, it is more important to study practices in realistic settings. However, there are only three longitudinal studies.

Tests are, in contrast to the case studies, primarily performed in an academic setting, which is not necessarily representative in terms of scale and artifacts. The papers are published from 2010 to 2019—one exception, published in 2007, see Fig.  3 . One explanation might be the developments in computing driving the trend to use large data sets.

We found only one study on open source, see Table  3 , which is also longitudinal. We speculate that requirements engineering is sometimes seen as a business activity where someone other than the developers decides what should be implemented. In open-source projects, there is often a delegated culture where there is no clear product manager or similar deciding what to do, albeit there can be influential individuals such as the originator or core team member. We believe this entails that quality requirements engineering is different in open-source projects than when managed within an organization. It would be interesting to see if this hypothesis holds for requirements engineering in general and not just quality requirements. We believe, however, that by studying forums, issue management, and reviews that open-source projects are an untapped resource for quality requirements research.

An overview of papers included—method and accumulated number of publications per year

We classify the studies according to rigor, as proposed by Ivarsson and Gorschek [ 58 ]. We assess the design and validity rigor. Table  4 presents our evaluation of design and validity rigor in the papers. An inclusion criterion is that there should be an empirical study, not just examples or anecdotes. Hence, it is not surprising that overall studies score well in our rigor assessment.

High rigor is important for validations studies—to allow for replications—which is also the case for 11 out of 23 studies. The number increases to 13 if we include papers with a rigor score 0.5 for both design and validity and to 19 if we focus solely on the design rigor. Interestingly, we found no replication studies. Furthermore, the number of studies on a single (similar) approach or solution is in general low. We speculate that the originators of a solution have an interest in performing empirical studies on their solution. However, it seems unusual that practitioners or empiricists with no connection to the original solution or approach try to apply it. Furthermore, we also speculate that academic quality requirements research is not addressing critical topics for industry as there seems not to be an interest in applying and learning more about them. This implies that the research on quality requirements might need to better understand what are the real issues facing software developing organizations in terms of quality requirements.

The validity part of rigor is also important for evaluations and experience papers. Strict replications are typically not possible. However, understanding the contextual factors and validity are key in interpreting the results and assessing their applicability in other cases and contexts. 22 of the 58 evaluation and 1 of the 3 experience papers do not have a well-described validity section (rigor score 0), and 10 evaluation and 1 experience paper have a low score (rigor score 0.5). Hence, we conclude that the overall strength of evidence is weak.

4.1.1 Validations

Experiments are, in general, the most rigorous type of empirical study with the most control. However, it is difficult to scale to a realistic scenario. We found four experiments validating quality requirements with human subjects, see Table  5 .

We note that all experiments are performed with students—at varying academic levels. This might very well be appropriate for experiments [ 54 ]. We notice that there are only four experiments, which might be justified by: (1) Experiments as a method is not well accepted nor understood in the community. (2) Scale and context are key factors for applied fields such as requirements engineering, making it more challenging to design relevant experiments.

Several empirical studies study methods or tools by applying them to a data set or document set. We categorize those as tests, see Table  6 . We found three themes for tests.

Automatic analysis—The aim is to evaluate an algorithm or statistical method to automatically analyze a text, usually a requirements document.

Tool—The aim is to evaluate a tool specifically.

Runtime analysis—Evaluating the degree of satisfaction of quality requirements during runtime.

Tests are also fairly rigorous in that it is possible to control the study parameters well. It can also be possible to perform realistic studies, representative of real scenarios. The challenge is often to attain data that is representative. The data set is described in Table  7 .

The most commonly used data set is the DePaul07 data set [ 27 ]. It consists of 15 annotated specifications from student projects at DePaul University from 2007. This data set consists of requirements specification—annotated to functional and quality requirements as well as the type of quality requirement—from student projects.

There are few examples where data from commercial projects have been used. The data do not seem to be available for use by other researchers. There are examples where data from public organizations—such as government agencies—are available and used, e.g., the EU procurement specification, see Table  7 .

The most common type of data is a traditional requirements document, written in structured text. There are also a couple of instances where use case documents are used. For non-requirements specific artifacts, manuals, data use agreements, request for proposals (RFPs), and app reviews are used. From the papers in this systematic literature view, artifacts such as backlogs, feature lists, roadmaps, pull requests, or test documents do not seem to have been included.

4.1.2 Evaluations

It is usually not possible to have the same rigorous control of all study parameters in case studies [ 38 ]. However, it is often easier to have realistic scenarios, more relevant for a practical setting. We found both case studies performed in an industry context with practitioners as well as in an academic context with primarily students at different academic levels. We found 43 papers presenting case study reports on quality requirements, see Fig.  4 (all details can be found in Table  10 ). We separate case studies that explicitly evaluate a specific tool, method, technique, framework, etc., and exploratory case studies aiming to understand a specific context rather than evaluating something specific.

Case studies sometimes study a specific object, e.g., a tool or method, see Table  10 . We found 25 case studies explicitly studying a particular object. Two objects are evaluated more than once, otherwise just one case study per object. We found no longitudinal cases; hence, the case studies are executed at one point in time and not followed up at a later time. The QUPER method is studied in several case studies in several different contexts (see Table  10 ). There are several case studies for the NFR method; however, it seems the context is similar or the same in most of the cases (row 2).

Overview of the case studies on quality requirements—43 papers of the 84 included in this literature review. Scale refers to the context of the case study—small: sampling parts of the context (e.g., one part of a company) or is overall a smaller context (e.g., example system), medium: sampling a significant part of the context or a larger example system, large: sampling all significant parts of a context of an actual system (not example or made up). The context is also classified according to where the case studies are executed. Academic means primarily by students (at some academic level). Mixed means the case studies are executed in both an academic and industry context. For details, please see Table  10

We found 18 exploratory case studies on quality requirements where a specific object wasn’t the focus, see Table  10 . Rather, the goal is to understand a particular theme of quality requirements. Eight case studies want to understand details of quality requirements, e.g., the prevalence of a specific type of quality requirement or what happens in the lifecycle of a project. Five case studies have studied the process around quality requirements, two studies on sources of quality requirements (in particular app reviews), two studies in particular on developers’ view on quality requirements (specifically using StackOverflow), and lastly one study on metric related to quality requirements. We found two longitudinal case studies.

The goal of a survey is to understand a broader context without performing any intervention [ 38 ]. Surveys can be used either very early in the research process before there is a theory to find interesting hypotheses or late in the process to understand the prevalence of a theme from a theory in a certain population. We found 15 surveys, see Table  8 ; 5 interviews, 9 questionnaires, one both. The goals of the surveys are a mix of understanding practices around the engineering of quality requirements and understanding actual quality requirements as such.

Overall, the surveys we found are small in terms of the sample of the population. In most cases, they do not report from which population they sample. The most common theme is the importance of quality requirements and specific sub-characteristics—typically according to ISO9126 [ 57 ] or ISO25010 [ 56 ]. However, we cannot draw any conclusions as sampling is not systematic and the population unclear. We believe it is not realistic to systematically sample any population and achieve a statistically significant result on how important quality requirements are nor which sub-characteristics are more or less important. We speculate that, besides the sampling challenge, the variance among organizations and point in time will likely be large, making the practical implications of such studies of questionable value.

4.1.3 Experience

In action research, the researchers are more active and part of the work than, e.g., in a case study [ 38 ]. Whereas a case study does not necessarily evaluate a specific object, action research typically reports some kind of intervention where something is changed or performed. We found 3 papers we classify as action research types of experience papers [ 104 ], see Table  9 .

Two of the studies are performed at one point in time [ 2 , 71 ]. One study is longitudinal, describing the changes to the processes and practices around quality requirements over several years [ 75 ]. Interestingly, all three studies directly refer to ISO9126 [ 57 ] or ISO25010 [ 56 ].

4.2 RQ2 What are the problems and challenges for quality requirements identified by empirical studies?

We have grouped the studies on quality requirements themes thematically to analyze the problems and challenges that are identified in the empirical studies. The groups are developed iteratively among the researchers, initially from the author keywords in the included papers and then iteratively refined.

4.2.1 Quality requirements and other requirements

There is an academic debate on what quality requirements are and what they should be called [ 48 ]. We found a study indicating that quality requirements—sometimes called non-functional requirements—are functional or behavioral [ 41 ]. This is in line with other studies that report that a mix of requirements type is common [ 14 ]. Two studies find that architects address quality requirements the same way as other requirements [ 33 , 84 ], also confirmed in other surveys [ 83 ]. However, there are also research studies indicating a varying prevalence and explicitness than other requirements [ 14 , 41 , 49 , 80 , 90 ]. We interpret the current state of evidence to be unclear on the handling of quality requirements. We speculate that the answers to opinion surveys might be biased towards the expected “right” answer—as expected by the researcher—rather than the actual viewpoint of the respondent.

4.2.2 Importance and prevalence of quality requirements

Many papers present results related to the importance and prevalence of quality requirements—or sub-characteristics of quality requirements. Four papers present results from artifact analysis [ 14 , 22 , 80 , 90 ]. We found eight personal opinion survey papers  [ 8 , 9 , 12 , 23 , 34 , 35 , 47 , 97 ]. Similarly, a list of quality requirements types is developed through a survey for service-oriented applications [ 11 ]. Furthermore, we found three papers analyzing app store reviews [ 50 , 60 , 72 ] and two papers developer’s discussions on StackOverflow [ 1 , 110 ] and one paper studying 8 open-source projects communication [ 43 ]. The individual papers do not present statistical tests or variance measures. Furthermore, we found no papers elaborating on a rationale for why the distribution of sub-characteristics of quality requirements are more or less prevalent or seems as important by the subjects. We hypothesize that the importance of different quality requirements types varies over time, domain, and with personal opinion. This implies that there is no general answer to the importance of different quality requirements types. Rather, we believe it is important to adapt the quality requirements activities—such as planning and prioritization—to the specific context rather than to use predefined lists.

4.2.3 Specification of quality requirements

We found three case study papers reporting on artifact analysis of realistic requirements documents [ 14 , 41 , 90 ]. The practice seems to vary in how quality requirements are written; quantification, style, etc. One paper reporting on a scope decision database analysis [ 80 ]. The prevalence of quality requirements features is low and varies over time. Two interview surveys, furthermore, find quantification varies for the different cases as well as for the different quality requirements types [ 33 , 97 ]. Four surveys indicate that quality requirements are often poorly documented and without template [ 9 , 20 , 23 , 108 ]. Overall, the studies mostly report the usage of informal specification techniques (structured text) rather than specific modeling notations.

4.2.4 Roles perspective

Different roles—for example, project manager, architect, product manager—view and work with quality requirements differently. Two interview surveys report that architects are often involved in the elicitation and definition of quality requirements [ 9 , 33 ]. Furthermore, the clients or customers—in a bespoken context—are not explicit nor active in the elicitation and definition of quality requirements [ 33 ]. We found six papers collecting opinion data on the priority of quality requirements types from a role perspective [ 8 , 9 , 35 , 51 , 61 , 97 ]. We did not find any particular trend nor general view for different roles, except that when asked subjects tend to answer that quality requirements as a topic is important and explicitly handled—albeit that there are improvement potentials. Hence, it seems to us that, again, there might not be a general answer to the importance of different quality requirements types.

One study found that architects—despite being one source of quality requirements—are not involved in the scoping [ 34 ]. Another study found that relying on external stakeholders might lead to long lead-times and incomplete quality requirements [ 80 ]. We found one study on quality requirements engineering in an agile context. They report that communication and unstated assumptions are major challenges for quality requirements [ 4 ]. Even though opinion surveys indicate that subjects—independent of roles—claim to prioritize and explicitly work with quality requirements, there are indications that implicit quality requirements engineering is common and this leads to misalignment.

We find evidence of how different roles perceive and handle quality requirements to be insufficient to draw any particular conclusions.

4.2.5 Lifecycle perspective

We found two papers presenting results related to changes over time for the prevalence of different quality requirements types. Ernst and Mylopoulos study 8 open-source project [ 43 ] and Olsson et al. study scope decisions from a company [ 80 ]. Ernst and Mylopoulos did not find any specific pattern across the 8 open-source project in terms of prioritization or scoping of quality requirements. Olsson et al. concluded that there was an increase in the number of quality-oriented features and the acceptance of quality requirements in the scope decision process later in the product lifecycle compared to early in the product lifecycle. We found one student experiment on the stability of prioritization within the release of a smaller project [ 29 ]. They conclude that interoperability and reliability are more stable in terms of project priority whereas usability and security changed priority more in the release cycle. Lastly, we found a paper presenting a study on the presence of “Not a Problem” issue reports in the defect flow compared to how precise quality requirements are written [ 53 ]. The main result is that the more precise quality requirements are written, the lower the amount of “Not a Problem” issue reports.

The number of studies is small, which makes it difficult to draw any conclusions. However, we speculate that what happens over time is also likely to vary and be context specific. We hypothesize that there might be general patterns that, for example, products early in the lifecycle tend to overlook quality requirements, whereas products later in the lifecycle tend to focus more on quality requirements. Furthermore, it might also be differences in the handling of quality requirements depending on how close to release the project is. We see these topics as relevant to study in more detail. Longitudinal studies, involving different artifacts and sources information, e.g., issue report systems, can be an interesting way forward.

4.2.6 Prioritization

We found two case studies on quality requirements prioritization [ 33 , 96 ]. Berntsson Svensson et al. conducted an interview study with product managers and project leaders [ 96 ]. They found that ad hoc prioritization and priority grouping of quality requirements are the most common. Furthermore, they found that project leaders are more systematic (55% prioritize ad hoc) compared to the product managers (73% prioritize ad hoc). Daneva et al. found in their interview study that architects are commonly involved in prioritization of quality requirements [ 33 ]. They identified ad hoc and priority grouping as the most common approach to prioritization. Daneva et al., furthermore, found that 7 out of the 20 architects they interviewed considered themselves the role that sets the priority for quality requirements.

In summary, we find there is overall a lack of understanding of quality requirements prioritization. The studies indicate the involvement of different roles, which we believe warrants further research. Furthermore, the lack of systematic prioritization seems to be in line with requirements in general and not just for quality requirements.

4.2.7 Sources of quality requirements

There can be several sources of requirements, both roles as well as artifacts. As reported before, architects are sometimes involved in the elicitation and definition of quality requirements [ 9 , 33 ]. Three studies have identified user reviews on mobile app markets as a potential source of quality requirements [ 50 , 60 , 103 ]. One study found that users are not sufficiently involved in the elicitation [ 49 ]. However, we did not find studies on, for example, usage data or customer services data as a means to elicit and analyze quality requirements.

4.3 RQ3 Which quality requirements solution proposals have been empirically validated?

Several techniques, tools, etc., have been proposed to address problem and challenges with quality requirements. The results of the evaluations or validation of different quality requirements solutions are grouped after similarity. The sections are ordered according to the number of studies.

4.3.1 Automatic analysis

One research direction which has gained popularity is different forms of automatic analysis. The idea is that a tool can be developed to support human engineers in different aspects of quality requirements engineering. All studies we found reported positive results.

We found a number of papers investigating automatic identification and classification of quality requirements from different types of sources [ 6 , 10 , 24 , 28 , 66 , 70 , 74 , 86 , 91 , 94 , 95 , 100 , 106 , 109 ]. The different papers test different algorithms and approaches on different data sets. The most commonly used data set is from a project course at DePaul University from 2007. That data set has annotated requirements (functional or quality requirements) as well as quality requirements types, see Tables  6 and 7 . Overall, the studies are executed in an academic context (11 out of 14) and all at a small scale which might not be representative for realistic commercial cases. Furthermore, it is often assumed the presence of requirements documents, which might not be the case for agile contexts.

We found two papers presenting studies of user reviews in app stores, e.g., Apple app store or Google play [ 60 , 72 ], both rigorous. Similar to other work on automatic classification, the two studies evaluated different algorithms to identify and classify quality requirements in app reviews.

We found one paper on early aspect mining [ 85 ]. The study evaluated a tool to detect quality requirements aspects in a requirements document. Based on the detection, quality requirements are suggested to the requirements engineer. Another study evaluated a use case tool with explicit quality requirements for an agile context [ 44 ]. Both studies imply feasibility but cost or amount of effort of using them in large-scale realistic cases are not studied.

We found one study on runtime adaptations of quality requirements for self-managing systems [ 42 ]. Rather than defining fixed values for trade-offs among quality requirements, quality requirements are defined as intervals that can be optimized in runtime, depending on the specific operational conditions. They test their approach on two example systems, which show better compliance when using their approach than not.

We summarize that, while there are many tests and experiments, there are few studies of realistic scale and with realistic artifacts on automatic analysis in an quality requirements context. We also find that there is a lack of costs of running the automatic analysis, such as preparation of data, needs in terms of hardware and software, and knowledge needed by an analyst. We conclude that automatic analysis shows promise in an academic setting but has yet to be studied in a realistic scale case study or action research.

4.3.2 Goal modeling

We found two experiments on different extensions of i*, validating usefulness and correctness of the extensions compared to the original i* approach [ 99 , 111 ]. The experiments are conducted in an academic setting. Both experiments conclude that the extensions are better. Based on these experiments, we cannot say anything in general about modeling of quality requirements and usefulness of i* in general.

Researchers have performed several case studies [ 26 , 30 , 31 , 32 ]. The researchers and case context are similar and all present how the NFR method and goal modeling can work in different situations. One case study evaluated a process where business models and a quality requirements catalog are used to finally build a goal model for relevant quality requirements [ 19 ]. We found one case study using goal modeling to support product family quality requirements using goal modeling [ 79 ]. All of these case studies are of low rigor both in terms of design and validity. We have found one paper describing a test of generating goal graphs from textual requirements documents [ 86 ], and another paper testing a tool for goal modeling in an agile context [ 44 ]. Both papers indicate feasibility, i.e., the techniques seem to work in their respective context.

Overall, the evidence point to that goal modeling—in various forms—can be used and does add benefits in terms of visualization and systematic reasoning. However, we have not found any realistic scale case studies on quality requirements, nor any data on effort or impact on other parts of the development. We have not found any surveys on modeling techniques used for quality requirements. Hence, we have not found evidence of the use of goal modeling in industry specifically for quality requirements. We judge the collected evidence that goal modeling does have potential benefits but they have not been evaluated in realistic scale projects with a systematic evaluation of the whole quality requirements process.

4.3.3 Quality models and ISO9126 / ISO25010

ISO9126 [ 57 ]—and the updated version in ISO25010 [ 56 ]—is used in many of the papers we found. Al-Kilidar et al. validated the usefulness of ISO9126 in an experiment with students [ 3 ]. They conclude that the standard is difficult to interpret and too general to be useful. However, the experiment has a low rigor score—0.5 for design and 0 for validity—and lacks relevant validity description.

ADEG-NFR use ISO25010 [ 56 ] as catalogue of quality requirements types [ 93 ]. The IESE NFR method also uses the ISO standard as the basis for creating checklists and quality models [ 36 ]. Sibisi and van Waveren proposes a similar approach as the IESE NFR method, using the ISO standard as a starting point in customizing checklists and quality model [ 92 ]. Two papers present evaluations of two approaches combining ISO9126 quality models with goal modeling [ 2 , 19 ]. Another paper evaluated an approach where a checklist was derived from ISO9126 to guide the elicitation [ 67 ]. Similarly, two papers evaluate workshop and brainstorming approaches to elicitation and analysis based on ISO9126, which they propose to complement with multi-stakeholder workshops [ 65 , 101 ]. Lastly, the Quamoco approach suggests connecting the abstract quality model in the ISO standard with concrete measurements [ 102 ].

We found one paper defining a catalog of quality requirements for service-oriented applications [ 11 ]. They proposed an initial list of quality requirements which was evaluated with practitioners using a questionnaire-based approach. Mohagheghi and Aparicio conducted a three-year-long project at the Norwegian Labour and Welfare Administration [ 75 ]. The aim was to improve the quality requirements. Lochmann et al. conducted a study at Siemens, Germany [ 71 ]. The business unit in question develops a traffic control system. They introduced a quality model approach to the requirements process for quality requirements. The approach is based on ISO9126 [ 57 ].

All of the approaches suggest incorporating the quality requirements specific parts into the overall requirements process as well as tailoring to the needs of the specific organization. The different approaches seem to be recognized as useful in realistic settings, leading to a more complete understanding of the quality requirements scope with a reasonable effort. It further seems as if the tailoring part is important to gain relevance and acceptance from the development organizations—especially when considering the experiment from Al-Kilidar et al. [ 3 ].

4.3.4 Prioritization and release planning

We found one method—QUPER—focused explicitly on prioritization and release planning [ 88 ]. The researchers behind the method have performed several case studies to evaluate it [ 13 , 16 , 17 , 18 ]. We also found a prototype tool evaluation [ 17 ]. This is the single most evaluated approach. QUPER is the only approach we found which is explicitly focused on prioritization and release planning.

We found one paper proposing and evaluating an approach to handle interaction and potentially conflicting priorities among quality requirements [ 46 ]. This is similar to QUARCC, which is tested in a tool evaluation [ 55 ].

We summarize that QUPER has been evaluated both with academics and with practitioners in realistic settings. However, the long-term impact of using QUPER seems not to have been studied. However, other than QUPER, we conclude that there is no strong evidence for other solutions for prioritization and release planning of quality requirements.

4.3.5 Metrics and quality models

We found two papers evaluating the connection between key metrics to measure quality requirements types and user satisfaction [ 62 , 69 ]. The results imply that quality requirements metrics—measuring the presence of quality requirements types according to ISO9126 [ 57 ] in the specifications—is correlated with user satisfaction. Hence, even though this is not a longitudinal study, there are implications that good quality requirements engineering practices might increase user satisfaction. Both studies are personal opinion surveys, which makes it difficult to evaluate causality and root-cause. Furthermore, they measured at one point in time.

We found one paper proposing and evaluating an approach to create metrics to evaluate the responses to a request for proposal (RFP) [ 89 ]. The metrics are based on recommendations from authorities and focused on process metrics rather than product metrics. They report in their case study that they could identify specifications with deficiencies in quality requirements.

We summarize that there is not a lot of evidence on the usage of metrics in connection to quality requirements. We find that the studies have identified interesting hypotheses that can be evaluated both in an academic setting through experiments or case studies as well as in real settings through case studies or action research.

4.3.6 Knowledge management

We found three papers on different aspects of knowledge management to address quality requirements engineering. Balushi et al. report on a study at the University of Manchester [ 2 ]. They applied the ElicitO framework on a project to enhance the university website. The ontology in ElicitO implements ISO9126 [ 57 ]. The MERliNN framework suggests procedures to identify and manage knowledge flows in the elicitation and analysis process [ 21 ]. One paper evaluating a tool for QUARCC and S-Cost—knowledge-based tools for handling inter-relationships among quality requirements and stakeholders [ 55 ]. All three approaches report improved completeness of quality requirements and aligned terminology among stakeholders.

We summarize that knowledge management solutions are not well studied in the quality requirements context. We also note that there are similarities between knowledge management and quality models—which is also evident as one of the studies used ISO 9126 [ 57 ].

4.3.7 Others

We found one paper evaluating MOQARE, a misuse oriented approach to find adversarial quality requirements [ 52 ]. Another paper found that creating a clearer template and instructions for how to write a specification improved not only the quality requirements but also the attitude towards quality requirements [ 59 ]. One paper evaluates a method for how to select an appropriate technique depending on the relevant quality requirements and context factors such as lifecycle phase, etc [ 25 ]. Kopczyńska et al. experimented on a template approach for eliciting quality requirements [ 64 ]. They define a template as a regular expression. The experiment is performed with students in the third year at university. They find that using templates improved completeness and overall quality of the quality requirements. However, using templates did not speed up the elicitation process. As this is a small-scale validation, more research is needed to understand how it performs in a realistic setting.

5 Discussion

The results of our systematic literature review indicate that there are many quality requirements engineering aspects that warrant further research. The small number of studies found—84 papers over 30 years—point to a lack of studies. Furthermore, it seems to us that there is a divide between academically proposed solutions and needs of practitioners.

5.1 RQ1 Which empirical methods are used to study quality requirements?

A central research question when performing a systematic literature review is to try and answer a specific question through empirical evidence from several studies [ 63 ]. There is a tendency towards more empirical studies on quality requirements in Fig.  2 . We found 1–5 papers per year in the 1990s and 4–11 papers per year in the 2010s. However, considering that the scientific community is producing more and more papers every year, the tendency to more empirical studies in quality requirements might be smaller than that of the empirical software engineering community as a whole. A naive search on Scopus for “empirical software engineering” resulted in 50–100 papers per year during the 1990s and 450–800 papers per year in the 2010s. When we further limit the result to “requirements,” we end up with 1–20 in the 1990s and 100–250 papers per year in the 2010s. Ambreen et al. argued that quality requirements research is one of the emerging areas in their mapping study [ 7 ]. In their mapping study, they found 1–7 papers per year in the 1990s and 17–32 papers 2005–2012 Footnote 1 . In a recent paper in IEEE Software, quality requirements or non-functional requirements occurred in less than 1% as a keyword in papers in the Requirements Engineering Journal and at the REFSQ conference and not at all in the top-ten list for the Requirements Engineering conference [ 98 ]. This is an indication that the statement that quality requirements being an emerging area of research needs to be nuanced. We believe there is a need for further research on quality requirements. However, with the results we got from our study, it seems that the research on quality requirements might be less directed towards the practical challenges facing industry. This, however, is something that needs more research to be confirmed.

It should be noted that we have not included studies where specific sub-characteristics of quality requirements, such as security or usability, are the focus. Ambreen et al., however, included also those [ 7 ]. Hence, the figures we presented might be on the lower side regarding the number of empirical studies. At the same time, we included 84 papers, whereas Ambreen et al. found 36 papers. If we limit our results to papers before 2013—i.e., to the same period as Ambreen et al.—we found 43 papers.

In terms of the type of research performed, we see a similar distribution among validation, evaluation, and experience studies as Ambreen et al. [ 7 ]—see Table  3 . Evaluation research is the most common and case study in industry the largest category. As noted in Sect.  4.1 , however, the rigor overall is weak. Furthermore, replications, longitudinal, and evaluation studies of a particular solution are rare. Similarly, we found only 4 experiments that can infer that the field could benefit from larger research initiatives planning several studies over many years. This would enable researchers to plan multiple studies, combining different research methods and larger sampling of the relevant population.

5.2 RQ2 What are the problems and challenges for quality requirements identified by empirical studies?

We found studies claiming quality requirements are treated the same way as other types of requirements. We also found studies claiming companies do prioritize quality requirements and other studies claiming quality requirements are not handled properly. Furthermore, we found several studies attempting to discern which of the sub-characteristics—such as security or performance—are more important than others in a certain context. The studies we found are conducted in different contexts, domains, and different research methods. We interpret the empirical data that, on the one hand, different sub-characteristics might warrant individual attention, on the other, different research questions and methods are needed to understand and address industry-relevant challenges. We hypothesize that the requirements engineering community has not yet found a good way to analyze the quality requirements practices and challenges. We believe longitudinal studies is one way forward to deepen the understanding of quality requirements over the lifecycle of a product family rather than individual products or even just one point in time. This, of course, is not isolated to quality requirements and would entail changes to how projects are funded to allow for bigger projects.

Overall, we summarize that quality requirements are written informally without a specific notation or modeling approach, there is no clear industry practice, and documentation of quality requirements seems to be performed in the same way as other requirements. Firstly, we speculate that there is a lack of understanding from practitioners of the specific needs for quality requirements, as they do not seem to prioritize separate handling. We believe the key to improving the understanding of the importance of quality requirements is to better understand the consequences and implications of quality requirements. Secondly, we believe the results imply that quality requirements engineering need to align well with other requirements topics as the cost of separate handling might deter usage.

We did not find any studies connecting quality requirements to business value nor success criteria such as timely delivery or increased sales. There are some attempts to connect user satisfaction to quality requirements and defect flows to quality requirements. There are also some studies trying to discern perspectives internally—e.g., that of architects. We also found some studies trying to understand app reviews as a source of requirements. However, we did not find any other studies attempting to identify other sources or ways of eliciting or analyzing new requirements. Interestingly, we found only one study explicitly on open-source software. There are some studies on agile methods, but we did not find any studies on DevOps, nor any studies bridging the gap between engineering and business. We propose to study quality requirements in more contexts, especially software ecosystems where commercial organizations cooperate both on development as well as operations. Furthermore, open-source is increasingly common. Hence, we believe these areas warrant more research.

We believe—at least for certain quality requirements sub-characteristics—it is key to understand the actual usage by actual users to discern which quality requirements to address. We propose data-driven approaches as an important trend for quality requirements, seen with different automatic analysis techniques of app stores. Another example of unexploited potential is customer service data—whether through, e.g., issues or reviews. We lack a clear data-driven perspective, where usage data, as one example, is studied with a quality requirements perspective. Furthermore, we believe the community needs to understand and address the connection between quality requirements and external factors such as business value or project success. An often-cited study by Finkelstein et al. claim quality requirements as one source of problem [ 45 ]. We recognize, though, that this type of research is both challenging to design, expensive to perform, and difficult to get rigorous and relevant results. We see a need for future research in quality requirements to study not quality requirements in isolation but as part of larger studies where quality requirements are one of the research questions.

5.3 RQ3 Which quality requirements solution proposals have been empirically validated?

Quality models—as an over-arching principle—is the most common approach to elicitation. The different approaches typically propose tailoring of a generic quality model—often ISO9126 [ 57 ] or ISO25010 [ 56 ]—and a process with workshops to elicit and analyze quality requirements. The studies overwhelmingly report success with such strategies, albeit few have concrete and validated numbers for effort nor lead-time. However, it seems to us that there is no solid evidence of the cost-effectiveness of quality models, a lack of evidence of adherence over time, nor clear data for success factors from a more complete scope when using quality models. We see an opportunity for companies already using—or willing to introduce—quality models which should make it reasonable to conduct longitudinal studies on quality models as future work.

We only found one modeling approach—goal modeling. Goal modeling is—similar to quality models—well researched. There are experiments and case studies. However, we could not find any surveys nor action research. Goal modeling is, just as quality models, integrated often integrated into a process encompassing all aspects of quality requirements. We interpret the lack of quality requirements modeling studies as follows: (1) Modeling of quality requirements cannot be separated from the modeling of other requirements. This is in line with quality requirements artifact studies, which often reports similar or identical specification for quality requirements and other requirements. (2) The modeling solutions proposed by academia are not something practitioners see as applicable or relevant.

We found very few studies on data-driven requirements engineering [ 73 ] in the context of quality requirements. Rather, there seems to be a focus on the requirements specification, e.g., with quality models, goal modeling, and with the automatic analysis studies on mining specifications for quality requirements. Furthermore, we could not find any approaches integrating quality requirements engineering with, for example, DevOps and continuous experimentation. We see a gap for studies on how user feedback (reviews, customer service data, etc.) and usage data (measurements when using a software product or service) can be used for quality requirements prioritization, elicitation, and release planning. Furthermore, we propose that evaluating the trend over time as a means to better understand the connection between quality requirements and user satisfaction. Especially, we see an opportunity with the lead-time from an improvement in the quality requirements—through the implementation—and the lead-time from downward-trending user satisfaction to actions taken—using various measurements—are important to develop relevant early forecasting metrics and improved prioritization mechanisms which consider estimations of the user satisfaction.

We note that the validation studies we found tend to report an improvement and rarely conclude that the proposals as a whole do not work. The explanations can be many, but publication bias can be one, another might be confirmation bias. We see this as an indication that empirical software engineering as an area is still developing and maturing.

5.4 Limitations and threats to validity

5.4.1 construct validity.

Construct validity refers to the decisions on method and tools, and whether they are appropriate for the research questions. We utilized a hybrid search strategy. The risk is if the start set for the snowballing approach is insufficient, all papers will not be found when snowballing. However, Mourao et al. recently published an evaluation that a hybrid strategy is an appropriate alternative [ 76 ]. Hence, the hybrid method is considered to be appropriate for our research questions.

Start set I includes 274 papers between the years 1991 and 2014. Start set II includes 173 papers between 1990 and 2019—of which 70 are from the period 2015–2019. The snowballing iterations included 83 papers from 1976–2019. 447 of the 530 (84%) of papers screened come from Start set I and the Start set II of papers. Furthermore, based on our experience, we believe we have included several key papers on empirical evidence on quality requirements. Hence, we believe that the fact that most of the included papers come from Start set I and the Start set II indicates that we have likely found the majority of all relevant papers. This implies that our method selection is appropriate.

5.4.2 Internal validity

A validity threat is if papers are excluded even though they should have been included or erroneous classification. We ensured that all border-line cases were screened by at least two researchers and a sample of all papers was also screened by at least two researchers to mitigate this, see Sect.  3.4 . We also followed a pre-defined method thoroughly. In the end, the process resulted in:

For Start set I, all papers were screened by at least 2 researchers.

For Start set II, 61% of the papers were screened by at least two researchers.

For the two snowballing iterations, 29% and 60% of the papers respectively were screened by at least two researchers.

66% of the 193 papers included in the full read were read by at least two researchers, including reviewing the classification.

83% of excluded papers in the screening step and 77% of the excluded papers in the full read step were read by at least two researchers.

Excluded papers were reviewed more often by at least two researchers than included papers, mitigating the threat of excluding papers that should be included. We argue that threats to internal validity are low.

There is a risk that we missed relevant papers are we excluded relevant papers as we excluded papers focusing on a specific type of quality requirements, e.g., performance or security. The risk is related to, on the one hand, terminology and, on the other, that the specific empirical study is on a specific type but the method or phenomena applies to quality requirements in general. For the former, we believe that our selection of search terms in the extended step is by far the most prevalent, hence, it should not be a big problem. Furthermore, since we also use a snowballing approach, this threat is further minimized. For the latter, we cannot completely dismiss the threat as some empirical evidence might not be presented on other papers even though the results might be applicable. We excluded 5 papers based on these criteria. Hence, we conclude that even though this is a threat, it is not likely to largely impact the internal from our paper.

5.4.3 Conclusion validity

We followed a systematic process to address threats to the conclusion validity. Furthermore, we report the steps and results in such a way that it should be possible to replicate them. The threat to conclusion is the inclusion/exclusion primarily, which entails a human judgment and thereby susceptible for errors. However, as mentioned for internal validity threats, we used a peer review process among the authors to minimize the threats of human errors.

5.4.4 External validity

External validity concerns the applicability of the results of our study. We believe the systematic hybrid process limits this threat as we do not exclude research communities nor do we exclude studies even if particular keywords are missing. However, we did not analyze different domains in detail as that information was not available in sufficient detail in enough studies. It might be that different domains exhibit different characteristics in terms of quality requirements engineering. Hence, the results should be applied after careful consideration.

6 Conclusion

The results of our systematic literature review indicate that there are many quality requirements engineering aspects that warrant further research. We judge that 84 papers over 30 years point to a lack of studies. However, this is something that should be studied in more detail to be confirmed. Furthermore, it seems to us that there is a divide between academically proposed solutions accepted by practitioners. The proposed solutions are rarely evaluated in realistic settings—and replications are non-existent. Furthermore, practitioners rarely report using any specific approach for quality requirements. At the same time, the existing surveys are small, have an unclear sample and population, and are rarely connected to any theory. We, therefore, hypothesize that overall, there is a lack of clear empirical evidence for what software developing organizations should adopt. This, again, is something that warrants further research to understand the needs of practitioners and their relation to proposed solutions found in the literature.

For practitioners, there are some recommendations of what has worked in realistic contexts. Quality models with the associated processes, QUPER, and the NFR method have been reported as useful in several studies. However, it is not clear what the return of investment is nor the long-term effect. Still, we believe our results indicate those to be a good starting point if an organization should improve their quality requirements practices. Furthermore, goal modeling has been evaluated in academic settings with positive results. However, we could not find any evaluations in a realistic setting specifically for quality requirements. In the context where a document or specification is received, different automatic analysis approaches seem to be able to help in identifying quality requirements. However, we could not find any available tools nor clear integration in the overall software engineering process. Hence, even though these solutions show potential, the effort needed to apply them in practice is unclear.

For researchers, we see a need for longitudinal studies on quality requirements. There are examples of solutions evaluated at one point in time. However, we could not find any studies on the long-term effect and costs of changing how companies work with quality requirements. We believe that the product or portfolio lifecycle is particularly under-researched. Furthermore, we believe there is a lack of understanding of the challenges and needs in realistic settings, as the solutions proposed by researchers seem to fail in getting acceptance from practitioners. This is a rather difficult issue for individuals to address, rather the requirements engineering community should try to establish a new way of performing research where larger and longer studies are viable.

Furthermore, there are only a few studies on sources of quality requirements in general and data-driven alternatives specifically. We believe there is potential in sources such as usage data, customer service data, and continuous experimentation to complement stakeholder analysis, expert input, and focus groups. The former has the potential to take in a breadth of input closer to the actual users while the latter will focus on fewer persons’ opinions or experiences which will be less representative of the actual usage.

We limited our systematic literature review to quality requirements in general and excluding sub-categories such as security or usability. We believe it would be interesting to perform a similar study on the different sub-categories. For one, there might be differences in the sub-categories both regarding the strength of evidence and the types of solutions proposed. On the other, it might be that it does not make sense to have one solution for all types of quality requirements categories.

Their results do not include papers after 2012. Hence, we choose a slightly different interval.

Ahmad A, Feng C, Li K, Asim SM, Sun T (2019) Toward empirically investigating non-functional requirements of ios developers on stack overflow. IEEE Access 7:61145–61169. https://doi.org/10.1109/access.2019.2914429

Article   Google Scholar  

Al Balushi TH, Sampaio PRF, Loucopoulos P (2013) Eliciting and prioritizing quality requirements supported by ontologies: a case study using the e licit o framework and tool. Expert Syst 30(2):129–151

Al-Kilidar H, Cox K, Kitchenham B (2005) The use and usefulness of the iso/iec 9126 quality standard. In: International symposium on empirical software engineering. IEEE, pp 126–132. https://doi.org/10.1109/ISESE.2005.1541821

Alsaqaf W, Daneva M, Wieringa R (2019) Quality requirements challenges in the context of large-scale distributed agile: an empirical study. Inf Softw Technol 110:39–55. https://doi.org/10.1016/j.infsof.2019.01.009

Alsaqaf W, Daneva M, Wieringa R (2017) Quality requirements in large-scale distributed agile projects–a systematic literature review. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 219–234. https://doi.org/10.1007/978-3-319-54045-0_17

Amasaki S, Leelaprute P (2018) The effects of vectorization methods on non-functional requirements classification. In: 2018 44th euromicro conference on software engineering and advanced applications (SEAA). IEEE, pp. 175–182. https://doi.org/10.1109/seaa.2018.00036

Ambreen T, Ikram N, Usman M, Niazi M (2018) Empirical research in requirements engineering: trends and opportunities. Requir Eng 23(1):63–95. https://doi.org/10.1007/s00766-016-0258-2

Ameller D, Galster M, Avgeriou P, Franch X (2016) A survey on quality attributes in service-based systems. Softw Qual J 24(2):271–299

Ameller D, Ayala C, Cabot J, Franch X (2012) How do software architects consider non-functional requirements: an exploratory study. In: 2012 20th IEEE international requirements engineering conference (RE). IEEE, pp 41–50. https://doi.org/10.1109/re.2012.6345838

Baker C, Deng L, Chakraborty S, Dehlinger J (2019) Automatic multi-class non-functional software requirements classification using neural networks. In: 2019 IEEE 43rd annual computer software and applications conference (COMPSAC), vol 2. IEEE, pp 610–615. https://doi.org/10.1109/compsac.2019.10275

Becha H, Amyot D (2012) Non-functional properties in service oriented architecture - a consumer’s perspective. J Softw 7(3):575–587. https://doi.org/10.4304/jsw.7.3.575-587

Benslimane Y, Cysneiros LM, Bahli B (2007) Assessing critical functional and non-functional requirements for web-based procurement systems: a comprehensive survey. Requir Eng 12(3):191–198. https://doi.org/10.1007/s00766-007-0050-4

Berntsson Svensson R, Sprockel Y, Regnell B, Brinkkemper S (2012) Setting quality targets for coming releases with quper: an industrial case study. Requir Eng 17(4):283–298. https://doi.org/10.1007/s00766-011-0125-0

Berntsson Svensson R, Olsson T, Regnell B (2013) An investigation of how quality requirements are specified in industrial practice. Inf Softw Technol 55(7):1224–1236. https://doi.org/10.1016/j.infsof.2013.01.006

Berntsson Svensson R, Host M, Regnell B (2010) Managing quality requirements: a systematic review. In: 2010 36th EUROMICRO conference on software engineering and advanced applications. IEEE, pp 261–268. https://doi.org/10.1109/seaa.2010.55

Berntsson Svensson R, Olsson T, Regnell B (2008) Introducing support for release planning of quality requirements-an industrial evaluation of the quper model. In: Second international workshop on software product management. IEEE, pp 18–26

Berntsson Svensson R, Parker PL, Regnell B (2011) A prototype tool for quper to support release planning of quality requirements. In: Fifth international workshop on software product management (IWSPM). IEEE, pp 57–66. https://doi.org/10.1109/iwspm.2011.6046204

Berntsson Svensson R, Regnell B (2015) A case study evaluation of the guideline-supported quper model for elicitation of quality requirements. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 230–246. https://doi.org/10.1007/978-3-319-16101-3_15

Bittencourt RV, De Araujo RM (2011) Using business models to identify expected quality for information systems. Int J Bus Process Integr Manag 5(2):159–172. https://doi.org/10.1504/ijbpim.2011.040207

Borg A, Yong A, Carlshamre P, Sandahl K (2003) The bad conscience of requirements engineering: an investigation in real-world treatment of non-functional requirements. In: Software engineering research and practice in Sweden

Buitr SL, Pino FJ, Flores-Rios BL, Ibarra-Esquer JE, et al. (2017) A model for enhancing tacit knowledge flow in non-functional requirements elicitation. In: 2017 5th international conference in software engineering research and innovation (CONISOFT). IEEE, pp 25–33. https://doi.org/10.1109/conisoft.2017.00011

Calazans TA, Paldês RÁ, Canedo ED, Masson TE, Guimares FdA, Rezende MK, Braosi E, Kosloski RAD (2019) Quality requirements: analysis of utilization in the systems of a financial institution. In: ACM/IEEE international symposium on empirical software engineering and measurement (ESEM). IEEE, pp 1–6. https://doi.org/10.1109/esem.2019.8870151

Caracciolo A, Lungu MF, Nierstrasz O (2014) How do software architects specify and validate quality requirements?. In: European conference on software architecture. Springer, pp 374–389. https://doi.org/10.1007/978-3-319-09970-5_32

Casamayor A, Godoy D, Campo M (2010) Identification of non-functional requirements in textual specifications: a semi-supervised learning approach. Inf Softw Technol 52(4):436–445. https://doi.org/10.1016/j.infsof.2009.10.010

Chiam YK, Staples M, Ye X, Zhu L (2013) Applying a selection method to choose quality attribute techniques. Inf Softw Technol 55(8):1419–1436. https://doi.org/10.1016/j.infsof.2013.02.001

Chung L, Nixon BA (1995) Dealing with non-functional requirements: three experimental studies of a process-oriented approach. In: 17th international conference on software engineering. IEEE, pp 25–25. https://doi.org/10.1145/225014.225017

Cleland-Huang, J., Mazrouee S, Liguo H, Port D (2007) nfr. https://doi.org/10.5281/zenodo.268542

Cleland-Huang J, Settimi R, Zou X, Solc P (2007) Automated classification of non-functional requirements. Requir Eng 12(2):103–120. https://doi.org/10.1007/s00766-007-0045-1

Condori-Fernandez N, Lago P (2015) Can we know upfront how to prioritize quality requirements? In: 2015 IEEE fifth international workshop on empirical requirements engineering (EmpiRE). IEEE, pp 33–40. https://doi.org/10.1109/empire.2015.7431305

Cysneiros LM, do Prado Leite JS (1999) Integrating non-functional requirements into data modeling. In: Proceedings IEEE international symposium on requirements engineering. IEEE, pp 162–171

Cysneiros LM, do Prado Leite JCS (2004) Nonfunctional requirements: from elicitation to conceptual models. IEEE Trans Software Eng 30(5):328–350. https://doi.org/10.1109/tse.2004.10

Cysneiros LM, do Prado Leite JCS, Neto JDMS (2001) A framework for integrating non-functional requirements into conceptual models. Requir Eng 6(2):97–115. https://doi.org/10.1007/s007660170008

Article   MATH   Google Scholar  

Daneva M, Buglione L, Herrmann A (2013) Software architects’ experiences of quality requirements: what we know and what we do not know? In: international working conference on requirements engineering: foundation for software quality. Springer, pp 1–17. https://doi.org/10.1007/978-3-642-37422-7_1

Daneva M, Marczak S, Herrmann A (2014) Engineering of quality requirements as perceived by near-shore development centers’ architects in eastern europe: the hole in the whole. In: Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement. ACM, p 19. https://doi.org/10.1145/2652524.2652534

De La Vara JL, Wnuk K, Berntsson-Svensson R, Sánchez J, Regnell B (2011) An empirical study on the importance of quality requirements in industry. In: Proceedings of the 23rd international conference on software engineering and knowledge engineering. pp 438–443. https://doi.org/10.1007/s10270-008-0108-x

Doerr J, Kerkow D, Koenig T, Olsson T, Suzuki T (2005) Non-functional requirements in industry-three case studies adopting an experience-based nfr method. In: 13th IEEE international conference on requirements engineering (RE’05). IEEE, pp 373–382. https://doi.org/10.1109/re.2005.47

Dörr J, Kerkow D, Von Knethen A, Paech B (2003) Eliciting efficiency requirements with use cases. In: Ninth international workshop on requirements engineering: foundation for software quality. In conjunction with CAiSE, vol 3

Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering. Springer, pp 285–311. https://doi.org/10.1007/978-1-84800-044-5_11

Ebert C (1998) Putting requirement management into praxis: dealing with nonfunctional requirements. Inf Softw Technol 40(3):175–185. https://doi.org/10.1016/s0950-5849(98)00049-4

Eckhardt J, Vogelsang A, Fernández DM (2016) Are “non-functional” requirements really non-functional? an investigation of non-functional requirements in practice. In: 2016 IEEE/ACM 38th international conference on software engineering (ICSE). IEEE, pp 832–842. https://doi.org/10.1145/2884781.2884788

Eckhardt J, Vogelsang A, Fernández DM (2016) Are “non-functional” requirements really non-functional? an investigation of non-functional requirements in practice. In: IEEE/ACM 38th international conference on software engineering (ICSE). IEEE, pp 832–842. https://doi.org/10.1145/2884781.2884788

Edwards R, Bencomo N (2018) Desire: further understanding nuances of degrees of satisfaction of non-functional requirements trade-off. In: Proceedings of the 13th international conference on software engineering for adaptive and self-managing systems. ACM, pp 12–18. https://doi.org/10.1145/3194133.3194142

Ernst NA, Mylopoulos J (2010) On the perception of software quality requirements during the project lifecycle. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 143–157. https://doi.org/10.1007/978-3-642-14192-8_15

Farid WM, Mitropoulos FJ (2012) Normatic: A visual tool for modeling non-functional requirements in agile processes. In: Proceedings of IEEE southeastcon. IEEE, pp 1–8 https://doi.org/10.1109/secon.2012.6196989

Finkelstein A, Dowell J (1996) A comedy of errors: the london ambulance service case study. In: Proceedings of the 8th International Workshop on Software Specification and Design. pp. 2–4. IEEE

García-Mireles GA, De La Rubia MÁM, García F, Piattini M (2015) Identifying quality characteristic interactions during software development. In: International conference on evaluation of novel approaches to software engineering (ENASE). IEEE, pp 78–89. https://doi.org/10.5220/0005351800780089

García-Mireles GA (2016) Identifying relevant product quality characteristics in the context of very small organizations. Comput Sci Inf Syst 13(3):875–900. https://doi.org/10.2298/csis160809034g

Glinz M (2007) On non-functional requirements. In: 15th IEEE international requirements engineering conference (RE 2007). IEEE, pp 21–26. https://doi.org/10.1109/re.2007.45

Grimshaw DJ, Draper GW (2001) Non-functional requirements analysis: deficiencies in structured methods. Inf Softw Technol 43(11):629–634. https://doi.org/10.1016/s0950-5849(01)00171-9

Groen EC, Kopczyńska S, Hauer MP, Krafft TD, Doerr J (2017) Users-the hidden software product quality experts?: a study on how app users report quality aspects in online reviews. In: IEEE 25th international requirements engineering conference (RE). IEEE, pp 80–89. https://doi.org/10.1109/re.2017.73

Haigh M (2010) Software quality, non-functional software requirements and it-business alignment. Software Qual J 18(3):361–385. https://doi.org/10.1007/s11219-010-9098-3

Herrmann A, Paech B (2008) Moqare: misuse-oriented quality requirements engineering. Requir Eng 13(1):73–86

Ho CW, Williams L, Robinson B (2008) Examining the relationships between performance requirements and “not a problem” defect reports. In: 16th IEEE international requirements engineering conference. IEEE, pp 135–144

Höst M, Regnell B, Wohlin C (2000) Using students as subjects-a comparative study of students and professionals in lead-time impact assessment. Empir Softw Eng 5(3):201–214. https://doi.org/10.1023/A:1026586415054

In H, Boehm BW (2001) Using winwin quality requirements management tools: a case study. Ann Softw Eng 11(1):141–174

ISO/IEC 25010:2011(E) (2011) Systems and software engineering - systems and software quality requirements and evaluation (SquaRE)

ISO/IEC 9126-1:2001 (2001) Software engineering - product quality - part 1: quality model

Ivarsson M, Gorschek T (2011) A method for evaluating rigor and industrial relevance of technology evaluations. Empir Softw Eng 16(3):365–395. https://doi.org/10.1007/s10664-010-9146-4

Jacobs S (1999) Introducing measurable quality requirements: a case study. In: Proceedings IEEE international symposium on requirements engineering (Cat. No. PR00188). IEEE, pp 172–179

Jha N, Mahmoud A, Mining non-functional requirements from app store reviews. Empir Softw Eng 1–37. https://doi.org/10.1007/s10664-019-09716-7

Johansson E, Wesslén A, Bratthall L, Host M (2001) The importance of quality requirements in software platform development-a survey. In: Proceedings of the 34th annual hawaii international conference on system sciences. IEEE, p 10. https://doi.org/10.1109/HICSS.2001.927252

Jung HW (2007) Validating the external quality subcharacteristics of software products according to iso/iec 9126. Comput Stand Interfaces 29(6):653–661. https://doi.org/10.1016/j.csi.2007.03.004

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering

Kopczyńska S, Nawrocki J, Ochodek M (2018) An empirical study on catalog of non-functional requirement templates: usefulness and maintenance issues. Inf Softw Technol 103:75–91. https://doi.org/10.1016/j.infsof.2018.06.009

Kopczyńska S, Nawrocki J (2014) Using non-functional requirements templates for elicitation: a case study. In: 2014 IEEE 4th international workshop on requirements patterns (RePa). IEEE, pp 47–54. https://doi.org/10.1109/repa.2014.6894844

Kurtanović Z, Maalej W (2017) Automatically classifying functional and non-functional requirements using supervised machine learning. In: 2017 IEEE 25th international requirements engineering conference (RE). IEEE, pp 490–495. https://doi.org/10.1109/re.2017.82

Kusters RJ, van Solingen R, Trienekens JJ (1999) Identifying embedded software quality: two approaches. Qual Reliab Eng Int 15(6):485–492

Lawrence B, Wiegers K, Ebert C (2001) The top risk of requirements engineering. IEEE Softw 18(6):62–63. https://doi.org/10.1109/52.965804

Leung HK (2001) Quality metrics for intranet applications. Inf Manag 38(3):137–152. https://doi.org/10.1016/s0378-7206(00)00060-4

Lian X, Cleland-Huang J, Zhang L (2017) Mining associations between quality concerns and functional requirements. In: 2017 IEEE 25th international requirements engineering conference (RE). IEEE, pp 292–301. https://doi.org/10.1109/re.2017.68

Lochmann K, Fernández DM, Wagner S (2012) A case study on specifying quality requirements using a quality model. In: 19th Asia-Pacific software engineering conference, vol 1. IEEE, pp. 577–582. https://doi.org/10.1109/apsec.2012.57

Lu M, Liang P (2017) Automatic classification of non-functional requirements from augmented app user reviews. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering. ACM, pp 344–353. https://doi.org/10.1145/3084226.3084241

Maalej W, Nayebi M, Johann T, Ruhe G (2015) Toward data-driven requirements engineering. IEEE Softw 33(1):48–54. https://doi.org/10.1109/MS.2015.153

Mahmoud A, Williams G (2016) Detecting, classifying, and tracing non-functional software requirements. Requir Eng 21(3):357–381. https://doi.org/10.1007/s00766-016-0252-8

Mohagheghi P, Aparicio ME (2017) An industry experience report on managing product quality requirements in a large organization. Inf Softw Technol 88:96–109. https://doi.org/10.1016/j.infsof.2017.04.002

Mourão E, Pimentel JF, Murta L, Kalinowski M, Mendes E, Wohlin C (2020) On the performance of hybrid search strategies for systematic literature reviews in software engineering. Inf Softw Technol. https://doi.org/10.1016/j.infsof.2020.106294

Mourão E, Kalinowski M, Murta L, Mendes E, Wohlin C (2017) Investigating the use of a hybrid search strategy for systematic reviews. In: International symposium on empirical software engineering and measurement (ESEM). IEEE, pp 193–198. https://doi.org/10.1109/esem.2017.30

Mylopoulos J, Chung L, Nixon B (1992) Representing and using nonfunctional requirements: a process-oriented approach. IEEE Trans Softw Eng 6:483–497. https://doi.org/10.1109/32.142871

Niemelä E, Immonen A (2007) Capturing quality requirements of product family architecture. Inf Softw Technol 49(11–12):1107–1120. https://doi.org/10.1016/j.infsof.2006.11.003

Olsson T, Wnuk K, Gorschek T (2019) An empirical study on decision making for quality requirements. J Syst Softw 149:217–233. https://doi.org/10.1016/j.jss.2018.12.002

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A (2013) Software quality requirements: a systematic mapping study. In: 20th Asia-Pacific software engineering conference (APSEC), vol 1. IEEE, pp 231–238. https://doi.org/10.1109/apsec.2013.40

Paech B, Kerkow D (2004) Non-functional requirements engineering-quality is essential. In: 10th international workshop on requirements engineering foundation for software quality

Phillips LB, Aurum A, Svensson RB (2012) Managing software quality requirements. In: 38th euromicro conference on software engineering and advanced applications. IEEE, pp 349–356. https://doi.org/10.1109/seaa.2012.65

Poort ER, Martens N, Van De Weerd I, Van Vliet H (2012) How architects see non-functional requirements: beware of modifiability. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 37–51. https://doi.org/10.1007/978-3-642-28714-5_4

Rago A, Marcos C, Diaz-Pace JA (2013) Uncovering quality-attribute concerns in use case specifications via early aspect mining. Requir Eng 18(1):67–84

Rahimi M, Mirakhorli M, Cleland-Huang J (2014) Automated extraction and visualization of quality concerns from requirements specifications. In: IEEE 22nd international requirements engineering conference (RE). IEEE, pp 253–262. https://doi.org/10.1109/re.2014.6912267

Regnell B, Svensson RB, Olsson T (2008) Supporting roadmapping of quality requirements. IEEE Softw 25(2):42–47. https://doi.org/10.1109/ms.2008.48

Regnell B, Höst M, Svensson RB (2007) A quality performance model for cost-benefit analysis of non-functional requirements applied to the mobile handset domain. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 277–291. https://doi.org/10.1007/978-3-540-73031-6_20

Saito Y, Monden A, Matsumoto K (2012) Evaluation of non functional requirements in a request for proposal (rfp). In: Joint conference of the 22nd international workshop on software measurement and the 2012 seventh international conference on software process and product measurement. IEEE, pp 106–111. https://doi.org/10.1109/iwsm-mensura.2012.23

Shahrokni A, Feldt R (2013) Industrial challenges with quality requirements in safety critical software systems. In: 39th Euromicro conference on software engineering and advanced applications. IEEE, pp 78–81. https://doi.org/10.1109/seaa.2013.57

Sharma VS, Ramnani RR, Sengupta S (2014) A framework for identifying and analyzing non-functional requirements from text. In: Proceedings of the 4th international workshop on twin peaks of requirements and architecture. ACM, pp 1–8. https://doi.org/10.1145/2593861.2593862

Sibisi M, Van Waveren CC (2007) A process framework for customising software quality models. In: AFRICON. IEEE, pp 1–8. https://doi.org/10.1109/afrcon.2007.4401495

Silva A, Pinheiro PR, Albuquerque A, Barroso J (2017) Evaluation of an approach to define elicitation guides of non-functional requirements. IET Softw 11(5):221–228. https://doi.org/10.1049/iet-sen.2016.0302

Singh P, Singh D, Sharma A (2016) Rule-based system for automated classification of non-functional requirements from requirement specifications. In: 2016 International conference on advances in computing, communications and informatics (ICACCI). IEEE, pp 620–626. https://doi.org/10.1109/icacci.2016.7732115

Slankas J, Williams L (2013) Automated extraction of non-functional requirements in available documentation. In: 2013 1st international workshop on natural language analysis in software engineering (NaturaLiSE). IEEE, pp 9–16. https://doi.org/10.1109/naturalise.2013.6611715

Svensson RB, Gorschek T, Regnell B, Torkar R, Shahrokni A, Feldt R, Aurum A (2011) Prioritization of quality requirements: state of practice in eleven companies. In: IEEE 19th international requirements engineering conference. IEEE, pp 69–78. https://doi.org/10.1109/re.2011.6051652

Svensson RB, Gorschek T, Regnell B, Torkar R, Shahrokni A, Feldt R (2011) Quality requirements in industrial practice-an extended interview study at eleven companies. IEEE Trans Softw Eng 38(4):923–935. https://doi.org/10.1109/tse.2011.47

Tenbergen B, Daun M (2019) Is requirements-engineering research delivering what it promised?: a review of its accomplishments and opportunities after 10 years. IEEE Softw 36(4):6–11. https://doi.org/10.1109/ms.2019.2909127

Teruel MA, Navarro E, López-Jaquero V, Montero F, Jaen J, González P (2012) Analyzing the understandability of requirements engineering languages for cscw systems: a family of experiments. Inf Softw Technol 54(11):1215–1228. https://doi.org/10.1016/j.infsof.2012.06.001

Tóth L, Vidács L (2018) Study of various classifiers for identification and classification of non-functional requirements. In: International conference on computational science and its applications. Springer, pp 492–503. https://doi.org/10.1007/978-3-319-95174-4_39

Vanhanen J, Mäntylä MV, Itkonen J (2009) Lightweight elicitation and analysis of software product quality goals: a multiple industrial case study. In: 2009 third international workshop on software product management. IEEE, pp 42–52. https://doi.org/10.1109/iwspm.2009.5

Wagner S, Lochmann K, Heinemann L, Kläs M, Trendowicz A, Plösch R, Seidl A, Goeb A, Streit J (2012) The quamoco product quality modelling and assessment approach. In: Proceedings of the 34th international conference on software engineering. IEEE Press, pp 1133–1142. https://doi.org/10.1109/icse.2012.6227106

Wang T, Liang P, Lu M (2018) What aspects do non-functional requirements in app user reviews describe? an exploratory and comparative study. In: 2018 25th Asia-pacific software engineering conference (APSEC). IEEE, pp 494–503. https://doi.org/10.1109/apsec.2018.00064

Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng 11(1):102–107. https://doi.org/10.1007/s00766-005-0021-6

Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: International conference on evaluation and assessment in software engineering. p 38. https://doi.org/10.1145/2601248.2601268

Younas M, Jawawi D, Ghani I, Shah MA Extraction of non-functional requirement using semantic similarity distance. Neural Comput Appl 1–15. https://doi.org/10.1007/s00521-019-04226-5

Yu E (1997) Towards modelling and reasoning support for early-phase requirements engineering. In: International symposium on requirements engineering. pp 226–35. https://doi.org/10.1109/ISRE.1997.566873

Yusop N, Zowghi D, Lowe D (2008) The impacts of non-functional requirements in web system projects. Int J Value Chain Manag. https://doi.org/10.1504/ijvcm.2008.016116

Zhang W, Yang Y, Wang Q, Shu F (2011) An empirical study on classification of non-functional requirements. In: The twenty-third international conference on software engineering and knowledge engineering (SEKE 2011). pp 190–195

Zou J, Xu L, Yang M, Zhang X, Yang D (2017) Towards comprehending the non-functional requirements through developers eyes: an exploration of stack overflow using topic analysis. Inf Softw Technol 84:19–32. https://doi.org/10.1016/j.infsof.2016.12.003

Zubcoff J, Garrigós I, Casteleyn S, Mazón JN, Aguilar JA, Gomariz-Castillo F (2019) Evaluating different i*-based approaches for selecting functional requirements while balancing and optimizing non-functional requirements: a controlled experiment. Inf Softw Technol 106:68–84. https://doi.org/10.1016/j.infsof.2018.09.004

Download references

Open access funding provided by RISE Research Institutes of Sweden.

Author information

Authors and affiliations.

RISE Research Institutes of Sweden AB, Gothenburg, Sweden

Thomas Olsson & Efi Papatheocharous

Mälardalen University, Västerås, Sweden

Séverine Sentilles

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Thomas Olsson .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work is partially supported by a research grant for the ORION project (reference number 20140218) from The Knowledge Foundation in Sweden.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Olsson, T., Sentilles, S. & Papatheocharous, E. A systematic literature review of empirical research on quality requirements. Requirements Eng 27 , 249–271 (2022). https://doi.org/10.1007/s00766-022-00373-9

Download citation

Received : 10 April 2020

Accepted : 17 January 2022

Published : 08 February 2022

Issue Date : June 2022

DOI : https://doi.org/10.1007/s00766-022-00373-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic literature review
• Quality requirements
• Non-functional requirements
• Empirical evidence
• Find a journal
• Publish with us
• Track your research

• Search Menu
• Advance Articles
• Editor's Choice
• Supplements
• Open Access Articles
• Research Collections
• Review Collections
• Author Guidelines
• Submission Site
• Open Access Options
• Self-Archiving Policy
• About Health Policy and Planning
• About the London School of Hygiene and Tropical Medicine
• HPP at a glance
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, methodology, acknowledgements.

• < Previous

Health system resilience: a literature review of empirical research

• Article contents
• Figures & tables
• Supplementary Data

Louise Biddle, Katharina Wahedi, Kayvan Bozorgmehr, Health system resilience: a literature review of empirical research, Health Policy and Planning , Volume 35, Issue 8, October 2020, Pages 1084–1109, https://doi.org/10.1093/heapol/czaa032

• Permissions Icon Permissions

The concept of health system resilience has gained popularity in the global health discourse, featuring in UN policies, academic articles and conferences. While substantial effort has gone into the conceptualization of health system resilience, there has been no review of how the concept has been operationalized in empirical studies. We conducted an empirical review in three databases using systematic methods. Findings were synthesized using descriptive quantitative analysis and by mapping aims, findings, underlying concepts and measurement approaches according to the resilience definition by Blanchet et al . We identified 71 empirical studies on health system resilience from 2008 to 2019, with an increase in literature in recent years (62% of studies published since 2017). Most studies addressed a specific crisis or challenge (82%), most notably infectious disease outbreaks (20%), natural disasters (15%) and climate change (11%). A large proportion of studies focused on service delivery (48%), while other health system building blocks were side-lined. The studies differed in terms of their disciplinary tradition and conceptual background, which was reflected in the variety of concepts and measurement approaches used. Despite extensive theoretical work on the domains which constitute health system resilience, we found that most of the empirical literature only addressed particular aspects related to absorptive and adaptive capacities, with legitimacy of institutions and transformative resilience seldom addressed. Qualitative and mixed methods research captured a broader range of resilience domains than quantitative research. The review shows that the way in which resilience is currently applied in the empirical literature does not match its theoretical foundations. In order to do justice to the complexities of the resilience concept, knowledge from both quantitative and qualitative research traditions should be integrated in a comprehensive assessment framework. Only then will the theoretical ‘resilience idea’ be able to prove its usefulness for the research community.

Key Messages

The way in which resilience is currently applied in the empirical literature does not match its theoretical foundations.

In order to do justice to the complexities of the resilience concept, knowledge from both quantitative and qualitative research traditions should be integrated in a comprehensive assessment framework.

The word ‘resilience’ origins from the Latin prefix ‘re-’ (back) and the verb ‘salire’ (to jump, leap). In science, it has long been used by engineering and material science to describe the ability of a material to absorb energy without losing its original form or characteristics ( Hollnagel 2009 ). Over time, different disciplines adopted and adapted the term, adding different interpretations and facets to it: In ecology, resilience describes the persistence of ecological systems and measures a system’s ability to absorb changes of variables and maintain relationships between different populations ( Holling 1973 ). In psychology, resilience is understood as the individual human capability to cope with crises, losses or hardships without negative consequences ( Tugade and Fredrickson 2004 ).

In the last decade, the concept of resilience has also gained popularity in global public health. This development is reflected by major UN frameworks adopted in the last decade: The 2005–15 Hyogo Framework for Action ( UNISDR, 2005 ) was subtitled ‘Building the Resilience of Nations and Communities to Disasters’. Its successor, the 2015–30 Sendai Framework for Disaster Risk Reduction ( UNISDR, 2015 ), increases the focus on health in the disaster preparedness discourse and correspondingly calls for health resilience. Various sustainable development goals point to resilience as a target (1.5: ‘resilience of the poor’, 2.4 ‘resilient agricultural practices’, 11b ‘resilience to disasters’; Bahadur et al. 2015 ; UNISDR 2015 ) In a 2016 editorial of Bulletin of the World Health Organisation (WHO), health system resilience is named as a critical concept for global health, in the same vein as health system strengthening, universal health coverage and health security ( Kutzin and Sparkes 2016 ).

The shifting conceptualization of health system resilience

While definitions and concepts of health systems resilience differ substantially throughout the literature, all have a common core: they regard resilience as the degree of change a system can undergo while maintaining its functionality. The concept of resilience was introduced to the health systems literature from the ecological sciences through an increased understanding of health systems as complex adaptive systems ( Blanchet and James 2013 ). In this context, the idea of resilience, defined as ‘a measure of the amount of change a system can experience while maintaining the same controls on structure and function’ ( Blanchet and James 2013 ), can act as a useful tool to help us understand health system dynamics. The ecological idea that strategies to enhance resilience can be absorptive, adaptive or transformative depending on the impact and intensity of the crisis has been particularly impactful in the health system resilience discourse.

Popularized further during the Ebola crisis, health system resilience underwent a conceptual shift; from a mere ‘system’ capacity to recognizing the contribution of individuals and their agency within that system and acknowledging the wider social, economic and political context in which responses occur. Critics argued that the application of the resilience concept—as a ‘top-down’ approach—obscured important factors which prevented an adequate response to the Ebola crisis. They emphasized instead the importance of ‘understanding and reducing local power disparities, building the trustworthiness of health actors […] both between and during crisis’ which improves the ‘everyday functioning of the health system’ ( Martineau 2016 ). In response to these criticisms, Barasa et al. (2017) proposed the idea of ‘everyday resilience’, emphasizing in particular the importance of the capacities and resources available to individuals faced with delivering health services every day. Everyday resilience may especially be of relevance, they argued, in low- and middle-income countries where managers may ‘routinely face structural and policy instability, such as changes in governance structures, payment delays, abrupt and imposed policy directives […], unstable authority delegations, unpredictable staff and […] changing patient and community expectations’ ( Barasa et al. 2017 ).

Similarly, Blanchet et al. (2017) proposed a new model of understanding health systems resilience which focuses not just on the outcome of the resilience process (i.e. absorptive, adaptive and transformative capacities), but also on the underlying management capacities of the system and its actors to response to change: knowledge, uncertainties, interdependence and legitimacy ( Box 1 ). These operational dimensions are interlinked with each other and together characterize the management of resilience in health systems ( Figure 1 ). While these two more recent conceptualizations of resilience can be understood as different in terms of taking a ‘top-down’ ( Blanchet et al. 2017 ) and ‘bottom-up’ ( Barasa et al. 2017 ) approach, they both acknowledge the importance of the context in which the resilience process takes place and the agency of actors involved, and thus represent two sides of the same coin.

Management capacities:

Knowledge—‘Capacity to collect, integrate and analyse different forms of knowledge and information’

Uncertainties—‘Ability to anticipate and cope with uncertainties and surprises’

Interdependence—‘Capacity to manage interdependence: to engage effectively with and handle multiple- and cross-scale dynamics’

Legitimacy—‘Capacity to build or develop legitimate institutions that are socially accepted and contextually adapted’

Three levels of resilience:

Absorptive capacity—‘capacity of a health system to continue to deliver the same level (quantity, quality and equity) of basic healthcare services and protection to populations despite the shock using the same level of resources and capacities’

Adaptive capacity—‘capacity of the health system actors to deliver the same level of healthcare services with fewer and/ or different resources, which requires making organisational adaptations’

Transformative capacity—‘the ability of health system actors to transform the functions and structure of the health system to respond to a changing environment’

Conceptual overview of health system resilience, adapted from Blanchet et al. (2017) .

Conceptual influences from other fields

In addition to the conceptualization of resilience outlined above, other disciplinary fields have influenced the discourse on health system resilience, most notably the disaster management and healthcare quality literature.

In the disaster management sciences, resilience discussions were initially focused on the maintenance of infrastructure, functionality of health care facilities and continued service delivery ( Crowe et al. 2014 ; Balbus et al. 2016 ; Cimellaro et al. 2017 ) operationalizing resilience as ‘capability of a health system to mitigate the impact of major external disruptions on its ability to meet the needs of the population during the disaster’ ( Crowe et al. 2014 ). However, experiences of Hurricane Catrina in the USA shifted the dominant discourse in the disaster management literature to the concept of community resilience ( Wulff et al. 2015 ; Olu 2017 ). Community resilience proposes that the key to a good disaster response lies in communities, and their ability to ‘prepare, respond, and recover’ from major events through a range of measures including increased social connectedness, adaptive health and social systems and emergency preparedness planning ( Wulff et al. 2015 ).

A further prominent influence on the health system resilience discussions has been the concept of ‘resilience engineering’ or ‘health care resilience’, emerging from the healthcare quality literature. This approach, developed as a critique to traditional views of healthcare safety as an ‘absence of failures’, defines safety as the ‘ability to succeed under varying conditions’ ( Hollnagel et al. 2006 ). It thus focuses on nurturing the everyday functioning of healthcare teams and facilities to strengthen resilience and reduce clinical mistakes. A recent review on the topic has found that this approach has garnered significant attention in both the primary and secondary literature since its emergence around 2012 ( Ellis et al. 2019 ).

The need for a review of the empirical literature

Existing literature reviews have been conducted on the theoretical conceptualization of health system resilience ( Turenne et al. 2019 ) and the factors contributing to resilient health systems ( Barasa et al. 2018 ). The concept of resilience has also been extensively discussed outside the health sector ( Tanner et al. 2017 ). While grasping the theoretical background of the concept is certainly crucial, understanding how theory is translated into evidence is equally important for assessing the usefulness of the ‘resilience idea’ for the research community. However, so far there has been no critical appraisal of how the concept of health system resilience has been operationalized and applied in the empirical literature.

We thus conducted an empirical review of health system literature in order to better understand how the resilience concept has been operationalized in empirical studies. Within this research aim, we address three specific sub-questions: (1) What are the key aspects (methodological approach, geographic focus, health system building block addressed and crisis/challenge discussed) of research on health system resilience and how have these changed over time? (2) What concepts and frameworks on health system resilience have been used to operationalize resilience in the health systems literature? (3) What is the scope of empirical research on health system resilience within current definitions of the concept? We thus provide an overview of the existing empirical literature on health systems research which can be used to further develop the concept and inform its operationalization in future studies.

We conducted a review of empirical literature, following systematic review methodology in line with the understanding brought forward by Moher et al . (2015) . This included a systematic literature search, and a rigorous and systematic data screening and extraction process ( Peters et al. 2015 ).

Searches were conducted in Medline, Social Science Citation Index and CINAHL (Cumulative Index to Nursing and Allied Health Literature) using Resilien* AND a health system related terms (see Box 2 ).

Search terms:

((((((((secondary health care [mh]) OR primary health care [mh]) OR health services [mh]) OR delivery of health care [mh]) OR health services research [mh])) OR ((((((((((((““health system””) OR ““health systems””) OR ““health care system””) OR ““health care systems””) OR ““health care””) OR ““health care sector””) OR ““health care sectors””) OR ““health service””) OR ““health services””) OR ““service delivery””) OR ““health care service””) OR ““health care services””))) AND Resilien*

The searches were conducted on 18 October 2019 and were limited to articles published since 2008 in English or German language to keep the extent of the review feasible. The search produced 6136 publications for screening after the removal of 794 duplicates [see Figure 2 for the PRISMA flow diagram in line with Moher et al. (2009) ].

Prisma flow diagram.

Due to the high number of items, we used a three-stage screening process, eliminating non-relevant articles at the stage of title-, abstract- and full text-screening. Items were excluded if they did not report primary data, or were concerned with individual/psychological resilience including resilience of healthcare providers (e.g. nurses, physicians), resilience in the non-health space (e.g. social resilience, resilience of urban environments and resilience of biological systems), community resilience without link to health systems or articles that were concerned neither with health systems nor resilience. We also excluded articles which were concerned with health system resilience, but only used the term as a ‘buzzword’, without further definition, discussion or operationalization of the concept. As the research objective was to understand the application and use of resilience in health system research, items with any research design, geographic scope and health system focus were included.

After the abstract-screening stage, 517 references remained, with another 444 references excluded after screening full texts (see Figure 2 ). Both abstract- and full text-screening were carried out by the first and second author with joint synthesis until consensus was reached. Two further articles were from the reference lists of the literature reviews by Barasa et al. (2018) and Turenne et al. (2019) met the inclusion criteria for the present study and were included in the review. The remaining articles were divided into two categories: (1) those papers which specifically assessed health system resilience by including this as a specific research objective or applying a framework allowing for the operationalization of health system resilience (‘key papers’) and (2) articles reporting research which led to a discussion of health system resilience or how to achieve health system resilience.

Data extraction was carried out by the first and second author using Microsoft Excel. To answer the first research question on key aspects of the empirical health systems literature, data on type of research (primary/secondary research), discipline of the first author, the health system building block studied [according to World Health Organization (2010) ], the type of crisis or conflict studied, study location (country, continent, low-/middle-/high-income country), the organizational level being studied (e.g. global, national or regional) and type of data used were extracted from all identified studies.

To answer the second and third research objectives, only those studies directly measuring or assessing health system resilience (‘key papers’) were analysed. In order to evaluate the use of existing empirical frameworks in the empirical literature (second objective), information on frameworks used was extracted if these guided either the data collection or analysis process, or both. To further extract the scope of empirical research in terms of aspects or elements of the concept being addressed (third objective), we were guided by the conceptual framework of Blanchet et al. (2017) . We used this framework because it captures the various ways in which resilience is used in the empirical literature: it describes both the management capacities essential for a resilient system (management capacities: knowledge, uncertainties, interdependence, legitimacy) as well as those describing the outcome (three levels of resilience: absorptive, adaptive and transformative capacities). It thus is able to capture a broad range of research on post ex ante and ex post ( Béné et al. 2015 ) aspects of the resilience process. Research articles were classified within this framework using the definitions listed in Box 2.

We synthesized the findings by combining a narrative synthesis with descriptive quantitative analysis of key aspects addressed. We further tabulated and mapped aims, findings, underlying concepts and measurement approaches according to the resilience definition by Blanchet et al. (2017) . Indicators used to measure aspects of resilience in quantitative and mixed methods studies were also extracted and mapped according to their respective resilience domain and the level of data collection (national, organizational, staff or population/patient level).

A total of 71 articles met our inclusion criteria, comprising 40 research papers specifically measuring or addressing health system resilience and 31 discussing health system resilience using empirical research ( Figure 2 , see Supplementary file for full list of studies).

Quantitative synthesis and mapping of empirical literature in health system resilience

The literature was found to be fairly evenly distributed across continents: Africa ( n  = 18; 25%), Europe ( n  = 18; 25%), Asia ( n  = 15; 21%), North America ( n  = 15; 21%) and Australia ( n  = 2; 3%), with four studies reporting data across continents. The exception was South America, where no empirical papers were found. The majority of research was conducted in high-income countries ( n  = 37; 52%), with 18 studies (25%) in middle-income countries and 13 studies (18%) in low-income countries. We found an increase in literature in recent years (62% of studies published since 2017).

The majority of research ( n  = 58; 82%) addressed a specific crisis or challenge. Overall, infectious disease outbreaks was the most frequently addressed challenge ( n  = 14; 20%), followed by natural disasters ( n  = 11; 15%) and climate change ( n  = 8; 11%). Other challenges were conflicts ( n  = 4; 6%), migration ( n  = 4; 6%), financial crises ( n  = 2; 3%) and terrorist attacks ( n  = 1; 1%). Several articles addressed chronic, non-crisis-related challenges ( n  = 12; 17%): changes in team composition ( n  = 1; 1%), human error ( n  = 5; 7%), everyday resilience ( n  = 3; 4%) and structural change ( n  = 2; 3%). While non-crisis-related challenges and climate-related studies dominated the early records from 2008 to 2014, over time, the diversity of addressed challenges has grown embracing financial crises from 2013, infectious disease outbreaks from 2015 triggered by the Ebola epidemic and migration from 2017 (see Figure 3 ).

Identified literature on health system resilience ( N  = 71) organized by type of challenge and year (2008–19).

In terms of health system building blocks addressed, a large proportion of studies ( n  = 34; 48%) focused on service delivery, while 14 (20%) did not focus on a particular health system building block but took a general perspective. Other building blocks addressed frequently include leadership and governance ( n  = 9; 13%) and health workforce ( n  = 8; 11%), while health information systems ( n  = 4; 6%), medicines and access to medicines ( n  = 2; 3%) and health system financing ( n  = 1; 1%) are addressed less frequently.

Overall, the empirical studies identified differed in terms of their disciplinary tradition or conceptual background. Studies from the public health sciences tended to converge in three groups: (1) quantitative studies focusing on service delivery, making use of service utilization indicators provide an easily accessible measure to assess resilience before, during and after a crisis ( Paterson et al. 2014 ; Gizelis et al. 2017 ; Sochas et al. 2017 ; Kozuki et al. 2018 ; Ray-Bennett et al. 2019 ), (2) qualitative studies focusing on the health workforce, influenced by ideas of ‘everyday resilience’ and addressing the contributions of social connectedness and leadership on health system resilience ( Mash et al. 2008 ; Witter et al. 2017 ; Raven et al. 2018 ; Brooke-Sumner et al. 2019 ; Thude et al. 2019 ), and (3) studies taking a broad perspective of health system resilience, looking at multiple health system building blocks or aspects of a health system to assess resiliency ( Ager et al. 2015 ; Ammar et al. 2016 ; Fukuma et al. 2017 ; Ling et al. 2017 ; Meyer et al. 2018 ; Watts et al. 2018 ).

However, influences from outside the public health sciences could also be identified in the empirical health system resilience literature. As a relatively distinct influence, the disciplines of engineering and architecture have contributed empirical research assessing the infrastructure and thermal resilience of healthcare facilities and structures ( Lomas et al. 2012 ; Iddon et al. 2015 ; Short et al. 2015 ; Dippenaar and Bezuidenhout 2019 ). A further relatively distinct influence has been the contribution of specific checklists to assess facility and organizational resilience from the fields of disaster management and emergency preparedness ( Paterson et al. 2014 ; Zhong et al. 2014a , 2015 ; Dobalian et al. 2016 ; Khan et al. 2018 ; Meyer et al. 2018 ). Also from the field of disaster management, but perhaps more intertwined with resilience in the way it has been conceptualized in the health systems literature, are studies assessing community resilience and its relationship with service delivery during a crisis ( O’Sullivan et al. 2013 ; Andrew et al. 2016 ; Toner et al. 2017 ; Alonge et al. 2019 ; Cohen et al. 2019 ). Finally, hailing from the tradition of medical sciences concerned with patient safety and quality of care, concepts of ‘health care resilience’ or ‘resilience engineering’ have also influenced the empirical literature on health system resilience ( Brattheim et al. , 2011 ; Franklin et al. , 2014 ; Falegnami et al. 2018 ; Merandi et al. , 2018 ; Patriarca et al. , 2018 ). While study object of these studies is also the health workforce, the focus is placed on the analysis of work processes and the avoidance of medical errors to maintain functionality of services.

Methodological analysis of key empirical papers

We identified 40 high-relevance empirical studies specifically assessing health system resilience. Fifteen articles used a quantitative methodology ( Table 1 ), nine articles applied mixed methods ( Table 2 ) and a further 16 used qualitative methods ( Table 3 ). Given the distinction between articles in terms of their thematic focus described above, we present articles in six thematic areas: assessing national-level health system resilience in the context of a specific crisis ( n  = 8; 20%), assessing health service delivery ( n  = 10; 25%), addressing health workforce issues ( n  = 7; 18%), taking a community resilience perspective ( n  = 3; 7%), looking at infrastructure and thermal resilience ( n  = 3; 7%) and developing emergency preparedness checklists and assessment tools ( n  = 9; 23%).

Overview of aims, methods, concepts used and dimensions of resilience addressed by quantitative research papers ( n  = 15)

Overview of aims, methods, concepts used and dimensions of resilience addressed by mixed methods research papers ( n  = 9)

Overview of aims, methods, concepts used and dimensions of resilience addressed by qualitative research papers ( n  = 16)

Assessing national-level health system resilience in context of a specific crisis

Of the eight studies which assessed an entire, national health system in the context of a particular crisis, two studies took a purely quantitative approach: Fukuma et al. (2017) assessed Japan’s health system responsiveness and resilience after the Great East Japan Earthquake and Watts et al. (2018) assessed the resilience of 101 health systems in the context of climate change. Fukuma et al. (2017) operationalized resilience by using composite routine data indicators during the time of crisis, including: service utilization, cause-specific mortality rates incl. suicides, number of hospitals, health expenditures, human resources and immunization coverage. Watts et al. (2018) assessed resilience by surveying for the presence of specific policy efforts and strategies in the context of climate change at a national level.

Three further studies assessed the resilience of a health system at the country level using a mixed methods approach. Ammar et al. (2016) studied the Lebanese health system in the context of the Syrian refugee crisis using a case study approach. Orru et al. (2018) assessed the ways in which the Estonian health system was able to assess and manage the health risks of climate change using a combination of document review, expert interviews and population survey data as applied to the WHO Operational Framework for Building Climate Resilient Health Systems ( World Health Organization 2015 ). Thomas et al. (2013) assessed the performance of the Irish health system in the face of the economic crisis by applying quantitative indicators developed from their own resilience framework to government documents and supplementing these with semi-structured interviews.

Finally, three qualitative studies considered national health system resilience. Ager et al. (2015) assessed key barriers to the provision of responsive service in the context of Boko Haram in Nigeria, Alameddine et al. (2019) assessed the resilience of Lebanon and Jordan’s health systems in the context of the Syrian crisis and Ling et al. (2017) assessed the resilience of Liberia’s health system during the Ebola crisis. All three studies used semi-structured interviews with health professionals and other key health stakeholders for data collection, with Ling et al. (2017) complementing these with focus group discussions.

Out of these eight studies, all studies except one ( Fukuma et al. 2017 ) applied a specific conceptual framework to study resilience. However, the frameworks used in the other studies vary, including frameworks developed by international or development agencies, such as the World Health Organization or the United Kingdom’s Department for International Development ( Ager et al. 2015 ; Orru et al. 2018 ; Watts et al. 2018 ), general health system frameworks ( Ammar et al. 2016 ) and resilience frameworks developed in the academic literature ( Thomas et al. 2013 ; Ling et al. 2017 ; Alameddine et al. 2019 ).

Assessing resilience of health service delivery

Ten studies focused on the resilience of health service delivery. Six studies assessed the delivery of emergency services, three focused on the delivery of maternal health services, while one considered the continuity of a community health worker programme.

Two quantitative studies from the USA take a specific look at the delivery of emergency services: Radcliff et al. (2018) analyse ambulatory care measures during and after a storm, while Simonetti et al. (2018) model the potential of the US blood supply system during an emergency. Both make use of available administrative data, with Radcliff et al. (2018) relying in utilization data from Veterans Affairs clinics and Simonetti et al. (2018) using data on the national availability of blood stocks. The provision of emergency services during a crisis is also explored in four qualitative studies. Two of these assess service provision in the context of a particular crisis: Ridde et al. (2016) describe the emergency response to the Ouagadougou Terrorist attack in Burkina Faso, using a mixture of observations and expert interviews as their data source and structuring insights around Kruk et al. ’s (2017) ‘resilience indicators’ framework. Landeg et al. (2019) assess the emergency response to localized flooding in the UK using semi-structured interviews with decision-makers and document analysis. Finally, two qualitative studies explore the functionality of emergency service processes: while Back et al. (2017) use policy analysis and observation to examine escalation policies in UK hospitals, Errett et al. (2019) use semi-structured interviews with key informants to identify the impact of disruption of maritime transportation on the provision of emergency services during a disaster.

Being the only purely quantitative study to do so, Sochas et al. (2017) analysed the utilization of reproductive, maternal and neonatal health services in Sierra Leone in the context of the Ebola crisis using antenatal health service utilisation data. Gizelis et al. (2017) also assessed the impact of the Ebola epidemic on maternity delivery services, using a mixed methods approach by complementing maternity service utilization data from population surveys with semi-structured interviews and focus group discussions. Ray-Bennett et al. (2019) looked at the provision of reproductive health services in the context of flooding in Bangladesh, applying a structured facility assessment tool complemented by structured interviews with patients.

Kozuki et al. (2018) use a process evaluation methodology to document the ability of an integrated community case management programme to continue operation during the active conflict of 2013 and 2014 in South Sudan. The authors use routine programme data, including reporting, supervision, contact, treatment and referral rates, as well as interviews and focus groups with key stakeholders to evaluate the programme’s resiliency.

Only two of these studies ( Ridde et al. 2016 ; Back et al. 2017 ), both using qualitative methodologies, apply a specific framework of health system resilience. All quantitative studies and one mixed methods study focus on the absorptive capacities of service delivery, while the other studies address a more varied set of resilience dimensions.

Health workforce issues

A total of seven studies were identified which address aspects of health workforce resilience. These include studies both from the tradition of ‘resilience engineering’, as well as research influenced by the concept of ‘everyday resilience’.

One quantitative and one mixed methods study were conducted in the field of resilience engineering and safety research. Falegnami et al. (2018) surveyed the resilience of anaesthesia professionals in different work conditions in Italy using the four cornerstones of resilience framework ( Hollnagel 200 9). In the same setting, Patriarca et al. (2018) applied the functional resonance analysis method to explore the potential of the tool in enhancing the resilience of anaesthesia practices, drawing on documentary studies, interviews, observations and patient pathway modelling to do so.

Three studies considered health workforce issues on the context of a specific crisis. Applying a mixed methods approach, Witter et al. (2017) explored the impact of shocks on the health workforce across different contexts in Uganda, Sierra Leone, Zimbabwe and Cambodia, with a particular focus on vulnerabilities and coping strategies employed. The authors employed a mixture of methods for analysis, including surveys, human resource data, document review and qualitative interviews. Also taking a cross-national perspective, Raven et al. (2018) conducted observations and in-depth interviews with healthcare workers and management in Sierra Leone during the time of the Ebola crisis and in Nepal during a major earthquake to explore coping strategies of staff in both settings. In Portugal, Russo et al. (2016) explored physician’s perceptions of the changes in their work environment during the economic crisis in semi-structured interviews.

Finally, two qualitative studies take an ‘everyday resilience’ perspective to understand the ability of health workers in dealing with everyday challenges. Comparing experiences in Kenya and South Africa, Gilson et al. (2017) synthesize information from documents, interviews, group discussions and observations to understand factors influencing everyday resilience of staff. In Denmark, Thude et al. (2019) conducted semi-structured interviews with healthcare staff to explore the resilience of the workforce faced with challenges in their work environment, including changing leadership structures.

Only two of these studies ( Gilson et al. 2017 ; Falegnami et al. 2018 ) make use of an explicit resilience framework in their analysis. The dimensions assessed in individual studies varies: while Gilson et al. (2017) and Raven et al. (2018) explore a broad range of management capacities and resilience outcomes, the other five studies focus on only one of these two aspects, with three studies restricted in their analysis to a single outcome dimension ( Witter et al. 2017 ; Patriarca et al. 2018 ; Thude et al. 2019 ).

Taking a community resilience perspective

Three studies approached health system resilience from a community perspective. Cohen et al. (2019) quantitatively analyse the relationship between community resilience and the public’s confidence in the availability of healthcare services during emergency situations in Israel. Data for this study were conducted using the conjoint community resilience assessment measurement tool ( Leykin et al. 2013 ) in a household survey. Alonge et al. (2019) apply a qualitative approach to understand the relationship between community resilience and health system resilience. Combining information from key informant interviews and a national stakeholder meeting, they look at the contribution of responsible leadership and social capital into the resilience of the health system during the Ebola outbreak in Liberia. Finally, Andrew et al. (2016) take a slightly different approach to the issue of community resilience, by focusing on the resilience of community organizations involved with the relief efforts in the aftermath of the Thailand floods in 2011. Applying Bruneau et al. (2003) framework on the seismic resilience of communities, the authors quantitatively assess whether the bonding or the bridging effect made a larger contribution on the ability of organizations to deliver essential services after the crisis.

Both quantitative community resilience studies made use of an explicit framework for their analysis, while the qualitative study did not. While one study ( Cohen et al. 2019 ) focused entirely on dimensions of resilience management capacities, the other two studies explored a mix of management capacities and outcomes.

Infrastructure and thermal resilience

Three studies assessed the infrastructure and thermal resilience of hospitals, taking a purely quantitative approach. Resilience in this context is understood as the capability of buildings to withstand extreme conditions such as heat or earthquakes. Iddon et al. (2015) , Lomas et al. (2012) and Short et al. (2015) assessed the thermal resilience for specific building styles of wards in the UK in order to ensure climate change resiliency. None of these studies used specific conceptual frameworks for their analysis. In terms of the dimensions of resilience addressed, they focused entirely on dimensions of outcome, rather than management capacities. All three studies considered ways in which hospital infrastructure was able to absorb temperature changes, with two studies additionally assessing the potential for adaptation in response to these changes.

Development of preparedness checklists and assessment tools

A total of nine articles described the development of checklists to prepare for future catastrophic events or tools with which such preparedness can be measured. These have been developed at different levels: six studies focused on healthcare facilities and hospitals, two studies considered communities, while one study developed a conceptual framework at the national level.

Four articles described the quantitative development of checklists or measurement tools for assessing resilience of healthcare facilities. Dobalian et al. (2016) developed a general hospital preparedness tool, while Zhong et al. (2015) developed a framework for measuring hospital resilience and applied it to 41 tertiary care hospitals in a province in China ( Zhong et al. 2014a ). Goncalves et al. (2019) adapted and validated the short-form version of the Benchmark resilience tool for assessing the resilience of healthcare organizations. Using a mixed methods approach, Paterson et al. (2014) developed a toolkit for assessing the resiliency of healthcare facilities in the context of climate change. The methods for development differ: while Zhong et al. (2015) and Paterson et al. (2014) , respectively, used a Delphi consultation and workshops for an expert evaluation of proposed domains, Dobalian et al. (2016) , Goncalves et al. (2019) and Zhong et al. (2014a ) used psychometric assessments to assess validity and reliability. One of the instruments was operationalized as a survey of workers ( Goncalves et al. 2019 ), while the other three carried out assessments at the organizational level—either by external evaluation ( Dobalian et al. 2016 ), as a survey completed by managers of the facility ( Zhong et al. 2014a , 2015 ) or as a toolkit for facilities aiming to improve their climate resiliency ( Paterson et al. 2014 ). Finally, Meyer et al. (2018) conduct semi-structured interviews with key informants involved in the Ebola response in the USA to develop an actionable checklist to enable preparedness for future responses.

Two further papers used a qualitative approach to develop checklist for enhancing community resilience in a health system context. O’Sullivan et al. (2013 ) identify levers to promote community resilience for health during disasters using a community-based participatory research approach. Toner et al. (2017) used experiences from Hurricane Sandy collected through key informant interviews and focus groups to develop a checklist for assessing and strengthening communities’ health sector resilience.

Finally, Khan et al. (2018) conducted focus groups to develop a framework comprising of essential elements of a resilient public health system in during emergencies, using the lens of complex adaptive health systems. They discuss the importance of recognizing the interconnectedness of actors and processes during an emergency response, acknowledging that these dimensions, while crucial, are particularly difficult to measure and quantify.

Many of these studies understandably did not use a specific resilience framework, as part of the research aim was to develop key dimensions of resilience in a particular context. However, three studies did use frameworks to guide the selection of their proposed dimensions ( Zhong et al. 2014a , 2015 ; Goncalves et al. 2019 ) or the development of topics for discussion in focus groups ( O’Sullivan et al. 2013 ). Checklists tended to focus on measuring the management capacities of facilities, organizations and systems, with a noticeable trend towards a more diverse set of dimensions among the qualitative studies. Only two studies ( O’Sullivan et al. 2013 ; Khan et al. 2018 ) considered assessment of the system’s ability for absorption and adaptation.

Conceptual analysis of key empirical studies

Conceptual frameworks used.

Across the empirical studies, a specific framework for assessing resilience was used by four quantitative studies, two mixed methods studies and seven qualitative studies. The types and disciplinary origins of the frameworks differed widely. Of the concepts developed in the health systems resilience discourse, the ‘resilience index’ framework ( Kruk et al. 2017 ), ‘resilience capacities’ framework ( Blanchet et al. 2017 ) and ‘everyday resilience’ framework ( Barasa et al. 2017 ) were used. From the resilience engineering discourse, the Concepts for Applying Resilience Engineering (CARE) model ( Anderson et al. 2016 ) and the Four Cornerstones of Resilience framework ( Hollnagel 2009 ) were applied. Notably, three frameworks from the area of community resilience were used: CCRAM model ( Leykin et al. 2013 ), framework to assess seismic resilience of communities ( Bruneau et al. 2003 ) and the resilient communities framework ( Norris et al. 2008 ). Other frameworks used included the UK government’s humanitarian policy ( DfID 2011 ) and the WHO Operational Framework for Building Climate Resilient Health Systems ( World Health Organization 2015 ). Only two frameworks ( Hollnagel 2009 ; Kruk et al. 2017 ) were used twice, all other studies used distinctive frameworks for their analysis.

Dimensions of resilience addressed

We used the framework formulated by Blanchet et al. (2017) as an analytical lens allows for a more in-depth analysis of the content and dimensions of resilience addressed across the empirical papers using the definitions of management capacities and levels of resilience provided in Box 1. Across the empirical papers, 12 studies focused exclusively on resilience domains in the ex ante ‘management capacities’ side of Blanchet et al. ’s resilience definition, while 14 studies focused exclusively on absorptive, adaptive or transformative levels of the resilience process. Fourteen studies considered both management capacities and resilience levels. Qualitative studies more often considered both management capacities and resilience levels, while quantitative studies more often exclusively focused on one of the two ( Figure 4a ). Among the management capacities, the dimension of ‘uncertainty’ was most frequently assessed by all types of research, followed by dimensions of ‘interdependence’, ‘knowledge’ and ‘legitimacy’, in that order ( Figure 4b ).

(a–c) Domains of resilience addressed by key papers ( n  = 40), by research methodology.

Among the ex post resilience levels, ‘absorptive capacities’ was most frequently addressed across research types, although qualitative research explored ‘absorptive capacities’ and ‘adaptive capacities’ to an equal extent ( Figure 4c ). Only a limited number of quantitative and mixed methods studies considered the ‘adaptive capacities’ and ‘transformative capacities’ dimensions of health system resilience.

Looking across management capacities and resilience levels, qualitative research was able to address a much broader range of dimensions than quantitative research, with individual studies often exploring multiple dimensions of the resilience concept ( Figure 4b and 4c ).

Quantitative indicators used

A total of 24 studies used quantitative indicators to measure different aspects of the resilience concept, with several studies using multiple indicators across multiple domains of responsiveness ( Table 4 ). The reported indicators were collected using different data collection strategies, including the use of routine data, observational data and primary survey data. The indicators further differed in the level at which data were collected, spanning national, organizational, staff and patient/population levels. Across the ‘management capacities’ domains, several indicators at different levels of data collection addressed the domains of knowledge, uncertainties and interdependence. However, only two indicators, both collected at population level, captured the legitimacy dimension. Across the ‘levels of resilience’ domains, several studies used indicators across different levels of data collection for the ‘absorption’ domain. However, only three indicators were used for the ‘adaptation’ domain, collected at national and organizational level, while no indicators were identified for the ‘transformation’ domain.

Resilience indicators used in quantitative and mixed methods studies ( n  = 24), by resilience domain and level of data collection

Routine data, document review or observation;.

Survey data.

The concept of health system resilience has soared in popularity in the health system field over the last years, not just in the theoretical or political discourse but also as an object of empirical inquiry. Its application has been incredibly diverse, with research from different disciplines applying the concepts in different healthcare sectors and in various settings. This diversity is not itself problematic. However, this review has demonstrated that empirical studies fundamentally differ in the way that resilience is understood in a health system context.

In terms of the content of the studies, much empirical research focuses on service delivery, health workforce or governance issues, whereas resilience of other health system building blocks is either barely studied, such as health financing, or only studied in high-income countries, as is the case of health information systems. This shows a distinct gap between the concepts and the operationalization of resilience in the context of health system research. If research on health system resilience is to live up to recent comprehensive definitions, the focus has to widen: all building blocks are interlinked and essential for well-functioning health systems, and should therefore not be analysed in singularity, but be considered jointly when assessing health system resilience.

Furthermore, despite much theoretical work on the dimensions which constitute health system resilience, we found that most of the empirical literature only addressed particular aspects. Applying the dimensions outlined by Blanchet et al. (2017) , we found that the importance of developing legitimate institutions appears to be neglected in empirical research. This is particularly concerning given that a lack of in healthcare institutions has recently emerged as one of the key barriers to the continued functioning of the health system, e.g. in the context of the Ebola outbreak ( Kittelsen and Keating 2019 ). The ability of health systems to demonstrate transformative capacities has been similarly under-evaluated, especially in quantitative research. Very few empirical studies took an approach to resilience that takes into account the various nuances in the conceptualization of the term which have recently emerged. This trend appeared to be particularly pronounced in those studies with a quantitative or mixed methods approach.

Thus, there is a mismatch between the conceptual models of health system resilience and the way resilience is understood and applied in empirical research both in terms of the breadth of health system factors considered and in terms of the resilience dimensions which are taken into account. Part of the issue may be that the empirical literature assessed in this review comes from a broad range of disciplines, with differing traditions of how ‘resilience’ is understood. While different traditions can offer unique and potentially complementary perspectives on the topic of resilience, this underlines the importance of more clarity in the empirical literature about which concepts and definitions are applied, and how these are then operationalized.

However, only very few empirical studies make use of an explicit conceptual framework for collection or analysis of data, thus not linking research objectives to the rich theoretical body of work on how resilience can be understood in a health system context. Arguably, those studies assessing resilience at a national level were most cognizant of using conceptual frameworks for their analysis. Our review showed that these studies were best able to capture the multiple dimensions of health system resilience. While several other studies aimed to measure health system resilience, they subsequently operationalized this concept in a very narrow way, e.g. by measuring only health service utilization, infrastructure resilience or emergency preparedness. Encouraging the use of an explicit framework for health system resilience could help to strengthen the links between the conceptualization and the operationalization of resilience, thus improving our understanding of health system resilience in different contexts and settings.

Our review further demonstrates that qualitative articles tend to employ a more comprehensive approach to the resilience concept than quantitative studies, which are often limited by availability of data and indicators to few aspects of resilience. The mismatch between concepts and research, therefore, appears to lie not in a lack of appreciation for the complexities of the resilience concept, but rather in a lack of measurable indices which reflect this complexity. While the proposed resilience index ( Kruk et al. 2017 ) specifies a list of potentially measurable indicators, so far these have only been operationalized in qualitative research. Similarly, the ‘resilience capacities’ framework specified by Blanchet et al. (2017) , and the ‘everyday resilience’ framework by Barasa et al. (2017) have been operationalized exclusively in qualitative research. All identified quantitative studies have utilized frameworks originating in discourses tangential to the health system resilience discourse.

Yet so far there has been no discussion about which aspects of the health system resilience frameworks are actually measurable. Within the ‘resilience capacities’ framework, the identified studies demonstrate that it is possible to measure ‘absorptive’ aspects by comparing levels of service provision and utilization in different circumstances. However, this is more challenging for ‘adaptive’ and ‘transformative’ aspects. Understanding whether a health system has truly transformed itself in response to a challenge needs to take into account multiple contextual factors and thus lends itself more naturally to be answered by qualitative methods and policy analysis, but also to complexity science. Equally, studies were able to quantitatively assess the presence or absence of preparedness plans to deal with uncertainties and data collection mechanisms for an improved knowledge of potential challenges, but quantifying the ability to handle cross-scale dynamics and develop legitimate institutions proved to be more difficult to capture. Incidentally, the identified studies developing resilience checklists and measurement tools all took a very narrow perspective of resilience by focusing on single healthcare facilities and organizations.

The key question in the development of a comprehensive resilience index, or a measure that allows for effective combination of quantitative and qualitative aspects, becomes whether the requirements to create a comparable measurement tool can be reconciled with the very broad and comprehensive definition of resilience which has emerged from an understanding of health systems as complex adaptive systems. According to Haldane et al. (2017) the resilience concept ‘should […] not be prescriptive, but have breadth and flexibility, recognize complexity, consider shocks and cumulative stresses, attempt to deal with disruptions and anticipate future failures’. It appears that, so far, the qualitative literature has been more successful in translating such a comprehensive framework into research practice, while quantitative studies have been limited both by theoretical models and a lack of appropriate data with which to measure resilience. Thus a key task for future researchers in the resilience field will be not only how the resilience concept can be operationalized, but—acknowledging that quantitative assessment of resilience in its entirety is illusionary—determine how measureable aspects can be combined with qualitative aspects in a way that allows for an assessment of health system resilience as a dynamic, complex phenomenon. Thus further research is required for the development of an operational framework on health system resilience which seamlessly integrates both qualitative and quantitative evidence; knowledge from existing guidelines on integrating quantitative and qualitative knowledge, e.g. in the realm of assessing the effectiveness of complex interventions, could be utilized for this purpose ( Noyes et al. 2019 ).

Our review adds to the existing conceptual review by Turenne et al. (2019) , who argue that the concept of health system resilience is still in infancy. We demonstrate the implications of this conceptual immaturity on existing empirical research: while the qualitative literature has explored the notion of health system resilience in its broad definition, the quantitative literature has been limited by the lack of clearly defined characteristics, preconditions and limits of the concept.

Our review makes a substantial contribution to the health systems research literature by analysing the operationalization of the health system resilience concept in empirical studies. Due to our inclusive search and broad inclusion criteria, we were able to consider a broad range of relevant articles from multiple disciplines and thus demonstrate the influence of other disciplines in the health systems research field. However, as search terms were geared to finding articles which specifically referred to the resilience concept, we may have missed empirical studies which operationalized aspects of resilience, but used different terminology. Further research could specifically identify such studies by using elements of the resilience definition instead of merely using the term itself. This could also help to better gain an understanding of how the concept of resilience overlaps with other health systems concepts such as health system strengthening or health system responsiveness and map potential synergies in assessment. We also did not include secondary research or grey literature in our review, which may provide further useful information on the operationalization of the resilience concept. Further research is needed to combine and integrate knowledge from these diverse sources in a comprehensive assessment framework.

A further limitation of our study is the initial exclusion of items based on titles, which was necessary due to the sheer number of results. This may have excluded several studies in associated disciplines, such as those relating to community resilience, which are of importance to the health systems resilience discourse. Findings of our review should be complemented by reviews of the resilience concept in other disciplines to check for congruence.

The health systems research community has made substantial advances in the conceptualization of health system resilience and its potential for the analysis of health systems in changing environments. However, the empirical literature has not yet caught up with the complexities of the concept: there is a mismatch between the nuances and the breadth of the concept at a theoretical level and the way it has been operationalized in empirical studies. In order to do justice to the complexities of the resilience concept, knowledge from both quantitative and qualitative research traditions should be integrated in a way that resilience as a complex, adaptive phenomenon. Only once a comprehensive assessment framework has been defined and applied across different research contexts will the theoretical ‘resilience idea’ be able to more convincingly prove its usefulness for the research community.

The authors are supported by a grant from the German Federal Ministry for Education and Research (BMBF) in the scope of the project RESPOND (Grant number: 01GY1611). Further grants are received by the German Science Foundation (DFG) in the scope of the Research Unit PH-LENS and its subproject NEXUS (Grant no: FOR 2928 / BO 5233/1-1). The funders had no influence on study design, analysis or decision to publish.

Conflict of interest statement . None declared.

Ethical approval. No ethical approval was required for this study.

Ager AK , Lembani M , Mohammed A et al.  2015 . Health service resilience in Yobe state, Nigeria in the context of the Boko Haram insurgency: a systems dynamics analysis using group model building . Conflict and Health 9 : 30 .

Google Scholar

Alameddine M , Fouad FM , Diaconu K et al.  2019 . Resilience capacities of health systems: accommodating the needs of Palestinian refugees from Syria . Social Science & Medicine 220 : 22 – 30 .

Alonge O , Sonkarlay S , Gwaikolo W et al.  2019 . Understanding the role of community resilience in addressing the Ebola virus disease epidemic in Liberia: a qualitative study (community resilience in Liberia) . Global Health Action 12 : 1662682 .

Ammar W , Kdouh O , Hammoud R et al.  2016 . Health system resilience: Lebanon and the Syrian refugee crisis . Journal of Global Health 6 : 020704 .

Anderson J , Ross A , Back J et al.  2016 . Implementing resilience engineering for healthcare quality improvement using the CARE model: a feasibility study protocol . Pilot and Feasibility Studies 2 : 61 .

Andrew S , Arlikatti S , Siebeneck L , Pongponrat K , Jaikampan K. 2016 . Sources of organisational resiliency during the Thailand floods of 2011: a test of the bonding and bridging hypotheses . Disasters 40 : 65 – 84 .

Back J , Ross AJ , Duncan MD et al.  2017 . Emergency department escalation in theory and practice: a mixed-methods study using a model of organizational resilience . Annals of Emergency Medicine 70 : 659 – 71 .

Bahadur A , Lovell E , Wilkinson E , Tanner T. 2015 . Resilience in the SDGs: Developing an Indicator for Target 1.5 That Is Fit for Purpose . London: Overseas Development Institute Briefing.

Balbus J , Berry P , Brettle M et al.  2016 . Enhancing the sustainability and climate resiliency of health care facilities: a comparison of initiatives and toolkits . Revista Panamericana de Salud Publica = Pan American Journal of Public Health 40 : 174 – 80 .

Barasa E , Mbau R , Gilson L. 2018 . What is resilience and how can it be nurtured? A systematic review of empirical literature on organizational resilience . International Journal of Health Policy and Management 7 : 491 – 503 .

Barasa EW , Cloete K , Gilson L. 2017 . From bouncing back, to nurturing emergence: reframing the concept of resilience in health systems strengthening . Health Policy and Planning 32 : iii91 – 4 .

Béné C , Frankenberger T , Nelson S. 2015 . Design, monitoring and evaluation of resilience interventions: conceptual and empirical considerations. IDS working Paper 459. Brighton: Institute of Development Studies.

Béné C , Godfrey-Wood R , Newsham A , Davies M. 2012 . Resilience: new utopia or new tyranny?—reflection about the potentials and limits of the concept of resilience about vulnerability reduction programmes. IDS working Paper 405. Brighton: Institute of Development Studies.

Blanchet K , James P. 2013 . The role of social networks in the governance of health systems: the case of eye care systems in Ghana . Health Policy and Planning 28 : 143 – 56 .

Blanchet K , Nam SL , Ramalingam B , Pozo-Martin F. 2017 . Governance and capacity to manage resilience of health systems: towards a new conceptual framework . International Journal of Health Policy and Management 6 : 431 – 5 .

Brattheim B , Faxvaag A , Seim A. 2011 . Process support for risk mitigation: a case study of variability and resilience in vascular surgery . BMJ Quality & Safety 20 : 672 – 9 .

Brooke-Sumner C , Petersen-Williams P , Kruger J , Mahomed H , Myers B. 2019 . ‘ Doing more with less’: a qualitative investigation of perceptions of South African health service managers on implementation of health innovations . Health Policy and Planning 34 : 132 – 40 .

Bruneau M , Chang SE , Eguchi RT et al.  2003 . A framework to quantitatively assess and enhance the seismic resilience of communities . Earthquake Spectra 19 : 733 – 52 .

Cimellaro GP , Malavisi M , Mahin S. 2017 . Using discrete event simulation models to evaluate resilience of an emergency department . Journal of Earthquake Engineering 21 : 203 – 26 .

Cohen O , Shapira S , Aharonson-Daniel L , Shamian J. 2019 . Confidence in health-services availability during disasters and emergency situations-does it matter?—lessons learned from an Israeli population survey . International Journal of Environmental Research and Public Health 16 : 3519 .

Crowe S , Vasilakis C , Skeen A et al.  2014 . Examining the feasibility of using a modelling tool to assess resilience across a health-care system and assist with decisions concerning service reconfiguration . Journal of the Operational Research Society 65 : 1522 – 32 .

DFID 2011 . Saving Lives, Preventing Suffering and Building Resilience: The UK Government’s Humanitarian Policy . London : DfID .

Google Preview

Dippenaar A , Bezuidenhout T. 2019 . The development of a robust risk management plan for the continuous supply of water to hospitals in the Western Cape Province . South African Journal of Industrial Engineering 30 : 190 – 204 .

Dobalian A , Stein JA , Radcliff TA et al.  2016 . Developing valid measures of emergency management capabilities within us department of veterans affairs hospitals . Prehospital and Disaster Medicine 31 : 475 – 84 .

Ellis LA , Churruca K , Clay-Williams R et al.  2019 . Patterns of resilience: a scoping review and bibliometric analysis of resilient health care . Safety Science 118 : 241 – 57 .

Errett NA , Tanner A , Shen X , Chang SE. 2019 . Understanding the impacts of maritime disruption transportation to hospital-based acute health care supplies and personnel in coastal and geographically isolated communities . Disaster Medicine and Public Health Preparedness 13 : 440 – 8 .

Falegnami A , Bilotta F , Pugliese F et al.  2018 . A multicountry comparative survey about organizational resilience in anaesthesia . Journal of Evaluation in Clinical Practice 24 : 1347 – 57 .

Franklin BD , Panesar SS , Vincent C , Donaldson LJ. 2014 . Identifying systems failures in the pathway to a catastrophic event: an analysis of national incident report data relating to vinca alkaloids . BMJ Quality & Safety 23 : 765 – 72 .

Fukuma S , Ahmed S , Goto R et al.  2017 . Fukushima after the Great East Japan Earthquake: lessons for developing responsive and resilient health systems . Journal of Global Health 7 : 010501 .

Gilson L , Barasa E , Nxumalo N et al.  2017 . Everyday resilience in district health systems: emerging insights from the front lines in Kenya and South Africa . BMJ Global Health 2 : e000224 .

Gizelis TI , Karim S , Ostby G , Urdal H. 2017 . Maternal health care in the time of ebola: a mixed-method exploration of the impact of the epidemic on . World Development 98 : 169 – 78 .

Goncalves L , Navarro JB , Sala R. 2019 . Spanish validation of the Benchmark Resilience Tool (short-form version) to evaluate organisational resilience . Safety Science 111 : 94 – 101 .

Haldane V , Ong SE , Chuah FL , Legido-Quigley H. 2017 . Health systems resilience: meaningful construct or catchphrase? . Lancet (London, England) 389 : 1513 .

Holling CS. 1973 . Resilience and stability of ecological systems . Annual Review of Ecology and Systematics 4 : 1 – 23 .

Hollnagel E. 2009 . The four cornerstones of resilience engineering. In: Nemeth CP, Hollnagel E, Dekker SWA (eds). Resilience Engineering Perspectives , Vol. 2: Preparation and Restoration Farnham, UK : Ashgate.

Hollnagel E , Woods DD , Leveson N. 2006 . Resilience Engineering: Concepts and Precepts . Aldershot, UK: Ashgate .

Iddon CR , Mills TC , Giridharan R , Lomas KJ. 2015 . The influence of hospital ward design on resilience to heat waves: an exploration using distributed lag models . Energy and Buildings 86 : 573 – 88 .

Kailes JI , Enders A. 2007 . Moving beyond “special needs” A function-based framework for emergency management and planning . Journal of Disability Policy Studies 17 : 230 – 7 .

Khan Y , O’Sullivan T , Brown A et al.  2018 . Public health emergency preparedness: a framework to promote resilience . BMC Public Health 18 : 1344 .

Kittelsen SK , Keating VC. 2019 . Rational trust in resilient health systems . Health Policy and Planning 34 : 553 – 7 .

Kozuki N , Ericson K , Marron B , Lainez YB , Miller NP. 2018 . The resilience of integrated community case management in acute emergency: a case study from Unity State, South Sudan . Journal of Global Health 8 : 020602 .

Kruk ME , Ling EJ , Bitton A et al.  2017 . Building resilient health systems: a proposal for a resilience index . BMJ (Clinical Research ed.) 357 : j2323 .

Kruk ME , Myers M , Varpilah ST , Dahn BT. 2015 . What is a resilient health system? The Lancet 385 : 1910 – 2 .

Kutzin J , Sparkes SP. 2016 . Health systems strengthening, universal health coverage, health security and resilience . Bulletin of the World Health Organization 94 : 2 .

Landeg O , Whitman G , Walker-Springett K et al.  2019 . Coastal flooding and frontline health care services: challenges for flood risk resilience in the English health care system . Journal of Health Services Research & Policy 24 : 219 – 28 .

Leykin D , Lahad M , Cohen O , Goldberg A , Aharonson-Daniel L. 2013 . Conjoint community resiliency assessment measure-28/10 items (CCRAM28 and CCRAM10): a self-report tool for assessing community resilience . American Journal of Community Psychology 52 : 313 – 23 .

Ling EJ , Larson E , Macauley RJ et al.  2017 . Beyond the crisis: did the Ebola epidemic improve resilience of Liberia’s health system? Health Policy and Planning 32 : iii40 – 7 .

Lomas KJ , Giridharan R , Short CA , Fair AJ. 2012 . Resilience of ‘Nightingale’ hospital wards in a changing climate . Building Services Engineering Research & Technology 33 : 81 – 103 .

Martineau FP. 2016 . People-centred health systems: building more resilient health systems in the wake of the Ebola crisis . International Health 8 : 307 – 9 .

Mash BJ , Mayers P , Conradie H et al.  2008 . How to manage organisational change and create practice teams: experiences of a South African primary care health centre . Educ Health (Abingdon) 21 : 132 .

Merandi J , Vannatta K , Davis JT et al.  2018 . Safety II behavior in a pediatric intensive care unit . Pediatrics 141 : e20180018 .

Meyer D , Kirk Sell T , Schoch-Spana M et al.  2018 . Lessons from the domestic Ebola response: improving health care system resilience to high consequence infectious diseases . American Journal of Infection Control 46 : 533 – 7 .

Moher D, Shamseer L, Clarke M  et al. 2015. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.  Systematic Reviews ,   4 :  1.

Moher D, Liberati A, Tetzlaff J, Altman DG. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Annals of Internal Medicine , 151 : 264 – 269.

Norris FH , Stevens SP , Pfefferbaum B , Wyche KF , Pfefferbaum RL. 2008 . Community resilience as a metaphor, theory, set of capacities, and strategy for disaster readiness . American Journal of Community Psychology 41 : 127 – 50 .

Noyes J , Booth A , Moore G et al.  2019 . Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods . BMJ Global Health 4 : e000893 .

O'Sullivan TL , Kuziemsky CE , Toal-Sullivan D , Corneil W. 2013 . Unraveling the complexities of disaster management: a framework for critical social infrastructure to promote population health and resilience . Social Science & Medicine 93 : 238 – 46 .

Olu O. 2017 . Resilient health system as conceptual framework for strengthening public health disaster risk management: an African viewpoint . Frontiers in Public Health 5 : 263 .

Orru K , Tillmann M , Ebi KL , Orru H. 2018 . Making administrative systems adaptive to emerging climate change-related health effects: case of Estonia . Atmosphere 9 : 221 .

Paterson J , Berry P , Ebi K , Varangu L. 2014 . Health care facilities resilient to climate change impacts . International Journal of Environmental Research and Public Health 11 : 13097 – 116 .

Patriarca R , Falegnami A , Costantino F , Bilotta F. 2018 . Resilience engineering for socio-technical risk analysis: application in neuro-surgery . Reliability Engineering & System Safety 180 : 321 – 335 .

Peters MD , Godfrey CM , Khalil H et al.  2015 . Guidance for conducting systematic scoping reviews . International Journal of Evidence-Based Healthcare 13 : 141 – 6 .

Radcliff TA , Chu K , DER-Martirosian C , Dobalian A. 2018 . A model for measuring ambulatory access to care recovery after disasters . The Journal of the American Board of Family Medicine 31 : 252 – 9 .

Raven J , Wurie H , Witter S. 2018 . Health workers’ experiences of coping with the Ebola epidemic in Sierra Leone’s health system: a qualitative study . BMC Health Services Research 18 : 251 .

Ray-Bennett NS , Corsel DMJ , Goswami N , Ghosh A. 2019 . Understanding reproductive health challenges during a flood: insights from Belkuchi Upazila . Gates Open Research 3 : 788 .

Ridde V , Lechat L , Meda IB. 2016 . Terrorist attack of 15 January 2016 in Ouagadougou: how resilient was Burkina Faso’s health system? BMJ Global Health 1 : e000056 .

Russo G , Rego I , Perelman J , Barros PP. 2016 . A tale of loss of privilege, resilience and change: the impact of the economic crisis on physicians and medical services in Portugal . Health Policy 120 : 1079 – 86 .

Short CA , Renganathan G , Lomas KJ. 2015 . A medium-rise 1970s maternity hospital in the east of England: resilience and adaptation to climate change . Building Services Engineering Research and Technology 36 : 247 – 74 .

Simonetti A , Ezzeldin H , Walderhaug M , Anderson SA , Forshee RA. 2018 . An inter-regional US blood supply simulation model to evaluate blood availability to support planning for emergency preparedness and medical countermeasures . Disaster Medicine and Public Health Preparedness 12 : 201 – 10 .

Sochas L , Channon AA , Nam S. 2017 . Counting indirect crisis-related deaths in the context of a low-resilience health system: the case of maternal and neonatal health during the Ebola epidemic in Sierra Leone . Health Policy and Planning 32 : iii32 – 9 .

Tanner T , Bahadur A , Moench M. 2017 . Challenges for resilience policy and practice. London: Overseas Development Institute.

Thomas S , Keegan C , Barry S et al.  2013 . A framework for assessing health system resilience in an economic crisis: Ireland as a test case . BMC Health Services Research 13 : 450 .

Thude BR , Juhl AG , Stenager E , Plessen CV , Hollnagel E. 2019 . Staff acting resiliently at two hospital wards . Leadership in Health Services (Bradford, England) 32 : 445 – 57 .

Toner ES , Mcginty M , Schoch-Spana M et al.  2017 . A community checklist for health sector resilience informed by Hurricane Sandy . Health Security 15 : 53 – 69 .

Tugade MM , Fredrickson BL. 2004 . Resilient individuals use positive emotions to bounce back from negative emotional experiences . Journal of Personality and Social Psychology 86 : 320 – 33 .

Turenne CP , Gautier L , Degroote S et al.  2019 . Conceptual analysis of health systems resilience: a scoping review . Social Science & Medicine 232 : 168 – 80 .

UNISDR. 2005 . Hyogo framework for action 2005–2015: building the resilience of nations and communities to disasters. Extract from the final report of the World Conference on Disaster Reduction (A/CONF. 206/6). The United Nations International Strategy for Disaster Reduction Geneva.

UNISDR. 2015 . Sendai framework for disaster risk reduction 2015–2030. Proceedings of the 3rd United Nations World Conference on DRR , Sendai, Japan, 14 – 8 .

Watts N , Amann M , Ayeb-Karlsson S et al.  2018 . The Lancet Countdown on health and climate change: from 25 years of inaction to a global transformation for public health . The Lancet 391 : 581 – 630 .

Witter S , Wurie H , Chandiwana P et al.  2017 . How do health workers experience and cope with shocks? Learning from four fragile and conflict-affected health systems in Uganda, Sierra Leone, Zimbabwe and Cambodia . Health Policy and Planning 32 : iii3 – 13 .

World Health Organization. 2010 . Monitoring the Building Blocks of Health Systems: A Handbook of Indicators and Their Measurement Strategies . Geneva: World Health Organization.

World Health Organization. 2015 . Operational Framework for Building Climate Resilient Health Systems . Geneva: World Health Organization.

Wulff K , Donato D , Lurie N. 2015 . What is health resilience and how can we build it? Annual Review of Public Health 36 : 361 – 74 .

Zhong S , Clark M , Hou XY , Zang Y , Fitzgerald G. 2014 a. Validation of a framework for measuring hospital disaster resilience using factor analysis . International Journal of Environmental Research and Public Health 11 : 6335 – 53 .

Zhong S , Clark M , Hou XY , Zang YL , Fitzgerald G. 2014 b. Development of hospital disaster resilience: conceptual framework and potential measurement . Emergency Medicine Journal 31 : 930 – 8 .

Zhong S , Clark M , Hou XY , Zang Y , Fitzgerald G. 2015 . Development of key indicators of hospital resilience: a modified Delphi study . Journal of Health Services Research & Policy 20 : 74 – 82 .

• health care systems

Supplementary data

Email alerts, citing articles via.

• Recommend to Your Librarian

Affiliations

• Online ISSN 1460-2237
• Copyright © 2024 The London School of Hygiene and Tropical Medicine and Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

A Systematic Literature Review of Empirical Research on Epistemic Network Analysis in Education

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 16 April 2024

Designing a framework for entrepreneurship education in Chinese higher education: a theoretical exploration and empirical case study

• Luning Shao 1 ,
• Yuxin Miao 2 ,
• Shengce Ren 3 ,
• Sanfa Cai 4 &
• Fei Fan   ORCID: orcid.org/0000-0001-8756-5140 5 , 6  

Humanities and Social Sciences Communications volume  11 , Article number:  519 ( 2024 ) Cite this article

476 Accesses

1 Altmetric

Metrics details

• Business and management

Entrepreneurship education (EE) has rapidly evolved within higher education and has emerged as a pivotal mechanism for cultivating innovative and entrepreneurial talent. In China, while EE has made positive strides, it still faces a series of practical challenges. These issues cannot be effectively addressed solely through the efforts of universities. Based on the triple helix (TH) theory, this study delves into the unified objectives and practical content of EE in Chinese higher education. Through a comprehensive literature review on EE, coupled with educational objectives, planned behavior, and entrepreneurship process theories, this study introduces the 4H objective model of EE. 4H stands for Head (mindset), Hand (skill), Heart (attitude), and Help (support). Additionally, the research extends to a corresponding content model that encompasses entrepreneurial learning, entrepreneurial practice, startup services, and the entrepreneurial climate as tools for achieving the objectives. Based on a single-case approach, this study empirically explores the application of the content model at T-University. Furthermore, this paper elucidates how the university plays a role through the comprehensive development of entrepreneurial learning, practices, services, and climate in nurturing numerous entrepreneurs and facilitating the flourishing of the regional entrepreneurial ecosystem. This paper provides important contributions in its application of TH theory to develop EE within the Chinese context, and it provides clear guidance by elucidating the core objectives and practical content of EE. The proposed conceptual framework serves not only as a guiding tool but also as a crucial conduit for fostering the collaborative development of the EE ecosystem. To enhance the robustness of the framework, this study advocates strengthening empirical research on TH theory through multiple and comparative case studies.

Similar content being viewed by others

Extra-curricular support for entrepreneurship among engineering students: development of entrepreneurial self-efficacy and intentions

Deepa Subhadrammal, Martin Bliemel, … Helene de Burgh-Woodman

The mediating effects of entrepreneurial self-efficacy in the relationship between entrepreneurship education and start-up readiness

Adeshina Olushola Adeniyi

Individual entrepreneurial orientation for entrepreneurial readiness

Adeshina Olushola Adeniyi, Vangeli Gamede & Evelyn Derera

Introduction

In the era of the knowledge economy, entrepreneurship has emerged as a fundamental driver of social and economic development. As early as 1911, Schumpeter proposed the well-known theory of economic development, wherein he first introduced the concepts of entrepreneurship and creative destruction as driving forces behind socioeconomic development. Numerous endogenous growth theories, such as the entrepreneurial ecosystem mechanism of Acs et al. ( 2018 ), which also underscores the pivotal role of entrepreneurship in economic development, are rooted in Schumpeter’s model. Recognized as a key means of cultivating entrepreneurs and enhancing their capabilities (Jin et al., 2023 ), entrepreneurship education (EE) has received widespread attention over the past few decades, especially in the context of higher education (Wong & Chan, 2022 ).

Driven by international trends and economic demands, China places significant emphasis on nurturing innovative talent and incorporating EE into the essential components of its national education system. The State Council’s “Implementation Opinions on Deepening the Reform of Innovation and Entrepreneurship Education in Higher Education” (hereafter referred to as the report) underscores the urgent necessity for advancing reforms in innovation and EE in higher education institutions. This initiative aligns with the national strategy of promoting innovation-driven development and enhancing economic quality and efficiency. Furthermore, institutions at various levels are actively and eagerly engaging in EE.

Despite the positive strides made in EE in China, its development still faces a series of formidable practical challenges. As elucidated in the report, higher education institutions face challenges such as a delay in the conceptualization of EE, inadequate integration with specialized education, and a disconnect from practical applications. Furthermore, educators exhibit a deficiency in awareness and capabilities, which manifests in a singular and less effective teaching methodology. The shortage of practical platforms, guidance, and support emphasizes the pressing need for comprehensive innovation and EE systems. These issues necessitate collaborative efforts from universities, industry, and policymakers.

Internationally established solutions for the current challenges have substantially matured, providing invaluable insights and guidance for the development of EE in the Chinese context. In the late 20th century, the concept of the entrepreneurial university gained prominence (Etzkowitz et al., 2000 ). Then, entrepreneurial universities expanded their role from traditional research and teaching to embrace a “third mission” centered on economic development. This transformation entailed fostering student engagement in entrepreneurial initiatives by offering resources and guidance to facilitate the transition of ideas into viable entrepreneurial ventures. Additionally, these entrepreneurial universities played a pivotal role in advancing the triple helix (TH) model (Henry, 2009 ). The TH model establishes innovation systems that facilitate knowledge conversion into economic endeavors by coordinating the functions of universities, government entities, and industry. The robustness of this perspective has been substantiated through comprehensive theoretical and empirical investigations (Mandrup & Jensen, 2017 ).

Therefore, this study aims to explore how EE in Chinese universities can adapt to new societal trends and demands through the guidance of TH theory. This research involves two major themes: educational objectives and content. Educational objectives play a pivotal role in regulating the entire process of educational activities, ensuring alignment with the principles and norms of education (Whitehead, 1967 ), while content provides a practical pathway to achieving these objectives. Specifically, the study has three pivotal research questions:

RQ1: What is the present landscape of EE research?

RQ2: What unified macroscopic goals should be formulated to guide EE in Chinese higher education?

RQ3: What specific EE system should be implemented to realize the identified goals in Chinese higher education?

The structure of this paper is as follows: First, we conduct a comprehensive literature review on EE to answer RQ1 , thereby establishing a robust theoretical foundation. Second, we outline our research methodology, encompassing both framework construction and case studies and providing a clear and explicit approach to our research process. Third, we derive the objectives and content model of EE guided by educational objectives, entrepreneurial motivations, and entrepreneurial process theories. Fourth, focusing on a typical university in China as our research subject, we conduct a case study to demonstrate the practical application of our research framework. Finally, we end the paper with the findings for RQ2 and RQ3 , discussions on the framework, and conclusions.

Literature review

The notion of TH first appeared in the early 1980s, coinciding with the global transition from an industrial to a knowledge-based economy (Cai & Etzkowitz, 2020 ). At that time, the dramatic increase in productivity led to overproduction, and knowledge became a valuable mechanism for driving innovation and economic growth (Mandrup & Jensen, 2017 ). Recognizing the potential of incorporating cutting-edge university technologies into industry and facilitating technology transfer and innovation, the US government took proactive steps to enhance the international competitiveness of American industries. This initiative culminated in the enactment of relevant legislation in 1980, which triggered a surge in technology transfer, patent licensing, and the establishment of new enterprises within the United States. Subsequently, European and Asian nations adopted similar measures, promoting the transformation of universities’ identity (Grimaldi et al., 2011 ). Universities assumed a central role in technology transfer, the formation of businesses, and regional revitalization within the knowledge society rather than occupying a secondary position within the industrial community. The conventional one-to-one relationships between universities, companies, and the government evolved into a dynamic TH model (Cai & Etzkowitz, 2020 ). Beyond their traditional roles in knowledge creation, wealth production, and policy coordination, these sectors began to engage in multifaceted interactions, effectively “playing the role of others” (Ranga & Etzkowitz, 2013 ).

The TH model encompasses three fundamental elements: 1) In a knowledge-based society, universities assume a more prominent role in innovation than in industry; 2) The three entities engage in collaborative relationships, with innovation policies emerging as a result of their mutual interactions rather than being solely dictated by the government; and 3) Each entity, while fulfilling its traditional functions, also takes on the roles of the other two parties (Henry, 2009 ). This model is closely aligned with EE.

On the one hand, EE can enhance the effectiveness of TH theory by strengthening the links between universities, industry, and government. The TH concept was developed based on entrepreneurial universities. The emerging entrepreneurial university model integrates economic development as an additional function. Etzkowitz’s research on the entrepreneurial university identified a TH model of academia-industry-government relations implemented by universities in an increasingly knowledge-based society (Galvao et al., 2019 ). Alexander and Evgeniy ( 2012 ) articulated that entrepreneurial universities are crucial to the implementation of triple-helix arrangements and that by integrating EE into their curricula, universities have the potential to strengthen triple-helix partnerships and boost the effectiveness of the triple-helix model.

On the other hand, TH theory also drives EE to achieve high-quality development. Previously, universities were primarily seen as sources of knowledge and human resources. However, they are now also regarded as reservoirs of technology. Within EE and incubation programs, universities are expanding their educational capabilities beyond individual education to shaping organizations (Henry, 2009 ). Surpassing their role as sources of new ideas for existing companies, universities blend their research and teaching processes in a novel way, emerging as pivotal sources for the formation of new companies, particularly in high-tech domains. Furthermore, innovation within one field of the TH influences others (Piqué et al., 2020 ). An empirical study by Alexander and Evgeniy ( 2012 ) outlined how the government introduced a series of initiatives to develop entrepreneurial universities, construct innovation infrastructure, and foster EE growth.

Overview of EE

EE occupies a crucial position in driving economic advancement, and this domain has been the focal point of extensive research. Fellnhofer ( 2019 ) examined 1773 publications from 1975 to 2014, introducing a more closely aligned taxonomy of EE research. This taxonomy encompasses eight major clusters: social and policy-driven EE, human capital studies related to self-employment, organizational EE and TH, (Re)design and evaluation of EE initiatives, entrepreneurial learning, EE impact studies, and the EE opportunity-related environment at the organizational level. Furthermore, Mohamed and Sheikh Ali ( 2021 ) conducted a systematic literature review of 90 EE articles published from 2009 to 2019. The majority of these studies focused on the development of EE (32%), followed by its benefits (18%) and contributions (12%). The selected research also addressed themes such as the relationship between EE and entrepreneurial intent, the effectiveness of EE, and its assessment (each comprising 9% of the sample).

Spanning from 1975 to 2019, these two reviews offer a comprehensive landscape of EE research. The perspective on EE has evolved, extending into multiple dimensions (Zaring et al., 2021 ). However, EE does not always achieve the expected outcomes, as challenges such as limited student interest and engagement as well as persistent negative attitudes are often faced (Mohamed & Sheikh Ali, 2021 ). In fact, the challenges faced by EE in most countries may be similar. However, the solutions may vary due to contextual differences (Fred Awaah et al., 2023 ). Furthermore, due to this evolution, there is a need for a more comprehensive grasp of pedagogical concepts and the foundational elements of modern EE (Hägg & Gabrielsson, 2020 ). Based on the objectives of this study, four specific themes were chosen for an in-depth literature review: the objectives, contents and methods, outcomes, and experiences of EE.

Objectives of EE

The objectives of EE may provide significant guidance for its implementation and the assessment of its effectiveness, and EE has evolved to form a diversified spectrum. Mwasalwiba ( 2010 ) presented a multifaceted phenomenon in which EE objectives are closely linked to entrepreneurial outcomes. These goals encompass nurturing entrepreneurial attitudes (34%), promoting new ventures (27%), contributing to local community development (24%), and imparting entrepreneurial skills (15%). Some current studies still emphasize particular dimensions of these goals, such as fostering new ventures or value creation (Jones et al., 2018 ; Ratten & Usmanij, 2021 ). These authors further stress the significance of incorporating practical considerations related to the business environment, which prompts learners to contemplate issues such as funding and resource procurement. This goal inherently underscores the importance of entrepreneurial thinking and encourages learners to transition from merely being students to developing entrepreneurial mindsets.

Additionally, Kuratko and Morris ( 2018 ) posit that the goal of EE should not be to produce entrepreneurs but to cultivate entrepreneurial mindsets in students, equipping them with methods for thinking and acting entrepreneurially and enabling them to perceive opportunities rapidly in uncertain conditions and harness resources as entrepreneurs would. While the objectives of EE may vary based on the context of the teaching institution, the fundamental goal is increasingly focused on conveying and nurturing an entrepreneurial mindset among diverse stakeholders. Hao’s ( 2017 ) research contends that EE forms a comprehensive system in which multidimensional educational objectives are established. These objectives primarily encompass cultivating students’ foundational qualities and innovative entrepreneurial personalities, equipping them with essential awareness of entrepreneurship, psychological qualities conducive to entrepreneurship, and a knowledge structure for entrepreneurship. Such a framework guides students towards independent entrepreneurship based on real entrepreneurial scenarios.

Various studies and practices also contain many statements about entrepreneurial goals. The Entrepreneurship Competence Framework, which was issued by the EU in 2016, delineates three competency domains: ideas and opportunities, resources and action. Additionally, the framework outlines 15 specific entrepreneurship competencies (Jun, 2017 ). Similarly, the National Content Standards for EE published by the US Consortium encompass three overarching strategies for articulating desired competencies for aspiring entrepreneurs: entrepreneurial skills, ready skills, and business functions (Canziani & Welsh, 2021 ). First, entrepreneurial skills are unique characteristics, behaviors, and experiences that distinguish entrepreneurs from ordinary employees or managers. Second, ready skills, which include business and entrepreneurial knowledge and skills, are prerequisites and auxiliary conditions for EE. Third, business functions help entrepreneurs create and operate business processes in business activities. These standards explain in the broadest terms what students need to be self-employed or to develop and grow a new venture. Although entrepreneurial skills may be addressed in particular courses offered by entrepreneurship faculties, it is evident that business readiness and functional skills significantly contribute to entrepreneurial success (Canziani & Welsh, 2021 ).

Contents and methods of EE

The content and methods employed in EE are pivotal factors for ensuring the delivery of high-quality entrepreneurial instruction, and they have significant practical implications for achieving educational objectives. The conventional model of EE, which is rooted in the classroom setting, typically features an instructor at the front of the room delivering concepts and theories through lectures and readings (Mwasalwiba, 2010 ). However, due to limited opportunities for student engagement in the learning process, lecture-based teaching methods prove less effective at capturing students’ attention and conveying new concepts (Rahman, 2020 ). In response, Okebukola ( 2020 ) introduced the Culturo-Techno-Contextual Approach (CTCA), which offers a hybrid teaching and learning method that integrates cultural, technological, and geographical contexts. Through a controlled experiment involving 400 entrepreneurship development students from Ghana, CTCA has been demonstrated to be a model for enhancing students’ comprehension of complex concepts (Awaah, 2023 ). Furthermore, learners heavily draw upon their cultural influences to shape their understanding of EE, emphasizing the need for educators to approach the curriculum from a cultural perspective to guide students in comprehending entrepreneurship effectively.

In addition to traditional classroom approaches, research has highlighted innovative methods for instilling entrepreneurial spirit among students. For instance, students may learn from specific university experiences or even engage in creating and running a company (Kolb & Kolb, 2011 ). Some scholars have developed an educational portfolio that encompasses various activities, such as simulations, games, and real company creation, to foster reflective practice (Neck & Greene, 2011 ). However, some studies have indicated that EE, when excessively focused on applied and practical content, yields less favorable outcomes for students aspiring to engage in successful entrepreneurship (Martin et al., 2013 ). In contrast, students involved in more academically oriented courses tend to demonstrate improved intellectual skills and often achieve greater success as entrepreneurs (Zaring et al., 2021 ). As previously discussed, due to the lack of a coherent theoretical framework in EE, there is a lack of uniformity and consistency in course content and methods (Ribeiro et al., 2018 ).

Outcomes of EE

Research on the outcomes of EE is a broad and continually evolving field, with most related research focusing on immediate or short-term impact factors. For example, Anosike ( 2019 ) demonstrated the positive effect of EE on human capital, and Chen et al. ( 2022 ) proposed that EE significantly moderates the impact of self-efficacy on entrepreneurial competencies in higher education students through an innovative learning environment. In particular, in the comprehensive review by Kim et al. ( 2020 ), six key EE outcomes were identified: entrepreneurial creation, entrepreneurial intent, opportunity recognition, entrepreneurial self-efficacy and orientation, need for achievement and locus of control, and other entrepreneurial knowledge. One of the more popular directions is the examination of the impact of EE on entrepreneurial intentions. Bae et al. ( 2014 ) conducted a meta-analysis of 73 studies to examine the relationship between EE and entrepreneurial intention and revealed little correlation. However, a meta-analysis of 389 studies from 2010 to 2020 by Zhang et al. ( 2022 ) revealed a positive association between the two variables.

Nabi et al. ( 2017 ) conducted a systematic review to determine the impact of EE in higher education. Their findings highlight that studies exploring the outcomes of EE have primarily concentrated on short-term and subjective assessments, with insufficient consideration of longer-term effects spanning five or even ten years. These longer-term impacts encompass factors such as the nature and quantity of startups, startup survival rates, and contributions to society and the economy. As noted in the Eurydice report, a significant impediment to advancing EE is the lack of comprehensive delineation concerning education outcomes (Bourgeois et al., 2016 ).

Experiences in the EE system

With the deepening exploration of EE, researchers have turned to studying university-centered entrepreneurship ecosystems (Allahar and Sookram, 2019 ). Such ecosystems are adopted to fill gaps in “educational and economic development resources”, such as entrepreneurship curricula. A growing number of universities have evolved an increasingly complex innovation system that extends from technology transfer offices, incubators, and technology parks to translational research and the promotion of EE across campuses (Cai & Etzkowitz, 2020 ). In the university context, the entrepreneurial ecosystem aligns with TH theory, in which academia, government, and industry create a trilateral network and hybrid organization (Ranga & Etzkowitz, 2013 ).

The EE system is also a popular topic in China. Several researchers have summarized the Chinese experience in EE, including case studies and overall experience, such as the summary of the progress and system development of EE in Chinese universities over the last decade by Weiming et al. ( 2013 ) and the summary of the Chinese experience in innovation and EE by Maoxin ( 2017 ). Other researchers take an in-depth look at the international knowledge of EE, such as discussions on the EE system of Denmark by Yuanyuan ( 2015 ), analyzes of the ecological system of EE at the Technical University of Munich by Yubing and Ziyan ( 2015 ), and comparisons of international innovation and EE by Ke ( 2017 ).

In general, although there has been considerable discussion on EE, the existing body of work has not properly addressed the practical challenges faced by EE in China. On the one hand, the literature is fragmented and has not yet formed a unified and mature theoretical framework. Regarding what should be taught and how it can be taught and assessed, the answers in related research are ambiguous (Hoppe, 2016 ; Wong & Chan, 2022 ). On the other hand, current research lacks empirical evidence in the context of China, and guidance on how to put the concept of EE into practice is relatively limited. These dual deficiencies impede the effective and in-depth development of EE in China. Consequently, it is imperative to comprehensively redefine the objectives and contents of EE to provide clear developmental guidance for Chinese higher education institutions.

Research methodology

To answer the research questions, this study employed a comprehensive approach by integrating both literature-based and empirical research methods. The initial phase focused on systematically reviewing the literature related to entrepreneurial education, aiming to construct a clear set of frameworks for the objectives and content of EE in higher education institutions. The second phase involved conducting a case study at T-University, in which the theoretical frameworks were applied to a real-world context. This case not only contributed to validating the theoretical constructs established through the literature review but also provided valuable insights into the practical operational dynamics of entrepreneurial education within the specific university setting.

Conceptual framework stage

This paper aims to conceptualize the objective and content frameworks for EE. The methodology sequence is as follows: First, we examine the relevant EE literature to gain insights into existing research themes. Subsequently, we identify specific research articles based on these themes, such as “entrepreneurial intention”, “entrepreneurial self-efficacy”, and “entrepreneurial approach”, among others. Third, we synthesize the shared objectives of EE across diverse research perspectives through an analysis of the selected literature. Fourth, we construct an objective model for EE within higher education by integrating Bloom’s educational objectives ( 1956 ) and Gagne’s five learning outcomes ( 1984 ), complemented by entrepreneurship motivation and process considerations. Finally, we discuss the corresponding content framework.

Case study stage

To further elucidate the conceptual framework, this paper delves into the methods for the optimization of EE in China through a case analysis. Specifically, this paper employs a single-case approach. While a single case study may have limited external validity (Onjewu et al., 2021 ), if a case study informs current theory and conceptualizes the explored issues, it can still provide valuable insights from its internal findings (Buchanan, 1999 ).

T-University, which is a comprehensive university in China, is chosen as the subject of the case study for the following reasons. First, T-University is located in Shanghai, which is a Chinese international technological innovation center approved by the State Council. Shanghai’s “14th Five-Year Plan” proposes the establishment of a multichannel international innovation collaboration platform and a global innovation cooperation network. Second, T-University has initiated curriculum reforms and established a regional knowledge economy ecosystem by utilizing EE as a guiding principle, which aligns with the characteristics of its geographical location, history, culture, and disciplinary settings. This case study will showcase T-University’s experiences in entrepreneurial learning, entrepreneurial practice, startup services, and the entrepreneurial climate, elucidating the positive outcomes of this triangular interaction and offering practical insights for EE in other contexts.

The data collection process of this study was divided into two main stages: field research and archival research. The obtained data included interview transcripts, field notes, photos, internal documents, websites, reports, promotional materials, and published articles. In the initial stage, we conducted a 7-day field trip, including visits to the Innovation and Entrepreneurship Institute, the Career Development Centre, the Academic Affairs Office, and the Graduate School. Moreover, we conducted semistructured interviews with several faculty members and students involved in entrepreneurship education at the university to understand the overall state of implementation of entrepreneurship education at the university. In the second stage, we contacted the Academic Affairs Office and the Student Affairs Office at the university and obtained internal materials related to entrepreneurship education. Additionally, we conducted a comprehensive collection and created a summary of publicly available documents, official school websites, public accounts, and other electronic files. To verify the validity of the multisource data, we conducted triangulation and ultimately used consistent information as the basis for the data analysis.

For the purpose of our study, thematic analysis was employed to delve deeply into the TH factors, the objective and content frameworks, and their interrelationships. Thematic analysis is a method for identifying, analyzing, and reporting patterns within data. This approach emphasizes a comprehensive interpretation of the data, as it extracts information from multiple perspectives and derives valuable conclusions through summary and induction (Onjewu et al., 2021 ). Therefore, thematic analysis likely serves as the foundation for most other qualitative data analysis methods (Willig, 2013 ). In this study, three researchers individually conducted rigorous analyses and comprehensive reviews to ensure the accuracy and reliability of the data. Subsequently, they engaged in collaborative discussions to explore their differences and ultimately reach a consensus.

Framework construction

Theoretical basis of ee in universities.

The study is grounded in the theories of educational objectives, planned behavior, and the entrepreneurial process. Planned behavior theory can serve to elucidate the emergence of entrepreneurial activity, while entrepreneurial process theory can be used to delineate the essential elements of successful entrepreneurship.

Theory of educational objectives. The primary goal of education is to assist students in shaping their future. Furthermore, education should directly influence students and facilitate their future development. Education can significantly enhance students’ prospects by imparting specific skills and fundamental principles and cultivating the correct attitudes and mindsets (Bruner, 2009 ). According to “The Aims of Education” by Whitehead, the objective of education is to stimulate creativity and vitality. Gagne identifies five learning outcomes that enable teachers to design optimal learning conditions based on the presentation of these outcomes, encompassing “attitude,” “motor skills,” “verbal information,” “intellectual skills,” and “cognitive strategies”. Bloom et al. ( 1956 ) argue that education has three aims, which concern the cognitive, affective, and psychomotor domains. Gedeon ( 2017 ) posits that EE involves critical input and output elements. The key objectives encompass mindset (Head), skill (hand), attitude (heart), and support (help). The input objectives include EE teachers, resources, facilities, courses, and teaching methods. The output objectives encompass the impacts of the input factors, such as the number of students, the number of awards, and the establishment of new companies. The primary aims of Gedeon ( 2017 ) correspond to those of Bloom et al. ( 1956 ).

Theory of planned behavior. The theory of planned behavior argues that human behavior is the outcome of well-thought-out planning (Ajzen, 1991 ). Human behavior depends on behavioral intentions, which are affected by three main factors. The first is derived from the individual’s “attitude” towards taking a particular action; the second is derived from the influence of “subjective norms” from society; and the third is derived from “perceived behavioral control” (Ajzen, 1991 ). Researchers have adopted this theory to study entrepreneurial behavior and EE.

Theory of the entrepreneurship process. Researchers have proposed several entrepreneurial models, most of which are processes (Baoshan & Baobao, 2008 ). The theory of the entrepreneurship process focuses on the critical determinants of entrepreneurial success. The essential variables of the entrepreneurial process model significantly impact entrepreneurial performance. Timmons et al. ( 2004 ) argue that successful entrepreneurial activities require an appropriate match among opportunities, entrepreneurial teams, resources, and a dynamic balance as the business develops. Their model emphasizes flexibility and equilibrium, and it is believed that entrepreneurial activities change with time and space. As a result, opportunities, teams, and resources will be unbalanced and need timely adjustment.

4H objective model of EE

Guided by TH theory, the objectives of EE should consider universities’ transformational identity in the knowledge era and promote collaboration among students, faculty, researchers, and external players (Mandrup & Jensen, 2017 ). Furthermore, through a comprehensive analysis of the literature and pertinent theoretical underpinnings, the article introduces the 4H model for the EE objectives, as depicted in Fig. 1 .

The 4H objective model of entrepreneurship education.

The model comprises two levels. The first level pertains to outcomes at the entrepreneurial behavior level, encompassing entrepreneurial intention and entrepreneurial performance. These two factors support universities’ endeavors to nurture individuals with an entrepreneurial mindset and potential and contribute to the region’s growth of innovation and entrepreneurship. The second level pertains to fundamentals, which form the foundation of the first level. The article defines these as the 4H model, representing mindset (Head), skill (Hand), attitude (Heart), and support (Help). This model integrates key theories, including educational objectives, the entrepreneurship process, and planned behavior.

First, according to the theory of educational objectives, the cognitive, emotional, and skill objectives proposed by Bloom et al. ( 1956 ) correspond to the key goals of education offered by Gedeon ( 2017 ), namely, Head, Hand, and Heart; thus, going forward, in this study, these three objectives are adopted. Second, according to the theory of planned behavior, for the promotion of entrepreneurial intention, reflection on the control of beliefs, social norms, and perceptual behaviors must be included. EE’s impact on the Head, Hand, and Heart will promote the power of entrepreneurs’ thoughts and perceptual actions. Therefore, this approach is beneficial for enhancing entrepreneurial intentions. Third, according to entrepreneurship process theory, entrepreneurial performance is affected by various factors, including entrepreneurial opportunities, teams, and resources. Consideration of the concepts of Head, Hand, and Heart can enhance entrepreneurial opportunity recognition and entrepreneurial team capabilities. However, as the primary means of obtaining external resources, social networks play an essential role in improving the performance of innovation and entrepreneurship companies (Gao et al., 2023 ). Therefore, an effective EE program should tell students how to take action, connect them with those who can help them succeed (Ronstadt, 1985 ), and help them access the necessary resources. If EE institutions can provide relevant help, they will consolidate entrepreneurial intentions and improve entrepreneurial performance, enabling the EE’s objective to better support the Head, Hand, and Heart.

Content model of EE

EE necessitates establishing a systematic implementation framework to achieve the 4H objectives. Current research on EE predominantly focuses on two facets: one focuses on EE methods to improve students’ skills, and the other focuses on EE outcome measurements, which consider the impact of EE on different stakeholders. Based on this, to foster innovation in EE approaches and enable long-term sustainable EE outcomes, the 4H Model of EE objectives mandates that pertinent institutions provide entrepreneurial learning, entrepreneurial practice, startup services, and a suitable entrepreneurial climate. These components constitute the four integral facets of the content model for EE, as depicted in Fig. 2 .

The content model of entrepreneurship education.

Entrepreneurial learning

Entrepreneurial learning mainly refers to the learning of innovative entrepreneurial knowledge and theory. This factor represents the core of EE and can contribute significantly to the Head component. It can also improve the entrepreneurial thinking ability of academic subjects through classroom teaching, lectures, information reading and analysis, discussion, debates, etc. Additionally, it can positively affect the Hand and Heart elements of EE.

Entrepreneurial practice

Entrepreneurial practice mainly refers to academic subjects comprehensively enhancing their cognition and ability by participating in entrepreneurial activities. This element is also a key component of EE and plays a significant role in the cultivation of the Hand element. Entrepreneurial practice is characterized by participation in planning and implementing entrepreneurial programs, competitions, and simulation activities. Furthermore, it positively impacts EE’s Head, Heart, and Help factors.

Startup services

Startup services mainly refer to entrepreneurial-related support services provided by EE institutions, which include investment and financing, project declaration, financial and legal support, human resources, marketing, and intermediary services. These services can improve the success of entrepreneurship projects. Therefore, they can reinforce the expectations of entrepreneurs’ success and positively impact the Heart, Hand, and Head objectives of EE.

Entrepreneurial climate

The entrepreneurial climate refers to the entrepreneurial environment created by EE institutions and their community and is embodied mainly in the educational institutions’ external and internal entrepreneurial culture and ecology. The environment can impact the entrepreneurial attitude of educated individuals and the Heart objective of EE. Additionally, it is beneficial for realizing EE’s Head, Hand, and Help goals.

Case study: EE practice of T-University

Overview of ee at t-university.

T-University is one of the first in China to promote innovation and EE. Since the 1990s, a series of policies have been introduced, and different platforms have been set up. After more than 20 years of teaching, research, and practice, an innovation and entrepreneurship education system with unique characteristics has gradually evolved. The overall goal of this system is to ensure that 100% of students receive such education, with 10% of students completing the program and 1% achieving entrepreneurship with a high-quality standard. The overall employment rate of 2020 graduates reached 97.49%. In recent years, the proportion of those pursuing entrepreneurship has been more than 1% almost every year. The T-Rim Knowledge-Based Economic Circle, an industrial cluster formed around knowledge spillover from T-University’s dominant disciplines, employs more than 400 T-University graduates annually.

In 2016, T-University established the School of Innovation & Entrepreneurship, with the president serving as its dean. This school focuses on talent development and is pivotal in advancing innovation-driven development strategies. It coordinates efforts across various departments and colleges to ensure comprehensive coverage of innovation and EE, the integration of diverse academic disciplines, and the transformation of interdisciplinary scientific and technological advancements (see Fig. 3 ).

T-University innovation and entrepreneurship education map.

T-University is dedicated to integrating innovation and EE into every stage of talent development. As the guiding framework for EE, the university has established the Innovation and EE sequence featuring “three-dimensional, linked, and cross-university cooperation” with seven educational elements. These elements include the core curriculum system of innovation and entrepreneurship, the “one top-notch and three excellences” and experimental zones of innovation and entrepreneurship talent cultivation model, the four-level “China-Shanghai-University-School” training programs for innovation and entrepreneurship, four-level “International-National-Municipal-University” science and technology competitions, four-level “National-Municipal-University-School” innovation and entrepreneurship practice bases, three-level “Venture Valley-Entrepreneurship Fund-Industry Incubation” startup services and a high-level teaching team with both full-time and part-time personnel.

T-University has implemented several initiatives. First, the university has implemented 100% student innovation and EE through reforming the credit setting and curriculum system. Through the Venture Valley class, mobile class, and “joint summer school”, more than 10% of the students completed the Innovation and EE program. Moreover, through the professional reform pilot and eight professional incubation platforms in the National Science and Technology Park of T-University and other measures, 1% of the students established high-quality entrepreneurial enterprises. Second, the university is committed to promoting the integration of innovation and entrepreneurship and training programs, exploring and practising a variety of innovative talent cultivation models, and adding undergraduate innovation ability development as a mandatory component of the training program. In addition, pilot reforms have been conducted in engineering, medicine, and law majors, focusing on integrating research and education.

T-University has constructed a high-level integrated innovation and entrepreneurship practice platform by combining internal and external resources. This platform serves as the central component in Fig. 3 , forming a sequence of innovation and entrepreneurship practice opportunities, including 1) the On-and-off Campus Basic Practice Platform, 2) the Entrepreneurship Practice Platform with the Integration of Production, Learning, and Research, 3) the Transformation Platform of Major Scientific Research Facilities and Achievements, and 4) the Strategic Platform of the T-Rim Knowledge-Based Economic Circle. All these platforms are accessible to students based on their specific tasks and objectives.

Moreover, the university has reinforced its support for entrepreneurship and collaborated with local governments in Sichuan, Dalian, and Shenzhen to establish off-campus bases jointly. In 2016, in partnership with other top universities in China, the university launched the Innovation and Entrepreneurship Alliance of Universities in the Yangtze River Delta. This alliance effectively brings together government bodies, businesses, social communities, universities, and funding resources in the Yangtze River Delta, harnessing the synergistic advantages of these institutions. In 2018, the university assumed the director role for the Ministry of Education’s Steering Committee for Innovation and Entrepreneurship. Through collaborations with relevant government agencies and enterprises, T-University has continued its efforts to reform and advance innovation and EE, establishing multiple joint laboratories to put theory into practice.

Startup service

In terms of entrepreneurial services, T-University has focused on the employment guidance center and the science and technology Park, working closely with the local industrial and commercial bureaus in the campus area to provide centralized entrepreneurial services. Through entities such as the Shanghai Municipal College Entrepreneurship Guidance Station, entrepreneurship seedling gardens, the science and technology park, and off-campus bases such as the entrepreneurship valley, the university has established a full-cycle service system that is tailored to students’ innovative and entrepreneurial activities, providing continuous professional guidance and support from the early startup stage to maturity.

Notably, the T-University Science and Technology Park has set up nine professional incubation service platforms that cover investment and financing, human resources, entrepreneurship training, project declaration, financial services, professional intermediaries, market promotion, advanced assessment, and the labor union. Moreover, the Technology Park has established a corporate service mechanism for liaison officers, counselors, and entrepreneurship mentors to ensure that enterprises receive comprehensive support and guidance. Through these services, T-University has successfully cultivated numerous high-tech backbone enterprises, such as New Vision Healthcare, Zhong Hui Ecology, Tongjie Technology, Tonglei Civil Engineering, and Tongchen Environmental Protection, which indicates the positive effect of these entrepreneurial services.

T-University places significant emphasis on fostering the entrepreneurial climate, which is effectively nurtured through the T-Rim Knowledge-Based Economic Circle and on-campus entrepreneurship activities. Moreover, T-University is dedicated to establishing and cultivating a dynamic T-Rim Knowledge-Based Economic Circle in strategic alignment with the district government and key agencies. This innovative ecosystem strategically centers around three prominent industrial clusters: the creative and design industry, the international engineering consulting services industry, and the new energy/materials and environmental technology industry. These industrial clusters provide fertile ground for graduates’ employment and entrepreneurial pursuits and have yielded remarkable economic outputs. In 2020, the combined value of these clusters surged to a staggering RMB 50 billion, with 80% of entrepreneurs being teachers, students, or alumni from T-University.

This commitment has led to the establishment of an intricate design industry chain featuring architectural design and urban planning design; it also supports services in automobile design, landscape design, software design, environmental engineering design, art media design, and associated services such as graphic production, architectural modeling, and engineering consulting.

The EE system at T-University

T-University has undertaken a comprehensive series of initiatives to promote EE, focusing on four key aspects: entrepreneurial learning, entrepreneurial practice, startup service, and the entrepreneurial climate. As of the end of 2021, the National Technology Park at T-University has cumulatively supported more than 3000 enterprises. Notably, the park has played a pivotal role in assisting more than 300 enterprises established by college students.

In its commitment to EE, the university maintains an open approach to engaging with society. Simultaneously, it integrates innovative elements such as technology, information, and talent to facilitate students’ entrepreneurial endeavors. Through the synergy between the university, government entities, and the market, EE cultivates a cadre of entrepreneurial talent. The convergence of these talents culminates in the formation of an innovative and creative industry cluster within the region, representing the tangible outcome of the university’s “disciplinary chain—technology chain—industry chain” approach to EE. This approach has gradually evolved into the innovative ecosystem of the T-Rim Knowledge-Based Economic Circle.

Findings and discussion

Unified macroscopic objectives of ee.

To date, a widespread consensus on defining EE in practical terms has yet to be achieved (Mwasalwiba, 2010 ; Nabi et al., 2017 ). Entrepreneurial education should strive towards a common direction, which is reflected in the agreement on educational objectives and recommended teaching methods(Aparicio et al., 2019 ). Mason and Arshed ( 2013 ) criticized that entrepreneurial education should teach about entrepreneurship rather than for entrepreneurship. Therefore, EE should not only focus on singular outcome-oriented aspects but also emphasize the cultivation of fundamental aspects such as cognition, abilities, attitudes, and skills.

This study embarks on a synthesis of the EE-related literature, integrating educational objective theory, planned behavior theory, and entrepreneurial process theory. The 4H model of EE objectives, which consists of basic and outcome levels, is proposed. This model aims to comprehensively capture the core elements of EE, addressing both students’ performance in entrepreneurial outcomes and their development of various aspects of foundational cognitive attributes and skills.

The basic level of the EE objective model includes the 4Hs, namely Head (mindset), Hand (skill), Heart (attitude), and Help (support). First, Head has stood out as a prominent learning outcome within EE over the past decade (Fretschner & Lampe, 2019 ). Attention given to the “Head” aspect not only highlights the development of individuals recognized as “entrepreneurs” (Mitra, 2017 ) but also underscores its role in complementing the acquisition of skills and practical knowledge necessary for initiating new ventures and leading more productive lives (Neck & Corbett, 2018 ).

Second, the Hand aspect also constitutes a significant developmental goal and learning outcome of EE. The trajectory of EE is evolving towards a focus on entrepreneurial aspects, and the learning outcomes equip students with skills relevant to entrepreneurship (Wong & Chan, 2022 ). Higher education institutions should go beyond fundamental principles associated with knowledge and actively cultivate students’ entrepreneurial skills and spirit.

Third, Heart represents EE objectives that are related to students’ psychological aspects, as students’ emotions, attitudes, and other affective factors impact their perception of entrepreneurship (Cao, 2021 ). Moreover, the ultimate goal of EE is to instill an entrepreneurial attitude and pave the way for future success as entrepreneurs in establishing new businesses and fostering job creation (Kusumojanto et al., 2021 ). Thus, the cultivation of this mindset is not only linked to the understanding of entrepreneurship but also intricately tied to the aspiration for personal fulfillment (Yang, 2013 ).

Fourth, entrepreneurship support (Help) embodies the goal of providing essential resource support to students to establish a robust foundation for their entrepreneurial endeavors. The establishment of a comprehensive support system is paramount for EE in universities. This establishment encompasses the meticulous design of the curriculum, the development of training bases, and the cultivation of teacher resources (Xu, 2017 ). A well-structured support system is crucial for equipping students with the necessary knowledge and skills to successfully navigate the complexities of entrepreneurship (Greene & Saridakis, 2008 ).

The outcome level of the EE objective model encompasses entrepreneurial intention and entrepreneurial performance, topics that have been extensively discussed in the previous literature. Entrepreneurial intention refers to individuals’ subjective willingness and plans for entrepreneurial behavior (Wong & Chan, 2022 ) and represents the starting point of the entrepreneurial process. Entrepreneurial performance refers to individuals’ actual behaviors and achievements in entrepreneurial activities (Wang et al., 2021 ) and represents the ultimate manifestation of entrepreneurial goals. In summary, the proposed 4H model of the EE objectives covers fundamental attitudes, cognition, skills, support, and ultimate outcomes, thus answering the question of what EE should teach.

Specific implementable system of EE

To facilitate the realization of EE goals, this study developed a corresponding content model as an implementable system and conducted empirical research through a case university. Guided by the 4H objectives, the content model also encompasses four dimensions: entrepreneurial learning, entrepreneurial practice, startup service, and entrepreneurial climate. Through a detailed exposition of the practical methods at T-university, this study provides support for addressing the question of how to teach EE.

In the traditional EE paradigm, there is often an overreliance on the transmission of theoretical knowledge, which leads to a deficiency in students’ practical experience and capabilities (Kremel and Wetter-Edman, 2019 ). Moreover, due to the rapidly changing and dynamic nature of the environment, traditional educational methods frequently become disconnected from real-world demands. In response to these issues, the approach of “learning by doing” has emerged as a complementary and improved alternative to traditional methods (Colombelli et al., 2022 ).

The proposed content model applies the “learning by doing” approach to the construction of the EE system. For entrepreneurial learning, the university has constructed a comprehensive innovation and EE chain that encompasses courses, experimental areas, projects, competitions, practice bases, and teaching teams. For entrepreneurial practice, the university has built a high-level, integrated innovation and entrepreneurship practice platform that provides students with the opportunity to turn their ideas into actual projects. For startup services, the university has established close collaborative relationships with local governments and enterprises and has set up nine professional incubation service platforms. For the entrepreneurial climate, the university cultivated a symbiotic innovation and EE ecosystem by promoting the construction of the T-Rim Knowledge-Based Economic Circle. Through the joint efforts of multiple parties, the entrepreneurial activities of teachers, students, and alumni have become vibrant and have formed a complete design industry chain and an enterprise ecosystem that coexists with numerous SMEs.

Development of a framework based on the TH theory

Through the exploration of the interactive relationships among universities, governments, and industries, TH theory points out a development direction for solving the dilemma of EE. Through the lens of TH theory, this study developed a comprehensive framework delineating the macroscopic objectives and practical methods of EE, as depicted in Fig. 4 . In this context, EE has become a common undertaking for multiple participants. Therefore, universities can effectively leverage the featured external and internal resources, facilitating the organic integration of entrepreneurial learning, practice, services, and climate. This, in turn, will lead to better achievement of the unified goals of EE.

Practical contents and objectives based on the triple helix theory.

Numerous scholars have explored the correlation between EE and the TH theory. Zhou and Peng ( 2008 ) articulated the concept of an entrepreneurial university as “the university that strongly influences the regional development of industries as well as economic growth through high-tech entrepreneurship based on strong research, technology transfer, and entrepreneurship capability.” Moreover, Tianhao et al. ( 2020 ) emphasized the significance of fostering collaboration among industry, academia, and research as the optimal approach to enhancing the efficacy of EE. Additionally, Ribeiro et al. ( 2018 ) underscored the pivotal role of MIT’s entrepreneurial ecosystem in facilitating startup launches. They called upon educators, university administrators, and policymakers to allocate increased attention to how university ecosystems can cultivate students’ knowledge, skills, and entrepreneurial mindsets. Rather than viewing EE within the confines of universities in isolation, we advocate for establishing an integrated system that encompasses universities, government bodies, and businesses. Such a system would streamline their respective roles and ultimately bolster regional innovation and entrepreneurship efforts.

Jones et al. ( 2021 ) reported that with the widespread embrace of EE by numerous countries, the boundaries between universities and external ecosystems are becoming increasingly blurred. This convergence not only fosters a stronger entrepreneurial culture within universities but also encourages students to actively establish startups. However, these startups often face challenges related to limited value and long-term sustainability. From the perspective of TH theory, each university can cultivate an ecosystem conducive to specialized entrepreneurial activities based on its unique resources and advantages. To do so, universities should actively collaborate with local governments and industries, leveraging shared resources and support to create a more open, inclusive, and innovation-supporting ecosystem that promotes lasting reform and sustainability.

There are two main ways in which this paper contributes to the literature. First, this study applies TH theory to both theoretical and empirical research on EE in China, presenting a novel framework for the operation of EE. Previous research has applied TH theory in contexts such as India, Finland, and Russia, showcasing the unique contributions of TH in driving social innovation. This paper introduces the TH model to the Chinese context, illustrating collaborative efforts and support for EE from universities, industries, and governments through the construction of EE objectives and content models. Therefore, this paper not only extends the applicability of the TH theory globally but also provides valuable insights for EE in the Chinese context.

Second, the proposed conceptual framework clarifies the core goals and practical content of EE. By emphasizing the comprehensive cultivation of knowledge, skills, attitudes, and resources, this framework provides a concrete reference for designing EE courses, activities, and support services. Moreover, the framework underscores the importance of collaborative efforts among stakeholders, facilitating resource integration to enhance the quality and impact of EE. Overall, the conceptual framework presented in this paper serves not only as a guiding tool but also as a crucial bridge for fostering the collaborative development of the EE ecosystem.

While EE has widespread global recognition, many regions still face similar developmental challenges, such as a lack of organized objectives and content delivery methods. This article, grounded in the context of EE in Chinese higher education institutions, seeks to address the current challenges guided by TH theory. By aligning EE with socioeconomic demands and leveraging TH theory, this study offers insights into the overall goals and practical content of EE.

This study presents a 4H objective model of EE comprising two levels. The first level focuses on outcomes related to entrepreneurial behavior, including entrepreneurial intentions and performance, which highlight the practical effects of EE. The second level is built as the foundation of the outcomes and encompasses the four elements of mindset, skill, attitude, and support. This multilayered structure provides a more systematic and multidimensional consideration for the cultivation of entrepreneurial talent. The framework offers robust support for practical instructional design and goal setting. Additionally, the research extends to the corresponding content model, incorporating four elements: entrepreneurial learning, entrepreneurial practice, startup services, and the entrepreneurial climate. This content model serves as a practical instructional means to achieve EE goals, enhancing the feasibility of implementing these objectives in practice.

Moreover, this study focused on a representative Chinese university, T-University, to showcase the successful implementation of the 4H and content models. Through this case, we may observe how the university, through comprehensive development in entrepreneurial learning, practice, services, and climate, nurtured many entrepreneurs and facilitated the formation of the innovation and entrepreneurship industry cluster. This approach not only contributes to the university’s reputation and regional economic growth but also offers valuable insights for other regions seeking to advance EE.

This study has several limitations that need to be acknowledged. First, the framework proposed is still preliminary. While its application has been validated through a case study, further exploration is required to determine the detailed classification and elaboration of its constituent elements to deepen the understanding of the EE system. Second, the context of this study is specific to China, and the findings may not be directly generalizable to other regions. Future research should investigate the adaptability of the framework in various cultural and educational contexts from a broader international perspective. Finally, the use of a single-case approach limits the generalizability of the research conclusions. Subsequent studies can enhance comprehensiveness by employing a comparative or multiple-case approach to assess the framework’s reliability and robustness.

In conclusion, this study emphasizes the need to strengthen the application of TH theory in EE and advocates for the enhancement of framework robustness through multiple and comparative case studies. An increase in the quantity of evidence will not only generate greater public interest but also deepen the dynamic interactions among universities, industries, and the nation. This, in turn, may expedite the development of EE in China and foster the optimization of the national economy and the overall employment environment.

Data availability

The datasets generated during and/or analyzed during the current study are not publicly available. Making the full data set publicly available could potentially breach the privacy that was promised to participants when they agreed to take part, in particular for the individual informants who come from a small, specific population, and may breach the ethics approval for the study. The data are available from the corresponding author on reasonable request.

Acs ZJ, Estrin S, Mickiewicz T et al. (2018) Entrepreneurship, institutional economics, and economic growth: an ecosystem perspective. Small Bus Econ 51(2):501–514. https://doi.org/10.1007/s11187-018-0013-9

Article   Google Scholar  

Ajzen I (1991) The theory of planned behavior. Organ Behav Hum Decis Process 50(2):179–211

Alexander U, Evgeniy P (2012) The entrepreneurial university in Russia: from idea to reality. Paper presented at the 10TH triple helix conference 2012

Allahar H, Sookram R (2019) Emergence of university-centred entrepreneurial ecosystems in the Caribbean. Ind Higher Educ 33(4):246–259. https://doi.org/10.1177/0950422219838220

Anosike P (2019) Entrepreneurship education as human capital: Implications for youth self-employment and conflict mitigation in Sub-Saharan Africa. Ind Higher Educ 33(1):42–54. https://doi.org/10.1177/0950422218812631

Aparicio G, Iturralde T, Maseda A (2019) Conceptual structure and perspectives on entrepreneurship education research: A bibliometric review. Eur res on manage and bus econ 25(3):105–113

Awaah F, Okebukola P, Shabani J et al. (2023) Students’ career interests and entrepreneurship education in a developing country. High Educ Skills Work-Based Learn 13(1):148–160

Awaah F (2023) In the classroom I enhance students understanding of entrepreneurship development—the culturo–techno-contextual approach. J Res Innov Teach Learn https://doi.org/10.1108/JRIT-08-2022-0047

Bae TJ, Qian S, Miao C et al. (2014) The relationship between entrepreneurship education and entrepreneurial intentions: a meta–analytic review. Entrep Theory Pract 38(2):217–254

Baoshan G, Baobao D (2008) 创业模型比较研究 [A comparative retrospective study of classic entrepreneurial models]. Foreign Econ Manag. 3:19–28

Google Scholar  

Bloom BS, Engelhart MD, Furst EJ, Hill WH, Krathwohl DR (1956) Taxonomy of educational objectives: the classification of educational goals. Handbook 1: Cognitive domain. David McKay, New York

Bourgeois A, Balcon M-P & Riiheläinen JM (2016) Entrepreneurship education at school in Europe. Eurydice Report. Education, Audiovisual and Culture Executive Agency, European Commission

Bruner JS (2009) The process of education. Harvard University Press

Buchanan DA (1999) The Logic of Political Action: an experiment with the epistemology of the particular. Br J Manag 10(s1):73–88. https://doi.org/10.1111/1467-8551.10.s1.7

Cai Y, Etzkowitz H (2020) Theorizing the triple helix model: past, present, and future. Triple Helix J 1–38. https://doi.org/10.1163/21971927-bja10003

Canziani BF, Welsh DHB (2021) How entrepreneurship influences other disciplines: an examination of learning goals. Int J Manag Educ 19(1). https://doi.org/10.1016/j.ijme.2019.01.003

Cao Q (2021) Entrepreneurial psychological quality and quality cultivation of college students in the higher education and moral education perspectives. Front Psychol 12:700334

Article   ADS   PubMed   PubMed Central   Google Scholar  

Chen H, Tang Y, Han J (2022) Building students’ entrepreneurial competencies in Chinese universities: diverse learning environment, knowledge transfer, and entrepreneurship education. Sustainability 14(15). https://doi.org/10.3390/su14159105

Colombelli A, Panelli A, Serraino F (2022) A learning-by-doing approach to entrepreneurship education: evidence from a short intensive online international program. Admin Sci 12(1). https://doi.org/10.3390/admsci12010016

Etzkowitz H, Webster A, Gebhardt C et al. (2000) The future of the university and the university of the future: evolution of ivory tower to entrepreneurial paradigm. Res Policy 29(2):313–330. https://doi.org/10.1016/S0048-7333(99)00069-4

Fellnhofer K (2019) Toward a taxonomy of entrepreneurship education research literature: a bibliometric mapping and visualization. Educ Res Rev 27:28–55

Fretschner M, Lampe HW (2019) Detecting hidden sorting and alignment effects of entrepreneurship education. J Small Bus Manage 57(4):1712–1737

Gagne RM (1984) Learning outcomes and their effects: useful categories of human performance. Am Psychol 39(4):377

Galvao A, Mascarenhas C, Marques C et al. (2019) Triple helix and its evolution: a systematic literature review. J Sci Technol Policy Manag 10(3):812–833. https://doi.org/10.1108/JSTPM-10-2018-0103

Gao J, Cheng Y, He H et al. (2023) The mechanism of entrepreneurs’ Social Networks on Innovative Startups’ innovation performance considering the moderating effect of the entrepreneurial competence and motivation. Entrep Res J 13(1):31–69. https://doi.org/10.1515/erj-2020-0541

Gedeon SA (2017) Measuring student transformation in entrepreneurship education programs. Educ Res Int 2017:8475460. https://doi.org/10.1155/2017/8475460

Greene FJ, Saridakis G (2008) The role of higher education skills and support in graduate self-employment. Stud High Educ 33(6):653–672. https://doi.org/10.1080/03075070802457082

Grimaldi R, Kenney M, Siegel DS et al. (2011) 30 years after Bayh–Dole: reassessing academic entrepreneurship. Res Policy 40(8):1045–1057

Hägg G, Gabrielsson J (2020) A systematic literature review of the evolution of pedagogy in EE research. Int J Entrep Behav Res 26(5):829–861

Hao Y (2017) Research on building curriculum system of entrepreneurship education for college students in China. In: Paper presented at the Proceedings of the 7th international conference on education, management, computer and medicine (EMCM 2016)

Henry E (2009) The entrepreneurial university and the triple helix model of innovation. Stud Sci Sci 27(4):481–488

Hoppe M (2016) Policy and entrepreneurship education. Small Bus Econ. 46(1):13–29

Jin D, Liu X, Zhang F et al. (2023) Entrepreneurial role models and college students’ entrepreneurial calling: a moderated mediation model. Front Psychol 14:1129495

Article   PubMed   PubMed Central   Google Scholar  

Jones C, Penaluna K, Penaluna A et al. (2018) The changing nature of enterprise: addressing the challenge of Vesper and Gartner. Ind High Educ 32(6):430–437

Jones P, Maas G, Kraus S et al. (2021) An exploration of the role and contribution of entrepreneurship centres in UK higher education institutions. J Small Bus Enterp Dev 28(2):205–228. https://doi.org/10.1108/JSBED-08-2018-0244

Jun C (2017) 欧盟创业能力框架: 创业教育行动新指南 [EU’s entrepreneurship competence framework: a new guide to entrepreneurship education]. Int Comp Educ 1:45–51

MathSciNet   Google Scholar  

Ke L (2017) 创新创业教育的国际比较与借鉴 [International comparison and reference of innovation and entrepreneurship education]. Stud Dialect Nat 9:73–78

Kim G, Kim D, Lee WJ, et al (2020) The effect of youth entrepreneurship education programs: two large-scale experimental studies. Sage Open 10(3). https://doi.org/10.1177/2158244020956976

Kolb A, Kolb D (2011) Experiential learning theory: a dynamic, holistic approach to management learning, education and development. In Armstrong SJ, Fukami C (Eds) Handbook of management learning, education and development. https://doi.org/10.4135/9780857021038.n3

Kremel A, Wetter-Edman K (2019) Implementing design thinking as didactic method in entrepreneurship education, the importance of through. Des J 22:163–175. https://doi.org/10.1080/14606925.2019.1595855

Kuratko DF, Morris MH (2018) Examining the future trajectory of entrepreneurship. J Small Bus Manag 56(1):11–23. https://doi.org/10.1111/jsbm.12364

Kusumojanto DD, Wibowo A, Kustiandi J, et al (2021) Do entrepreneurship education and environment promote students’ entrepreneurial intention? The role of entrepreneurial attitude. Cogent Educ 8(1). https://doi.org/10.1080/2331186X.2021.1948660

Mandrup M, Jensen TL (2017) Educational Action Research and Triple Helix principles in entrepreneurship education: introducing the EARTH design to explore individuals in Triple Helix collaboration. Triple Helix 4(1). https://doi.org/10.1186/s40604-017-0048-y

Maoxin Y (2017) 创业教育的中国经验 [“China’s experiences” of entrepreneurship education]. Educ Res 38(9):70–75

Martin BC, McNally JJ, Kay MJ (2013) Examining the formation of human capital in entrepreneurship: a meta-analysis of entrepreneurship education outcomes. J Bus Ventur 28(2):211–224

Mason C, Arshed N (2013) Teaching entrepreneurship to university students through experiential Learning: a case study. Ind High Educ 27(6):449–463

Mitra J (2017) Holistic experimentation for emergence: a creative approach to postgraduate entrepreneurship education and training. Ind High Educ 31(1):34–50

Mohamed NA, Sheikh Ali AY (2021) Entrepreneurship education: systematic literature review and future research directions. World J Entrep Manag Sustain Dev 17(4):644–661

Mwasalwiba ES (2010) Entrepreneurship education: a review of its objectives, teaching methods, and impact indicators. Educ+ Train 52(1):20–47

Nabi G, Liñán F, Fayolle A et al. (2017) The impact of entrepreneurship education in higher education: a systematic review and research agenda. Acad Manag Learn Educ 16(2):277–299. https://doi.org/10.5465/amle.2015.0026

Neck HM, Greene PG (2011) Entrepreneurship education: known worlds and new frontiers. J Small Bus Manag 49(1):55–70. https://doi.org/10.1111/j.1540-627X.2010.00314.x

Neck HM, Corbett AC (2018) The scholarship of teaching and learning entrepreneurship. Entrep Educ Pedagog 1(1):8–41

Okebukola P (2020) Breaking barriers to learning: the culture techno-contextual approach (CTCA). Sterling Publishers, Slough

Onjewu AKE, Sukumar A, Prakash KVD et al (2021) The triple helix: a case study of Centurion University of Technology and Management. In Jones P, Apostolopoulos N, Kakouris A, Moon C, Ratten V & Walmsley A (Eds), Universities and entrepreneurship: meeting the educational and social challenges (Vol 11, pp 199–218)

Piqué JM, Berbegal-Mirabent J, Etzkowitz H (2020) The role of universities in shaping the evolution of Silicon Valley’s ecosystem of innovation. Triple Helix J 1–45. https://doi.org/10.1163/21971927-bja10009

Rahman S (2020) Improving the power of lecture method in higher education. In Teaching learning and new technologies in higher education (pp 135–147)

Ranga M, Etzkowitz H (2013) Triple helix systems: an analytical framework for innovation policy and practice in the knowledge society. Ind. High Educ. 27(4):237–262. https://doi.org/10.5367/ihe.2013.0165

Ratten V, Usmanij P (2021) Entrepreneurship education: time for a change in research direction? Int J Manag Educ 19(1). https://doi.org/10.1016/j.ijme.2020.100367

Ribeiro ATVB, Uechi JN, Plonski GA (2018) Building builders: entrepreneurship education from an ecosystem perspective at MIT. Triple Helix 5(1). https://doi.org/10.1186/s40604-018-0051-y

Ronstadt R (1985) The educated entrepreneurs: a new era of EE is beginning. Am J Small Bus 10(1):7–23. https://doi.org/10.1177/104225878501000102

Tianhao L, Beiwei L, Yang L (2020) 国外创新创业教育发展述评与启示 [The development of innovation and entrepreneurship education in foreign countries: review and enlightenment]. Manag Innov Entrep 1:23–36

Timmons JA, Spinelli S, Tan Y (2004) New venture creation: entrepreneurship for the 21st century (Vol 6). McGraw-Hill/Irwin, New York

Wang SY, Wu XL, Xu M et al. (2021) The evaluation of synergy between university entrepreneurship education ecosystem and university students’ entrepreneurship performance. Math Probl Eng, https://doi.org/10.1155/2021/3878378

Weiming L, Chunyan L, Xiaohua D (2013) 我国高校创业教育十年: 演进, 问题与体系建设 [Research on ten-year entrepreneurship education in Chinese universities: evolution, problems and system construction]. Educ Res 6:42–51

Whitehead AN (1967) Aims of education: Simon and Schuster

Willig C (2013) EBOOK: introducing qualitative research in psychology. McGraw-Hill Education, UK

Wong HY, Chan CK (2022) A systematic review on the learning outcomes in entrepreneurship education within higher education settings. Assess Eval High Educ 47(8):1213–1230

Xu Y (2017) Research on the Construction of Support System of University Students’ Entrepreneurship Education under the Background of the New Normal [Proceedings Paper]. 2017 INTERNATIONAL CONFERENCE ON FRONTIERS IN EDUCATIONAL TECHNOLOGIES AND MANAGEMENT SCIENCES (FETMS 2017)

Yang J (2013) The theory of planned behavior and prediction of entrepreneurial intention among Chinese undergraduates. Soc Behav Personal Int J 41(3):367–376

Yuanyuan C (2015) 从 ABC 到 PhD: 丹麦创业教育体系的框架设计与特点 [From ABC to PhD: the guiding ideas and development framework of Danish EE (Entrepreneurship Education) system]. Int Comp Educ 8:7–13

Yubing H, Ziyan G (2015) 慕尼黑工业大学创业教育生态系统建设及启示 [The EE ecosystem of TUM and some recommendations to China]. Sci Sci Manag 10:41–49

Zaring O, Gifford E, McKelvey M (2021) Strategic choices in the design of entrepreneurship education: an explorative study of Swedish higher education institutions. Stud High Educ 46(2):343–358

Zhang W, Li Y, Zeng Q, et al. (2022) Relationship between entrepreneurship education and entrepreneurial intention among college students: a meta-analysis. Int J Environ Res Public Health, 19(19) https://doi.org/10.3390/ijerph191912158

Zhou C, Peng X-M (2008) The entrepreneurial university in China: nonlinear paths. Sci Public Policy 35(9):637–646

Download references

Acknowledgements

We express our sincere gratitude to all individuals who contributed to the data collection process. Furthermore, we extend our appreciation to Linlin Yang and Jinxiao Chen from Tongji University for their invaluable suggestions on the initial draft. Special thanks are also due to Prof. Yuzhuo Cai from Tampere University for his insightful contributions to this paper. Funding for this study was provided by the Chinese National Social Science Funds [BIA190205] and the Shanghai Educational Science Research General Project [C2023033].

Author information

Authors and affiliations.

School of Economics and Management, Tongji University, Shanghai, China

Luning Shao

Shanghai International College of Design & Innovation, Tongji University, Shanghai, China

Shanghai International College of Intellectual Property, Tongji University, Shanghai, China

Shengce Ren

Institute of Higher Education, Tongji University, Shanghai, China

College of Design and Innovation, Tongji University, Shanghai, China

Shanghai Industrial Innovation Ecosystem Research Center, Tongji University, Shanghai, China

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors contributed to the study’s conception and design. Material preparation, data collection, and analysis were performed by Luning Shao, Yuxin Miao, Sanfa Cai and Fei Fan. The first Chinese outline and draft were written by Luning Shao, Yuxin Miao, and Shengce Ren. The English draft of the manuscript was prepared by Fei Fan. All the authors commented on previous versions of the manuscript. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Fei Fan .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This research was approved by the Tongji University Ethics Committee for Human Research (No. tjdxsr079). The procedures used in this study adhere to the tenets of the Declaration of Helsinki.

Informed consent

Informed consent was obtained from all participants.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Shao, L., Miao, Y., Ren, S. et al. Designing a framework for entrepreneurship education in Chinese higher education: a theoretical exploration and empirical case study. Humanit Soc Sci Commun 11 , 519 (2024). https://doi.org/10.1057/s41599-024-03024-2

Download citation

Received : 22 May 2023

Accepted : 03 April 2024

Published : 16 April 2024

DOI : https://doi.org/10.1057/s41599-024-03024-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

Open government data: A systematic literature review of empirical research

Bernd w. wirtz.

German University of Administrative Science Speyer, Chair for Information & Communication Management, Postbox 1409, 67324 Speyer, Germany

Jan C. Weyerer

Marcel becker, wilhelm m. müller, associated data.

Open government data (OGD) holds great potential for firms and the digital economy as a whole and has attracted increasing interest in research and practice in recent years. Governments and organizations worldwide are struggling in exploiting the full potential of OGD and require a comprehensive understanding of this phenomenon. Although scientific debates in OGD research are intense and heterogeneous, the field lacks theoretical integration of OGD topics and their systematic consideration in the context of the digital economy. In addition, OGD has been widely neglected by information systems (IS) research, which promises great potential for advancing our knowledge of the OGD concept and its role in the digital economy. To fill in this gap, this study conducts a systematic literature review of 169 empirical OGD studies. In doing so, we develop a theoretical review framework of Antecedents, Decisions, Outcomes (ADO) to unify and grasp the accumulating isolated evidence on OGD in context of the digital economy and provide a theory-informed research agenda to tap the potential of IS research for OGD. Our findings reveal six related key topic clusters of OGD research and substantial gaps, opening up prospective research avenues and particularly outlining how IS research can inform and advance OGD research.

Supplementary information

The online version contains supplementary material available at 10.1007/s12525-022-00582-8.

Introduction

In the age of the digital economy, data have become a new currency and an indispensable asset for organizations. Data constitutes the foundation of innovative technologies and applications (e.g., AI and IoT) and data-driven insights and management are vital for organizational success. The advancing digitalization in the public sector over the last decade has led to large amounts of data, making the public sector one of the main producers of data in the digital economy. A substantial part of this data pool is freely provided to the public and is commonly referred to as open government data (OGD) (Kim, 2018 ; Lim, 2021 ). As the number and worldwide development of OGD initiatives continue to advance in light of its great importance (Attard et al., 2015 ; Piotrowski, 2017 ), the widely unexplored relationship between OGD and the digital economy becomes of increasing interest.

On the one hand, the digital economy itself constitutes an important driver of OGD adoption and the successful implementation of OGD programs, as IT firms, for instance, supply public administration with mission-critical tangible (e.g., hardware and software), human (e.g., IT consultants), and intangible (e.g., IT and data know how) IT resources. On the other hand, OGD constitutes a new source of innovation and economic growth for the digital economy. OGD offers the potential to create innovation and to increase economic value sustainably - for both the public and the private business sectors. It may serve organizations as a free and meaningful complementary data source in developing new products or services, as well as in improving business intelligence, R&D, and business processes (Magalhaes & Roseira, 2020 ). Thus, OGD and the digital economy are characterized by a reciprocal relationship, in which both sides benefit from each other.

While the public value of OGD in terms of leveling up the transparency of governmental activities, the political participation of citizens and the collaboration between governments and external stakeholders is well-documented (Lee et al., 2019 ; Ruijer et al., 2017 ), its great opportunities and importance for the digital economy and commercial use have been widely neglected. According to the World Wide Web Foundation ( 2017 ), the impact of OGD on the economy even in the top ten countries worldwide remains rather low, averaging four out of ten on their assessment scale. A recent survey of 178 U.S. firms on the use OGD further reveals that the frequency of application varies across different forms of use, ranging from 9% (data to fact) to 44% (data to service) (Magalhaes & Roseira, 2020 ). These figures indicate that firms and the digital economy as a whole seem to struggle in using OGD and exploiting its full potential. This is also reflected in the current research landscape, in which the OGD concept has been predominantly examined in public administration and public management research, while receiving little attention in the field of information systems (IS) and digital business research.

Given its relevance for the digital economy and close relatedness to information systems and various associated research streams (e.g., big data analytics, AI and IoT), it is essential to frame OGD more broadly in the context of the digital economy and build a bridge to IS and digital business research. The stronger involvement of the latter promises great potential for further advancing the OGD concept and filling in the gap pertaining to its role in the digital economy and commercial use, as demanded in the literature (Magalhaes & Roseira, 2020 ). In order to better familiarize the IS and digital business research community with OGD and meaningfully involve it in the scholarly discussion, it is essential to first convey a broad understanding of the concept, its research landscape, and specific starting points for potential research endeavours.

As the role of the digital economy in OGD initiatives and the value potential of OGD is influenced by the antecedents of OGD programs (e.g., sophistication of governmental data infrastructures), the decisions and actions taken by the government for implementing OGD (e.g., strategic positioning and scope of governmental OGD activity), as well as the achieved outcomes and impacts (e.g., efficiency gains through and acceptance of OGD), it seems particularly promising to examine OGD and its relevance for the digital economy along these dimensions.

The research field of OGD has been on the rise over the last decade. While the number and heterogeneity of contributions are increasing, comprehensive literature reviews remain scarce in the context of open government (Tai, 2021 ), in particular from an IS perspective. Most importantly, OGD research lacks theoretical foundation and integration of OGD topics (Hassan & Twinomurinzi, 2018 ), as well as their systematic examination in the context of the digital economy. Taken together, the literature fails to provide a theoretical framework combining theoretical and empirical insights on OGD with regard to its antecedents, decisions, and outcomes, in which the concept is framed more broadly in the context of the digital economy, and which yields a research agenda that meaningfully involves the field of IS and digital business research. To fill in this gap, we conduct a systematic literature review to address the following research questions: (1) what do we know about the antecedents, decisions, and outcomes of OGD and their relation in the context of the digital economy, and (2) how can IS and digital business research inform OGD research in this connection?

To answer these research questions, the remainder of the study is structured as follows: The next section discusses definitional issues of OGD, delineating it from the closely related concepts of open government and open data. We then present an overview of prior literature reviews related to OGD and illustrate their shortcomings and implications for the study at hand. Subsequently, we describe the methodological approach and results of the systematic review of OGD literature and develop an overarching theoretical framework to integrate and synthesize thematic clusters of OGD research. Based on this, we derive a research agenda for future research on OGD providing concrete research avenues for IS and digital business research. In the final section, the findings and implications are discussed in the context of prior research and the digital economy.

Defining open government data between the poles of open government and open data

OGD is closely related to other concepts, in particular, open government and open data. Although it may be viewed as a hybrid of both of these more general concepts (Sayogo et al., 2014 ), the extensive number of dedicated OGD studies in recent years indicates not only the increasing scholarly interest but also that OGD has established itself as a distinct concept and research stream separate from its superordinates, open government and open data. This also becomes apparent when looking at differentiated definitions of each concept. To begin with, open government is generally defined as “a multilateral, political, and social process, which includes in particular transparent, collaborative, and participatory action by government and administration” (Wirtz & Birkmeyer, 2015 , p. 382). Although OGD can be viewed as a manifestation thereof underlying the same principles of transparency, collaboration and participation (Wirtz et al., 2019 ), it sets itself apart from the general concept through its data character and thus its inherently closer link to information systems.

This data characteristic is – besides the openness – the common denominator of the OGD and the open data concept and separates both from the open government concept. A widely used definition of open data refers to data that “can be freely used, modified, and shared by anyone for any purpose” (Open Knowledge Foundation, 2021 ). The definition of open government and its delineation from open data has been subject to many debates in the literature (Bogdanović-Dinić et al., 2014 ; Karkin & Yavuz, 2017 ; Kim, 2018 ; Sayogo et al., 2014 ). Although some earlier approaches use both terms synonymously (Janssen et al., 2012 ; Veljković et al., 2014 ), there is meanwhile consensus in the literature that OGD constitutes a subform of open data and the special distinguishing mark is that OGD is data collected by means of public funding and/or provided by public sector organizations (Borgesius et al., 2015 ; Kim, 2018 ; Lim, 2021 ). Accordingly, OGD is defined as “non-confidential, non-privacy-restricted data collected using public funding that is made freely available for anyone to download” (Lim, 2021 , p. 1) or put more simple as “[p]ublic sector information made available to the public as open data” (Kim, 2018 , p. 20). Thus, its government relatedness is the decisive element distinguishing it from open data.

For a better understanding of the scope and nature of OGD, the OECD (Ubaldi, 2013 ) has developed a typology of OGD, distinguishing between seven major categories: (1) business data (e.g., chamber of commerce information and official business information with regard to company or industry data), (2) registers and data pertaining to patents, trademarks, and public tenders, (3) geographic data (e.g., topographic and address data), (4) legal data (e.g., court decisions, legislation data), (5) meteorological data (e.g., weather and climate data), (6) social data (e.g., population, employment, and public health data), and (7) transport data (e.g., vehicle registrations, traffic, and public transport data). This typology underlines the particularities of OGD and indicates its various application opportunities and value for businesses.

Prior literature reviews on OGD

The widespread scientific interest in OGD is reflected in a large number of studies, which have been the motivator and starting point for various overview studies. With a view to placing our systematic literature review in the existing field of literature reviews and determining its potential contribution to future OGD research, we first identified and analyzed the thematically relevant set of previous literature reviews. We systematically searched for literature reviews in different databases, including EBSCO (including Academic Search Premier, Business Source Premier, and EconLit with Full Text), Web of Science, ScienceDirect, ProQuest, and Google Scholar. This yielded a total of twelve dedicated literature reviews that were obtained for further analysis. To determine the scientific added value of our study, it is important to contrast the core structure and key topics of these literature reviews briefly and concisely, see Table  1 .

Overview of former literature reviews

The literature reviews identified can be classified into three clusters: (1) reviews treating OGD as a side aspect, (2) reviews focusing on a specific aspect of OGD literature, and (3) reviews with a general approach towards OGD literature. The first cluster contains four out of twelve reviews identified. These reviews do not clearly distinguish between open government, open data, and OGD, and thus mix OGD studies in their analysis with those from one of the other research streams. To begin with, Hossain et al. ( 2016 ) provide a general systematization of the research field of open data, addressing OGD as one of five subareas and deriving corresponding research implications. The other three reviews in this cluster focus on OGD including OGD studies as a subset in their analyses . While Wirtz and Birkmeyer ( 2015 ) concentrate on the development of an integrative framework to better understand open government in general, Criado et al. ( 2018 ) attempt to explain the phenomenon of open government by means of a comprehensive analysis of existing literature and provide a comprehensive overview without deriving overly specific research implications. Likewise, Tai ( 2021 ) also provides a comprehensive review of open government research, focusing on three aspects, namely its conceptual development, its use and implementation, as well as the impacts or outcomes of open government initiatives. However, an integrated consideration as applied by the above-mentioned reviews in the first cluster confounds a clear picture of OGD research and carries the risk of arriving at undifferentiated and ultimately inaccurate conclusions. Therefore, it is essential to conduct review studies that are solely dedicated to the field of OGD, as is the case with the second and third cluster of review studies.

The second cluster is the largest and is composed of six out of twelve literature reviews identified. These reviews analyze a certain segment of OGD literature depending on a selected subtopic. The work of Attard et al. ( 2015 ) clearly focuses on the description of OGD initiatives and their respective components. They are less concerned with mapping and structuring the literature as a whole but rather with analyzing OGD initiatives and related approaches. In contrast, Ruijer and Martinius ( 2017 ) set their focus more specifically by examining literature and deriving specific research implications in relation to the democratic impact of OGD. Safarov et al. ( 2017 ) have a different emphasis by orienting their literature evaluation and systematization towards the development of an OGD utilization framework and pointing out utilization-specific research opportunities. Moreover, the literature review of Haini et al. ( 2020 ) has a special view upon studies concerning influence factors of OGD adoption in public sector organizations, identifying 16 influence factors and classifying them according to three dimensions (i.e., technological, organizational, and environmental). In contrast, Purwanto et al. ( 2020 ) focus on the citizen perspective in their review and analyze studies that deal with drivers of and barriers to citizen engagement with OGD. They identify seven groups of drivers and three categories of barriers, developing a conceptual model of citizen engagement with OGD. Finally, Francey and Mettler ( 2021 ) review case studies and examine empirical evidence on the effects of OGD, deriving nine stylized facts. While all of the studies in the second cluster provide valuable insights into the field of OGD, they only do so for the respective subtopic analyzed. None of these reviews systematizes the entire field of research and identifies the implications for further necessary research. Although Safarov et al. ( 2017 ) make a well-conceived attempt to broadly analyze and systematize based on their grouping along four key topics and the further subdivision thereof, their findings still remain specific in that they are primarily concerned with the utilization of OGD. Thus, the reviews in this cluster do not allow to make profound comparisons among subtopics within the field or to draw general conclusions in order to improve our understanding of relationships among subtopics and the state of research as a whole. This can only be achieved by reviews with a comprehensive perspective, like those in the third cluster of our literature review analysis.

This cluster is the smallest and comprises only two reviews, indicating the lack of reviews with a comprehensive focus on OGD research. These approaches are most relevant to our study because they likewise address the OGD topic as a whole. In doing so, Zuiderwijk et al. ( 2014 ) examine individual studies in relation to their topic and theoretical foundation. They offer a brief outlook on potential fields of research related to the three core topics they identified, including theory and development; policies, use, and innovation; as well as infrastructures and technology. Saxena’s ( 2018 ) systematic literature review likewise classifies OGD studies into three general clusters, i.e. theoretical and conceptual research, applied research, and user-focused research. Despite their valuable contributions both studies lack theoretical foundation and integration of the clusters. Moreover, both reviews each propose a very general taxonomy to structure research. Both taxonomies contain three clusters and appear to be little differentiated given the heterogeneity of the current research landscape. Paired with their purely descriptive nature of analysis, they only provide basic research implications that lack thematic specification and thoroughness.

The above-mentioned studies in each cluster constitute a thorough selection of OGD-related literature reviews in peer-reviewed journals. However, a literature search in the databases of AIS, IEEE, and ACM shows that several literature reviews on OGD have also been published in conference proceedings, which also should be acknowledged at this point. These contributions can also be classified according to the proposed clusters and are subject to the same shortcomings and criticism. While the broad and very early approach of Novais et al. ( 2013 ) can be assigned to the third cluster of reviews, all of the other review attempts belong to the second cluster, as they focus on specific aspects in connection with OGD, in particular, barriers or problems associated with OGD implementation and development (Bachtiar et al., 2020 ; Crusoe & Melin, 2018 ; Neto et al., 2018 ; Roa et al., 2019 ), but also challenges and opportunities associated with OGD (Hassan & Twinomurinzi, 2018 ), or the impact of civil servants’ behavioral factors on the opening of government data (Kleiman et al., 2020 ).

Overall, the analysis of literature reviews confirms the conceptual autonomy of OGD and its independent research stream (emphasized in the above-mentioned definitional considerations), since eight out of twelve reviews are specifically dedicated to OGD. Our findings further show that previous review approaches lack theoretical integration of OGD issues and do not consider them in the context of the digital economy. Accordingly, they do not provide answers to our research questions of what we know about the antecedents, decisions, and outcomes of OGD and their relation in connection with the digital economy and how IS and digital business research can inform OGD research in this respect. Given the increasing importance of OGD and the digital economy as well as their reciprocal relationship, it is essential for the further development and a better understanding of the OGD concept to systematically theorize and synthesize the respective body of knowledge. Our systematic literature review goes beyond prior literature reviews and addresses their shortcomings by developing a theoretical review framework of antecedents, decisions, and outcomes of OGD, elaborating them in relation to the digital economy and deriving a theory-informed research agenda to tap the potential of IS and digital business research for OGD.

Methodology of the systematic literature review

Literature selection.

The literature review is based on established methodological recommendations regarding a general literature review’s overall structure and the related process of identification and selection of relevant studies (Tranfield et al., 2003 ; Webster & Watson, 2002 ). In order to comprehensively and systematically search for and select relevant studies, we followed further procedural guidelines according to the well-established PRISMA flow process adhering to its individual steps of identification, screening, eligibility, and final inclusion (Liberati et al., 2009 ).

To identify relevant records from established and relevant academic databases, we initially conducted a title, abstract, and subject search in different databases, including EBSCO (including Academic Search Premier, Business Source Premier, and EconLit with Full Text), Web of Science, ScienceDirect, and ProQuest . The search included the terms “open government data”, “data openness”, and “open data” in combination with “government” and “governance”. For the purpose of scientific rigor and quality, the search was limited to articles published in peer-reviewed academic journals in English language (Wang et al., 2019 ). Subsequent to the identification and elimination of duplicate records, editorial notes, and comments, the retrieved articles were first screened regarding title and abstract to determine and exclude irrelevant studies. The remaining articles were then subjected to a full-text review to exclude any studies that were not empirical and whose thematic focus was not clearly attributed to the field of OGD. This initial literature approach resulted in a total of 125 articles conforming to the selection criteria. To complement this set of literature with meaningful conference papers, we likewise searched the databases of AIS, IEEE, and ACM, yielding another 37 relevant articles. To minimize the risk of missing relevant studies, we finally screened the Google Scholar database using the same search terms with attention to the same criteria, since Google Scholar is the most comprehensive database (Gusenbauer, 2019 ; Martín-Martín et al., 2020 ) and is considered to be especially useful for identifying influential studies within specific fields of research (Martín-Martín et al., 2017 ; Zientek et al., 2018 ). In this way, seven additional eligible studies were identified and added to the selection, resulting in a final set of 169 relevant studies from the overall literature search, which represents the basis of the following preparation and analysis. Similarly to the entire selection process and assessment of eligibility, the further review, coding, and classification of the literature was performed by two reviewers. They were supported by a third reviewer who took a mediating role to assist once again in case of disagreement. The analysis of the literature consisted of two steps. The first step of our approach comprised the identification of key topic clusters in the literature by means of a bottom-up coding approach in order to determine what kind of topics are actually prevalent in the literature without constraining the result to certain areas. The second step referred to the theoretical integration of these clusters by means of a framework-based approach. In the following, we explain the methodological procedures underlying these two steps of analysis in more detail.

Identification of key topic clusters

In this first step of the analysis, the individual studies were assigned to individual clusters according to their respective content and thematic structure. Due to the thematic complexity of OGD and the associated heterogeneity of research, as well as different foci of the individual studies, the development and final formulation of the individual key topic clusters were designed and refined through a stepwise systematic coding process. This coding process relied on the approach of Saldaña ( 2013 ) and incorporated techniques of initial coding and pattern coding. Initial coding is an open form of coding, in which qualitative information is broken down into discrete aspects. While initial coding is the first step of analysis and serves “as a starting point to provide the researcher with analytic leads for further exploration” (Saldaña, 2013 , p. 101), pattern coding takes the analysis to a higher and more abstract level by refining the codes developed in the initial coding step and merging them into superordinate categories. The openness of this two-step approach already indicates that it follows an inductive procedure without a predefined coding scheme. This means that the formed concepts or categories emerge from the given data, which is characteristic for a bottom-up approach (Urquhart, 2013 ). Following this procedure, relevant information from the respective studies was initially coded. The resulting codes were then carefully and repeatedly examined to determine patterns in terms of similarities, correlations, and dissimilarities. The respective key topic clusters were then compared regarding their overall degree of similarity or distinction and refined, if necessary, in order to achieve optimum accuracy and consistency. This procedure yielded a final set of six key topic clusters, including (1) general/conceptual development (OGD theory), (2) drivers/barriers (OGD antecedents), (3) adoption/usage/implementation (OGD decisions), (4) success/performance/value (OGD outcomes), (5) acceptance/satisfaction/trust in government (OGD impacts), and (6) policies/regulation/law (OGD governance). The literature was then analyzed and structured according to these key topic clusters and a number of other classification criteria, including study type, method of analysis, data collection, and research perspective. The results of this step of analysis are depicted in the overview and evolution of the OGD literature.

Theoretical integration of key topic clusters

The second step referred to the theoretical integration of these clusters and thus their arrangement in a common complex of meaning. Here, we applied a framework-based approach (Paul & Criado, 2020 ), developing an overarching theoretical review framework that organizes the theoretical relationships among the identified thematic clusters of OGD in terms of a relationship map (Watson & Webster, 2020 ). This framework-based approach to literature was informed by previous literature reviews (Kessler & Chakrabarti, 1996 ; Lane et al., 2006 ; Raisch & Birkinshaw, 2008 ) and is particularly based on the antecedents, decisions, and outcomes (ADO) framework by Paul and Benito ( 2018 ), which is regarded as “an excellent framework to organize the findings (i.e., constructs and its ensuing relationships) of past research in a structured assembly” (Lim et al., 2021 , p. 537). The ADO framework approach appeared to be particularly suitable as it provides overarching and general theoretically linked dimensions to which the specific clusters could be meaningfully assigned. Thus, the framework-based approach, i.e. the predefined dimensions of the ADO framework and their relationships given by prior literature, provides an established but at the same time only rough grid, which is specified with the core clusters identified by means of the bottom-up method in the first step of the analysis. The theoretical review framework developed in this second step of analysis and the corresponding theoretical integration of the core clusters in the context of the digital economy are presented in the synthesis of OGD literature. The framework finally also serves as a point of reference for deriving the theory-informed research agenda for IS and digital business research (Fig. ​ (Fig.1 1 ).

Development of open government data research

Overview and evolution of the OGD literature

To provide a better understanding of the extent and evolution of the empirical OGD literature, this section gives a brief overview of its general development and current state. To begin with, Fig.  1 illustrates the distribution of qualitative and quantitative empirical OGD studies over the last 10 years.

Considering that OGD has evolved as an independent research stream out of general open government and open data research, it is not surprising that empirical research on OGD developed with a certain time lag in comparison to both of these more general research streams. Although OGD-related research was initially, in particular, an integral part of open government research, the first empirical and dedicated OGD studies appeared in 2011. Academic interest has increased significantly since 2014 and, measured by the number of empirical studies, of which a total of 107 (about 63%) studies are of a qualitative and 62 (about 37%) are of a quantitative design, remains high. The peak in 2016 and 2017 is due to a comparatively greater number of pertinent conferences and respective publications in these years. The decline in 2020 may be a result of the coronavirus pandemic, which has disrupted and delayed research projects and funding in general (Callaway et al., 2020 ).

Corresponding to the allocation of qualitative and quantitative empirical studies, the majority of the studies apply qualitative content analyses based on either an individual or comparative approach (61.54%). The application of quantitative methods is consequently lower in total, whereby publications using methods of complex empirical research, such as regression analysis and structural equation modeling, with a combined share of 18.93%, number even fewer, as opposed to publications based on descriptive statistics (19.53%). Figure  2 depicts the distribution of the applied methods of analysis.

Number of studies according to applied method of analysis

Table  2 presents the identified key topic clusters and provides selected descriptive statistics how these key topics have been approached in terms of study type, data collection, and research perspective.

Overview of classification criteria and descriptive statistics of the literature review

Table  2 shows that the largest share of the research focuses on the key topic (4) OGD outcomes and accounts for 28.99% of the literature reviewed, which is not surprising given the extensive impact of OGD on different performance and success levels. The key topic, with an almost equal number of assigned studies, is the group (3) OGD decisions with 28.40%, followed by the key topics (2) OGD antecedents with 15.98%, and (1) OGD theory with 11.24%. While the share of studies in key topic (6) OGD governance remains in the double-digit percentage range (10.65%), the level of scientific interest measured by the number of publications within key topic (5) OGD impacts is significantly lower (4.73%). Furthermore, like the overall distribution of qualitative and quantitative empirical research occurs the composition with regard to the individual key topic clusters, so that the number of qualitative studies clearly predominates in each key topic. Notably, key topic (5) OGD impacts constitutes an exception, where the exact opposite is the case. This pattern can be explained by the fact that research on OGD is still at a relatively early stage.

In summary, the analysis reveals the great scope and heterogeneity of the research landscape of OGD in terms of research focus and methodology. The pronounced imbalance between qualitative and quantitative studies in favor of the former indicates that OGD is still an emerging field of research. Given this emergent state of research, quantitative empirical studies are essential to confirm causality of theoretical relationships and effects of evolving issues proposed by conceptual or qualitative research, and to address associated concerns of validity. In particular, little empirical robust knowledge is available in the areas of acceptance/satisfaction/trust in government, policies/regulation/law general/conceptual development, and drivers/barriers. This also holds when it comes to understanding the user perspective in the context of OGD, which is generally neglected in the field, but in particular in these areas. A remarkable exception to this pattern is the area of acceptance/satisfaction/trust in government, which has so far only focused on the user perspective, while disregarding the provider perspective. However, this would be especially important in view of the struggling implementation and diffusion of OGD in several public organizations. The user perspective so far has also strongly emphasized the individual level (e.g., citizens) and should increasingly consider the organizational level (e.g., firms) for a better understanding of the role of OGD in the digital economy.

Synthesis of the OGD literature

The synthesis of the OGD literature is based on the theoretical review framework and theoretically integrates the previously identified key topic clusters with reference to the digital economy. Figure  3 depicts the review framework and the theoretical relationships among the identified key topic clusters.

Overarching theoretical review framework

The framework may serve as a thematic relationship map of empirical OGD research, particularly illustrating the associations among antecedents, decisions, and outcomes of OGD, as well respective focus areas of research and neglected topics. The antecedents in terms of the drivers and barriers explain the reasons for a certain behavior, while decisions determine the forms of behavior (i.e. adoption, usage, or implementation of OGD), and outcomes comprise the assessments that result from decisions and the associated behavior (i.e., success, performance, and value or acceptance, satisfaction, and trust in government) (Lim et al., 2021 ). All these processes take place in a governance and regulatory setting, in which policies, regulation, and law may affect this process in terms of institutional moderators. These layers underlie the general and conceptual development of OGD, which is the overarching object of action and knowledge, and thus constitutes the point of reference for all other elements in the framework. The synthesis of OGD literature is conducted along these dimensions in the following.

General conceptual development of OGD

Perspectives on ogd.

Regarding the general conceptual development of ‘Open Government Data’, various studies contrast four ways of perceiving the term in recent years (Alexopoulos et al., 2018 ; Gonzalez-Zapata & Heeks, 2015 ; Jetzek et al., 2013 ): (1) the bureaucratic perspective conceiving OGD as a bureaucratic mechanism to enhance information quality, effectiveness and efficiency of government policy making, and legitimacy of polices (cf. Alexopoulos et al., 2018 ; Gonzalez-Zapata & Heeks 2015 ), (2) the technological perspective conceiving OGD as a technological innovation of public administration building up a data infrastructure to host a freely available public database of accurate, complete, and timely public sector data (cf. McNutt et al., 2016 ; Meijer, 2015 ), (3) the political perspective conceiving OGD as a part of government accountability to the citizens, thus providing insights into government affairs, transparency of governmental action, and the option for civic participation in policymaking (cf. Zhao & Fan, 2018 ; Meijer, 2015 ), and (4) the economic perspective conceiving OGD as source of economic value creation, providing several opportunities for the commercialization of these data in new goods and services (cf. McBride et al., 2019 ; Zhao & Fan, 2018 ; Berrone et al., 2017 ).

The digital economy’s role in the OGD ecosystem

Against the background of the OGD ecosystem model presented by Dawes et al. ( 2016 ), these perspectives of the literature can be interpreted as to portray four fields of stakeholder interactions in OGD settings. In this context, the bureaucratic perspective focuses on the interaction between the policymakers and the implementing authorities by surveilling the effects (increase in the quality of information, the effectiveness of administrative action, the legitimacy of public policy) (cf. Alexopoulos et al., 2018 ), while the political perspective regards OGD as a means for democratic processes and decision-making, as it investigates the role of OGD in government accountability, transparency, and citizen participation. Correspondingly, the technological perspective portrays the interaction between OGD providers (public authorities) and OGD intermediaries (i.e., the digital economy) by stating the technical characteristics of the data infrastructure. The economic perspective, however, lays its focus upon the creation of value for the OGD customers, i.e., the citizens, by investigating how OGD yields public value to them. Against this background, firms of the digital economy assume an intermediary function matching technical data supply from the government with information demand of the OGD customers. In a nutshell, the task of digital firms in the OGD ecosystem is to access the data supplied by the government, to gather the information contained in OGD by electronic data processing and analytics software, and to commercialize this information in their products and services. Figure  4 outlines the OGD ecosystem and sketches the role of the digital economy as a data intermediary facilitating the interaction between the executive government authorities and the citizens.

Open government data ecosystem (based on Dawes et al., 2016 and Kassen, 2013 )

Scope of government activity and the digital economy

Besides the general role of the digital economy, both the scope of digital business opportunities and the business approach are crucial to the digital economy. In this context, the literature raises interesting points regarding the scope of government activity in data-based service provision. Some studies find evidence for governments simply providing public data and setting the legal and technical framework by data formats and access rights, while leaving further processing and marketizing of these data completely to interested stakeholders, like NGOs, companies, or private citizens (cf. Alexopoulos et al., 2018 ; Berrone et al., 2017 ; Dawes et al., 2016 ; McNutt et al., 2016 ). However, another strand of OGD literature finds more complex forms of governmental open data platforms, providing data via APIs and data-based apps that enable the user to filter and manipulate the chosen data set and to embed the data in other data processing programs (cf. McBride et al., 2019 ; Zhao & Fan, 2018 ; Berrone et al., 2017 ). In this case, government provides OGD products and services on its own in competition to possible private sector offerings. In this context, contemporary OGD research presents a spectrum of government involvement in the presentation and processing of publicly accessible data by presenting diverging roles of government in OGD programs, i.e. data provision and standard-setting versus data service platform hosting. Consequently, the scope of government activity and the sophistication of governmental data infrastructures for the compilation, analysis, and provision of public sector data significantly influences the economic margin and targets of digital private business with OGD. Besides the theoretical setting of the digital economy’s role in the OGD ecosystem, the antecedents of OGD programs, the decisions and actions taken by the government for OGD implementation, as well as the achieved outcomes and impacts also determine the position of the digital economy in OGD programs and how to create value from public sector data.

The following subsection provides a synthesis of the findings of previous research on the antecedents, decisions, and outcomes of OGD with special reference to the digital economy, elaborating their significance for IS and digital business research (Table  1 in the online Appendix summarizes these findings). The representative studies presented in the following subsection (and in Table  1 in the online Appendix) were selected due to their high resonance in scientific research (high Google Scholar citation score) and their publication in particularly influential scientific, peer-reviewed journals (high journal impact score).

Antecedents, decisions, and outcomes of OGD and the digital economy

Ogd antecedents: drivers and barriers.

When considering the antecedents and determinants of OGD programs, previous studies more often refer to barriers emerging from the OGD ecosystem (cf. Barry & Bannister, 2014 ; Janssen et al., 2012 ; Ruijer et al., 2017 ), rather than the drivers and enablers (cf. Young, 2020 ; Zhenbin et al., 2020 ; Susha et al., 2015 ). For the factors triggering or fostering OGD policies, the findings of previous studies distinguish among political and social factors, operational and technical properties of agency equipment, or economic opportunities for OGD usage. In case of political and social OGD determinants, political and social demand for transparency and accountability (Barry & Bannister, 2014 ; Janssen et al., 2012 ; Zhenbin et al., 2020 ) is perceived as a major trigger for OGD programs alongside with increasing citizen engagement and participation in government affairs (Young, 2020 ; Welch et al., 2016 ). Regarding the operational and technical drivers, previous studies highlight the importance of a cultural anchorage of electronic data processing and sharing in public administration (Zhenbin et al., 2020 , Yang et al., 2015 ) in combination with a well-developed data infrastructure within the agency operated by qualified specialists (Young, 2020 ; Welch et al. 2016 ). In this context, economic pressure arises from a large share of private companies providing public services to the citizens for profit. Studies such as Young ( 2020 ) find that the opportunity to augment extant or create new public services by using public sector data bears opportunities to create new sources for economic growth (cf. Young, 2020 ; Zhenbin et al., 2020 ; Susha et al., 2015 ). This is even more the case if the national economy possesses the resources for exploiting the information contained in public sector data (high GDP) and exhibits a large productivity in providing ICT services (high share of the IT industry) (cf. Young, 2020 ; Susha et al., 2015 ). In this context, the state of the digital economy as well as the maturity of governmental data infrastructures appear as drivers for both the successful implementation of OGD programs and the successful exploitation of these data in public services. Consequently, IT firms thus function as software and hardware suppliers to public administration in digitally underdeveloped economies, while they assume the role of a private sector competitor in the delivery of public services in digitally advanced countries.

Barriers to implementing an OGD policy emerge from problems with (1) data compilation on the part of the government or the executive agencies (institutional constraints), with (2) data access caused by technical failures or dysfunctional data portals (technical constraints), or with (3) data application on the part of the citizens (societal barriers). Accordingly, data compilation barriers refer to factors that hinder the respective agencies to collect, compile, or transfer suitable data due to legal constraints (Yang et al., 2015 ; Barry & Bannister, 2014 ), due to the complexity of the organizational structures of government agencies (Ruijer et al., 2017 ; Welch et al., 2016 ; Yang et al., 2015 ), and/or due to the lack of their data management capacities and capabilities (Ruijer et al., 2017 ; Young, 2020 ). In contrast, data access barriers emerge from the properties of the data infrastructure. Major impediments in data access arise from a lack in system interoperability if governmental software and data formats are not compatible with its civic counterparts (Smith & Sandberg, 2018 ; Barry & Bannister, 2014 ) or from a lack in technical support and constant updating of data platforms due to staff shortages (Janssen et al., 2012 ). Furthermore, the literature also finds that the introduction of registered access to public data creates another great obstacle for OGD as most people are unwilling to register officially on public data platforms for occasional data access (cf.Barry & Bannister, 2014 ; Ruijer et al., 2017 ). Regarding the obstacles emerging from the properties of the user, i.e. the citizens, previous research argues that the success of OGD programs is to be attached to the ability of society to make use of the published data. Obstacles emerge from the inability of the users to achieve a practical use of these data; this might either be due to the societal inability of information processing (e.g., low ICT equipment, low levels of education, low income, etc.) (Barry & Bannister, 2014 ; Ruijer et al., 2017 ), or due to the uselessness of the provided data such that the citizens cannot apply the information to achieve any value (Smith & Sandberg, 2018 ; Janssen et al., 2012 ). Considering these findings, all barriers provide starting points for digital business to step in and solve the issue. In case of data compilation constraints, IT firms adapt solutions from private sector products and services to provide a customized data infrastructure to public authorities aiming to publish their data. To overcome data access barriers, private IT firms host government data for public retrieval as business partners of public authorities and provide the information via their own data services and applications. Finally, to solve data application barriers, the digital economy provides IT specialists and data analysts processing government data and create a useful summary and analysis of OGD for the citizens.

OGD decisions: Adoption, usage, and implementation

Although the relevant drivers and obstacles open corresponding business opportunities for the digital economy, actual policy decisions regarding the adoption of OGD measures, as well as their implementation and subsequent use, are of crucial importance for business practice. As stated before, government activity in providing data-based applications to its citizens is of major importance for the type of digital business. Accordingly, previous research analyzed the decisions regarding OGD policy and strategy as well as the intensity of governmental OGD activities (Gascó-Hernandez et al., 2018 ; Dawes et al., 2016 ). Depending on the scope of governmental data processing and data-based service provision, Dawes et al. ( 2016 ) propose a spectrum of OGD policies presenting three archetypes of OGD strategy, starting with (1) the data-oriented OGD policy aiming at the provision of accurate, unbiased datasets from public sector entities without any further service features (cf. Wang & Lo, 2016 ; Yang & Wu, 2016 ), followed by (2) the intermediate program-oriented OGD policy providing public data via an OGD platform displaying basic data analysis features and APIs (cf. Chatfield & Reddick, 2017 ; Parycek et al., 2014 ), ending up with (3) the use- and user-oriented OGD policy focusing on the creation of public value by embedding public sector data within data-based public services (Gascó-Hernandez et al., 2018 ).

Despite these strategic considerations, governmental adoption decisions also have a major impact upon the organizational and technical preparations to get public administration ready for OGD (Chatfield & Reddick, 2017 ; Yang & Wu, 2016 ; Parycek et al., 2014 ). Closely connected to the strategic setting is the scope of publication permissions from high-level authorities ranging from data publication restrictions to the support of interactive data services. Furthermore, the government’s adoption decisions also shape the maturity of the authorities’ data infrastructure by defining the technical capacity as well as the interoperability and connectivity to citizen devices (Bonina & Eaton, 2020 ; Wang & Lo, 2016 ). Consequently, the ex-ante decisions regarding the adoption of OGD measures also define the way of doing business with OGD. In this regard, the strategic positioning of governmental OGD activities directly determines the scope of the intermediary role of the digital economy. In case of a data-oriented OGD program relying upon a mediocre public data infrastructure, the intermediary role of the digital economy achieves its climax as the government acts as a proper data provider, leaving data analysis, application, and embedment in public services completely to digital firms. However, privatization of data-based public services diminishes if the OGD program place special emphasis upon the user. For a user-oriented OGD program equipped with a well-developed public data infrastructure, utilizing OGD for providing data-based public services is completely in the hands of the government, whereas IT firms provide IT expertise and software solutions to the authorities.

Besides the determining character of ex-ante decisions for digital business with OGD, the ex-post decisions of the government flanking the OGD program also provide opportunities for the digital economy. Linked to the strategic setting of the OGD program is the decision for the target group and user profile of the program (Smith & Sandberg, 2018 ; Parycek et al., 2014 ). Depending on the respective policy intensity, government must decide whether (1) to grant general access for the average citizen in case of a user-oriented approach, or (2) to grant licensed commercial access enabling the embedment of OGD in the products and services offered by private IT firms in case of a program-oriented OGD approach, or (3) to grant access only to IT specialists for retrieving information via data analytics in case of a data-oriented OGD approach.

Furthermore, previous research also investigated the ensuing decisions concerning the interface design and the related features of OGD portals (Wirtz et al., 2019 ; Chatfield & Reddick, 2017 ). Accordingly, OGD portals diverge in the scope of the provided datasets, in the scope of the OGD interface as well as the scope of data service functions, ranging from mere data downloads from government websites to data service hubs created by OGD platforms. As a result, the user profile targeted by the OGD program as well as the design and features of the OGD interface shape business approaches for OGD. Accordingly, IT firms seek to gather, process, and capture value by commercializing OGD in products and services for the citizens in case of a licensed access and a low scope of OGD data service features, responding to the demand of proper data processing on the demand side of the OGD ecosystem. In case of limited specialist access and a high scope of data service functions, IT firms switch towards offering data analytics services to the authorities involved, equivalently responding to the demand of supply-sided data processing and analytics (cf. Bonina & Eaton, 2020 ).

Another relevant field for government decisions flanking the implementation of OGD programs refers to the creation of IT skills and technical expertise required for data management by public authorities (Gascó-Hernandez et al., 2018 ; Wirtz et al., 2019 ; Yang & Wu, 2016 ). Regarding the timescale and the addressees of these measures, current research distinguishes between short- to mid-term educational measures for public employees developing OGD skills and capabilities (cf. Safarov, 2019 ; Yang & Wu 2016 ) and long-term educational measures, increasing common IT knowledge among the population (cf. Gascó-Hernandez et al., 2018 ; Wirtz et al., 2018 ). Short- to mid-term OGD skill development is associated with a variety of options, ranging from internal IT trainings with the respective authorities (Yang & Wu, 2016 ) to joint ventures with the digital economy (Safarov, 2019 ). This decision area thus offers several linkages to digital business, spanning from the provision of training programs for public administration to learning-on-the-job in collaborative partnerships for OGD processing and evaluation. Regarding long-term public IT schooling, the government aims at building up IT skills and capabilities among the population in order to gain skilled employees for public administration (cf. Gascó-Hernandez et al., 2018 ). As a result, private-sector IT companies sell their know-how and IT expertise to educational institutions as mentoring partners for IT practice. All in all, the digital economy assumes the role of a catalyst in the field of digital education and training of the people - as trainers and administrative partners in the short term and as mentors in the long run.

OGD outcomes: Success, performance, and value

Finally, it is of crucial importance not only to the government and public administration whether an OGD program pays off in terms of efficiency, citizen satisfaction, and trust in government. For the digital economy, the question is whether accessing and utilizing OGD provides access to new products and services as well as whether OGD can create new markets for data-based public services. Regarding the outcomes achieved by OGD implementation, most studies refer to the internal effects upon the performance of public administration, such as efficiency gains in administrative procedures and public service provision (Mergel et al., 2018 ; Worthy, 2015 ), transparency of political decisions and policy-making (Wang & Shepherd, 2020 ; Marjanovic & Cecez-Kecmanovic, 2017 ; Jetzek et al., 2014 ), or behavioral effects upon public employees (Marjanovic & Cecez-Kecmanovic, 2017 ; Worthy, 2015 ).

In contrast to these specific administrative and political issues, some studies also refer to spill-over effects upon the interaction of citizens with public authorities (interaction effects), the distribution of information among the population (information effects), as well as the innovation of public services by utilizing OGD (commercialization/innovation effects). Considering interaction effects upon the participation and involvement of citizens into public affairs, previous studies observe a positive effect in citizen engagement in case of OGD programs. Although there is evidence of negative OGD effects upon the polarization in political debates due to different interpretations of government data (cf. Worthy, 2015 ), most studies report positive effects, such as public service innovation through co-creation with citizens and IT firms or synergy effects due to simplified data sharing in collaborations between government agencies and external service providers (Ruijer & Meijer, 2020 ; Máchová & Lněnička, 2017 ; Jetzek et al., 2014 ). Having this mind, interaction effects of OGD programs enable the digital economy to serve as a moderator, facilitating the interaction between government and citizens by easing information processing on the part of the citizens and communication to the citizens on the part of public administration. Furthermore, IT firms relying upon big data analytics might experience competitive advantages in comparison to their international competitors as the cost for gathering public sector data decreases significantly. Consequently, citizen engagement and data sharing provide economic growth potentials to the digital economy. This is also in line with the commercialization and innovation effects observed by several studies (Jetzek et al., 2019 ; Mergel et al., 2018 ; Jetzek et al., 2014 ). Accordingly, previous research finds evidence for OGD spillover effects to the private sector, as implementing OGD enables digital firms to access new information at lower cost, and to generate a footage in the public sector by developing new markets for data-based products and public services.

Acceptance, satisfaction, and trust in government

Considering the consequences on technology acceptance and citizen satisfaction triggered by OGD, previous research observes a positive impact fostered by several preconditions. In case of technology acceptance, studies find that a positive impact relies upon (1) sufficiently intense Internet usage among the population (Gonzálvez-Gallego et al., 2020 ; Afful-Dadzie & Afful-Dadzie, 2017 ), (2) the awareness of individual benefits that emerge when using and applying OGD (Zuiderwijk et al., 2015 ; De Kool & Bekkers, 2014 ), and (3) the degree of OGD usage obligation in G2C interactions (Gonzálvez-Gallego et al., 2020 ; Zuiderwijk et al., 2015 ). Considering citizen satisfaction, broad acceptance and public support of OGD and its application appear as necessary conditions alongside with a sufficiently high information quality, system quality, and service quality (cf. Gonzálvez-Gallego et al., 2020 ). Hence, the maturity of a country’s digital economy directly moderates the impact of OGD on technology acceptance and citizen satisfaction. This is due to developed digital economies displaying both a widespread use of ICT devices and their intensive usage, as well as common IT knowledge among the people. In addition, resident digital firms are in a much better position to support a well-functioning public data infrastructure in the case of an advanced IT industry.

In summary, it can be stated that from the perspective of public administration, the digital economy constitutes both a driver of OGD adoption and a warrant for successfully implementing an OGD program. From the perspective of the digital economy, however, OGD represents a new source of economic growth and business model innovation based upon the development of new resources, i.e., public sector data, and new business opportunities emerging during OGD adoption and implementation.

Research agenda

The preceding identification of OGD key topic clusters and their synthesis into a theoretical framework with special reference to the digital economy has revealed significant points of connection to IS and digital business research and enables us to develop a theory-informed research agenda for the latter. Although the prior literature review emphasized particularly the core dimensions of the ADO framework, the findings also yield implications for the key topics (1) OGD theory and (6) OGD governance.

(1) OGD theory: General/conceptual development

As the OGD ecosystem theorizes that firms of the digital economy assume an intermediary function matching technical data supply from public authorities with the demand for information on the part of the citizens, empirical research needs to verify how this assumption holds true in practice. Furthermore, future research needs to clarify the impact of government activity and OGD infrastructure maturity upon the business models of related IT firms. Consequently, McBride et al. ( 2019 ) postulate the need for further empirical research, which would enable comparison and differentiation of individual OGD services in their emergence, orientation, and goals. McBride et al. ( 2019 ) consider this especially with regard to data platforms and OGD services, which increasingly evolve from different sources. This corresponds with the implications pointed out by other researchers who identify further needs for empirical research on the characteristics of OGD sources in connection with different national contexts (Alexopoulos et al., 2018 ), data platforms collaboratively developed in joint ventures with IT firms (Meijer & Potjer, 2018 ), and the changes in the OGD portals’ datasets over time (Di Wang et al., 2018 ). Hence, more empirical research is needed, especially case studies regarding the economic OGD perspective, to determine the scope of involvement of private IT firms in OGD programs in general as well as their function within the whole OGD ecosystem in practice. Despite that, the scope of government activity in data-based service provision needs further investigation regarding its impact on the business approach of the digital economy. Consequently, the following questions may guide further research in this direction: How are firms of the digital economy involved in contemporary OGD programs? What is the function/business of digital IT firms in respective OGD programs? How does the scope of governmental OGD activity alter the business model of digital firms?

(2) OGD antecedents: Drivers/barriers

Considering external OGD drivers and barriers, the preceding analysis of OGD research revealed the productivity of the IT industry, as well as the GDP share of the digital economy as key drivers of successful OGD programs. Thus, establishing a causal linkage between the size of the IT industry, the share of the digital economy, and the maturity of OGD programs appears as a suitable goal for further empirical research. Linked to this idea is also the idea of Shao and Saxena ( 2019 ) raising the question of how a society’s cultural characteristics and traditional values act as drivers and/or barriers to the intentions of administrative implementation and the participation of external actors within OGD initiatives. Consequently, the following research questions appear as a good starting point for analyzing OGD drivers and barriers emerging from the digital environment: Does a high productivity of IT firms and large share of the digital economy increase the success of OGD initiatives? Which socioeconomic, demographic, and cultural characteristics of the economy drive or impede OGD implementation?

Turning towards drivers and barriers from inside public authorities, Zhenbin et al. ( 2020 ), for instance, name the need to further investigate which specific drivers influence the motivation of government agencies to engage in OGD development and public service innovation. This has been similarly formulated by Fan and Zhao ( 2017 ), who, in addition to examining the question of which influences generally exert pressure on the internal, organizational orientation in relation to OGD activities, also emphasize the need for further research on the extensive influence of the media. With regard to policy constraints, Young ( 2020 ) identifies the risk within public institutions of intentionally withholding data/information that could be detrimental to the publisher and postulates the need to investigate more closely the existence of these barriers and their potentially negative consequences in the future. Considering the findings from the qualitative literature synthesis, the question arises as to whether collaboration with private IT companies results in a reduction of barriers or an activation of drivers within the agency. This could be empirically determined and investigated in particular by means of interviews and questionnaires. Possible research questions in this direction would be: To what extent do data access, data processing, and data application in public services improve due to collaboration with private IT companies? To what extent do intensive G2B interactions regarding OGD contribute to its successful implementation?

(3) OGD decisions: Adoption/usage/implementation

While synthesizing the findings of previous studies, it became clear that the strategic positioning in OGD adoption, the target groups for OGD usage, as well as the organizational OGD readiness for OGD implementation have a significant impact on the orientation of the corresponding OGD business models. In light of these findings, two promising directions of research emerge for the IS research community investigating OGD in the context of the digital economy: (1) the empirical verification of the assumed correlation between the user-orientation of governmental OGD initiatives and the predominant customer alignment of IT firms’ OGD business models, and (2) the case-study-based investigation of the causal relationship between OGD access barriers and the share of the digital economy in providing data-based public services. Overall, the need for further, user-focused research is obvious and acknowledged. For example, there is a need to identify the types of datasets users of OGD require in order to enable even more active participation and usage (Chorley, 2017 ) and to understand how external users can be motivated to become permanent participants in OGD, while respecting their job situation and other cultural influences (Hermanto et al., 2018 ). Smith and Sandberg ( 2018 ) also point out that instead of the usual data-centric research, more user-centric OGD research is needed in future. In this context, the established theories of IS and digital business research such as the Technology Acceptance Model (TAM), the Unified Theory of Acceptance and Use of Technology (UTAUT), and the DeLone-McLean IS Success Model become particularly important for the further development of this field of research. Accordingly, the following research questions may guide scholars in conducting further research concerning the OGD adoption and usage approaches: How can IS theories and explanatory models, in particular, the TAM, UTAUT, and IS Success Model be applied in the context of OGD research and theory development to explain acceptance, adoption and usage behavior? How does governmental customization of OGD alter the value proposition and customer composition of OGD business models? What is the impact of OGD access restrictions on the business practices of the IT firms involved?

Another interesting avenue for further research connecting OGD to IS and digital business studies is the topic of building up relevant OGD skills and educational support. The findings from the literature synthesis suggest the digital economy to serve as a catalyst in digital education providing skills and knowledge in the short run, and innovative spirit and educational support in the long run. In this respect, Safarov ( 2019 ) points out that it might be useful to examine in more detail the design and impact of various OGD activities, such as open data awards or specific training programs. Several other researchers also discuss the necessity and value of findings based on integrative methods and trainings regarding the implementation and usage of OGD. In this way, among other things, experimental studies can be performed to determine which training methods can be used most successfully in relation to specific content and data sets in order to ensure a lasting curiosity and interest in OGD (Gascó-Hernández et al., 2018 ).

Further long-term studies will also show how government institutions’ perceptions and usage behavior change over time as the methods are compared (Altayar, 2018 ; Wang & Lo, 2016 ). Wirtz et al. ( 2018 ) postulate the need for further research to examine the degree to which the usage behavior of citizens changes over time and which situational and socio-cultural aspects play a role in this process. In this respect, in addition to the use of longitudinal studies, comparative cross-cultural or cross-country studies can also be used to identify relevant differences and investigate their consequences for user behavior (Saxena, 2018 ). Considering these demands for further research, the following research questions may inspire research regarding the role of the digital economy in creating digital OGD literacy: Do G2B partnerships in OGD increase the digital literacy of public employees? Do OGD training programs and educational measures have a greater effect on the trainees if education involves cooperation with IT firms?

(4) OGD outcomes: Success/performance/value

Since current research on OGD outcomes is concerned with the question of how OGD offers socioeconomic added value to society, there are also potential spin-offs for the digital economy in this context. In the preceding literature synthesis, it became clear that the establishment of OGD programs could generate spillover effects on the competitiveness and innovative strength of the digital economy. Accordingly, the empirical investigation of these effects by means of case studies and time series analyses appears to be a promising goal for further research. Specifically, the following research questions suggest themselves in this context: How does the successful implementation of OGD initiatives affect the competitiveness of IT firms? Is there evidence for a causal relationship between the implementation of OGD programs and economic growth in the digital economy?

However, answering these specific research questions depends largely on the ability to record and evaluate the performance and resultant success of OGD activities. Since the success of OGD activities to be determined or measured extends to many areas among public institutions and external stakeholders, it is generally difficult to comprehensively classify and evaluate success and failure. In response to the challenges posed by the above-mentioned reasons, Marmier and Mettler ( 2020 ) postulate the need for additional research on the level of dedicated quality measurement and evaluation of OGD and its measurement instruments. Similarly, Jetzek et al. ( 2019 ) argue that the answer to the question of how data constructs and their quality are to be measured at the societal level poses another future research need.

Another relevant issue involves the potential value contribution of OGD and describes the need for further research to identify the potential contribution of OGD activities in terms of overall value creation in terms of social, economic, and public value. The origin of this value creation lies in the fact that data from public institutions are first made available in an appropriate quality, wherefrom Luna-Reyes et al. ( 2019 ) derive the need for further research to identify suitable governance and leadership approaches and to examine their influence on the quality of the data to be emitted. Mergel et al. ( 2018 ) further emphasize the large amount of valuable innovations that can be triggered by OGD and point out the need for further research in this regard to broaden and strengthen existing knowledge. Magalhaes and Roseira ( 2020 ) present similar points and show that, albeit the increasing recognition of the potential value for the private business sector, the reasons for or against integrating OGD into business processes, and thus also the potential economic value that can be achieved, still often remain unexploited or even unclear. They emphasize the need for further in-depth analysis at the firm level in order to move from a general top view to explicit insights into the behavior of and consequences for firms in their interactions with OGD. A research question of central importance might consequently be: How can dedicated products and processes be explored and exploited in order to generate sustainable economic and public value in different OGD contexts?

(5) OGD impacts: Acceptance/satisfaction/trust in government

The synthesis of the existing literature on the topic of the consequences and impacts of OGD programs on the general acceptance of OGD, the satisfaction of citizens with its use, as well as the resulting trust in government policy suggests that these impacts are all the stronger in case of a well-developed digital economy. As argued above, this is due to (1) widespread usage of ICT devices among the population, (2) IT-related customer preferences and usage perceptions, and (3) technical support from private IT firms. Taking this implication as a starting point for further research raises the following questions: Does the maturity of the digital infrastructure moderate OGD acceptance and user satisfaction? Do joint ventures of government and private IT firms providing OGD services to the public increase trust in open government?

Against this background, the investigation of external stakeholders’ perceptions and preferences is of central importance and determines the need for further research to explore and scrutinize the differing perceptions and preferences of various stakeholders in terms of OGD activities and outcomes by international comparison. Further research efforts should therefore be undertaken to examine and compare preferences and perceived satisfaction at both the citizen (Saxena & Janssen, 2017 ) and corporate levels (Afful-Dadzie & Afful-Dadzie, 2017 ). Due to the small number of studies dedicated to OGD impacts, it is of interest to broaden the focus from the external stakeholders to an in-depth investigation of the acceptance and satisfaction of governmental agencies’ internal forces, as these act as a starting point or barrier to subsequent external perception and satisfaction (Barry & Bannister, 2014 ). Consequently, scientific progress within the field of OGD antecedents might also spark research efforts in OGD impacts.

(6) OGD governance: Policies/regulation/law

Following the research implications regarding the strategic alignment of OGD programs and the corresponding OGD policy intensity, two research areas become apparent within which further research efforts can contribute to a better understanding of the specific context: the normative composition and implementation of OGD and the potential impacts of norms and policies. For the research area of normative composition and implementation of OGD it is stated that the far-reaching innovations for the state and the economy emerging from the implementation and use of open data in general and OGD in particular, require dedicated and appropriate policies from state authorities. Thus, Khurshid et al. ( 2019 ) state that in the future it will be important to understand the reasons for slow diffusion and a consequently weak adoption of general data policies at the organizational and individual levels. Furthermore, procedural metadata standards and general data quality standards should be preceded by further research (Máchová & Lněnička, 2017 ; Shepherd et al., 2019 ).

In addition to general overview studies, further in-depth analyses of applied standards and directives should be conducted in the future, which in turn will help to provide stronger guidelines for the development of data policies. Regarding the potential impacts of norms and OGD policies, further research is needed to determine how the formulation and implementation of data policies and normative guidelines affect other core aspects, such as subsequent use or the general contribution to success (Kurtz et al., 2019 ). Moreover, it is necessary to investigate, how specific policies that focus on the commercial value of OGD contain the risk of conflict with other open data values (Zuiderwijk et al., 2016 ). In order to identify and classify corresponding dependencies and consequences in this context, comparative and qualitative exploratory approaches are promising to derive conclusions from related policies and directives.

In summary, a number of starting points for IS and digital business research emerge from the findings and insights of previous studies on the various OGD research areas. In the context of the consideration within the ADO framework, various parallels between the identified research questions also become apparent. To provide a general overview of these research implications, Fig.  5 reflects the relevant research questions and depicts their integration into the theoretical review framework in terms of a research agenda for IS and digital business research.

Theory-informed research agenda for IS and digital business research

Discussion and conclusion

Data have become an inherent part and essential driver of the digital economy. The field of OGD has been largely neglected by IS and digital business research, despite its great value potential for firms and the digital economy as a whole. As governments, public organizations, and firms worldwide are struggling in exploiting the full potential of OGD for the digital economy, it is essential to gain a comprehensive understanding of OGD and to frame the concept more broadly in the context of the digital economy in order to advance the field of research accordingly. On the one hand, this particularly requires greater involvement of the IS community in the very interdisciplinary field of OGD research, which is currently dominated by the public administration and public management perspective. On the other hand, it is necessary to theoretically integrate and synthesize the vast body of knowledge to identify research gaps and provide valid research directions.

An important requirement to achieve this is first and foremost conceptual clarity of OGD, which sometimes has been confounded with the related concepts of open government and open data. Our study goes beyond prior research (e.g., Hossain et al., 2016 ; Tai, 2021 ; Wirtz & Birkmeyer, 2015 ) by demonstrating and taking account of the – widely implicitly and tacitly assumed – conceptual autonomy of OGD and acknowledging it as an independent research stream closely related but still distinct from open government and open data research. This is a vital prerequisite for drawing differentiated and valid conclusions for the field and for gaining a clear understanding of the phenomenon. In this connection, we further build on and extend the general conceptual development of OGD and respective studies (e.g., Dawes et al., 2016 ; Kassen, 2013 ) by consolidating different OGD perspectives from the literature and by outlining the role of the digital economy in the OGD ecosystem and the digital economy’s relation to OGD-related government activity.

While previous research has made valuable contributions in structuring the OGD research landscape (e.g., Saxena, 2018 ; Zuiderwijk et al., 2014 ) and analysing certain OGD issues (e.g., Attard et al., 2015 ; Purwanto et al., 2020 ; Safarov et al., 2017 ), it fails to theoretically integrate the OGD concept and its key issues, and neglects the increasingly relevant relationship between OGD and the digital economy.

This study seeks to fill in this gap by conducting a systematic literature review of empirical OGD studies, which synthesizes the body of knowledge into a theoretical framework of OGD antecedents, decisions, and outcomes with special reference to the digital economy, and which further proposes a theory-informed research agenda for IS and digital business research.

Against this background, this study generally stands in line with and extends the findings of earlier comprehensive review approaches towards OGD literature, in particular those of Zuiderwijk et al. ( 2014 ) and Saxena ( 2018 ). However, these studies lack in the coherent linkage and the display of causal relationships between the different research areas as these studies mostly follow a descriptive approach attempting to present a common denominator of the characteristics of the individual studies. This study goes beyond their purely descriptive perspective by developing an overarching theoretical review framework that models the theoretical relationships of the thematic clusters identified in the literature analysis. In addition, this study also captures the more recent developments and novel empirical insights in the field of OGD. This is especially true for the area of OGD outcomes, for which research is based on a mature implementation of OGD systems in administrative practice, but also when it comes to issues such as organizational readiness and OGD skill development in the area of OGD decisions. Moreover, by examining the OGD literature with special reference to the digital economy, our study conceptually intersects with relevant IS and digital business research, demonstrating an interdisciplinary research approach that has been missing in prior OGD literature reviews.

Taken together, the theoretical attempt in conjunction with the focus on the digital economy and the associated inclusion of an IS perspective constitutes a new approach towards OGD literature that yielded novel insights into the field by integrating and explaining scientific progress in emergent topics such as in the areas of OGD decisions and OGD outcomes. Thus, the theoretical contribution of our study to the literature in terms of originality results from the theoretical review framework that theoretically integrates previously separated thematic clusters of OGD and their points of connection to IS and digital business research, thus improving our theoretical knowledge of the field of OGD and its relation to the digital economy. Overall, the synthesis of OGD literature into this theoretical framework represents the main response to our first research question of what we know about the antecedents, decisions, and outcomes of OGD and their relations in the context of the digital economy.

In this context, bridging the gap to digital business is of particular importance as this study represents the first attempt to transfer findings and insights from the mainly public administration- and public management-driven OGD studies to the IS and digital business research domains which might spark further progression in OGD research. The research agenda derived in accordance with the theoretical framework reveals how OGD research may relate to adjacent fields of IS and digital business research, such as interface design, IT and data governance, data security, big data analytics, open data, etc., and provides concrete opportunities and research questions in each thematic cluster.

Although the review provides valuable insights into each of the six key topics, the OGD outcomes appear to be of particular importance. This is not only indicated by the fact that this cluster already comprises the largest number of studies in relation to the other clusters, but also in view of very fundamental unresolved issues pertaining to the digital economy. We know today that the use of OGD opens up far-reaching opportunities for developing innovations and improving operational and business processes, for both the public and the private sector. Notwithstanding the awareness of those opportunities and increasing research on the potential benefits, the level of knowledge regarding how best to exploit and leverage economic value remains in many respects at incomplete (Magalhaes & Roseira, 2020 ; Ruijer & Meijer, 2020 ; Zuiderwijk et al., 2014 ). In particular in this context, but also in any of the other key topics, the research avenues identified indicate that OGD research may greatly benefit from the so far underrepresented IS and digital business perspective. As such it may serve as an important tool to build the bridge from OGD to IS and digital business research.

Overall, the research agenda synthesizes the answers to our second research question of how IS and digital business research can inform OGD research, in particular with regard to its role in the digital economy. The theoretical contribution of our study in terms of utility stems especially from the systematization of the complex and heterogeneous research landscape of OGD, as well as the theory-informed research agenda. The latter makes the field more accessible and tangible for IS and digital business research by showing what issues may be studied and how they are related.

However, our study is not without limitations. Merging information obtained from research databases bears a certain risk associated with information technology limitations and time delays that may prevent the full scope of relevant studies from being represented. In addition, our final sample is limited to studies in English language, which means that we may have missed potentially relevant studies in other languages. Bearing in mind that a complete selection is hardly feasible in terms of practicality and that the literature work on which this study is based was generated with respect to well-established methodological guidelines (Rowe, 2014 ; Webster & Watson, 2002 ), we are nevertheless convinced of the sufficient coverage and informative value provided by our relevant set. In addition, our analysis is limited to empirical studies and does not take account of conceptual approaches. Future research could examine whether the review framework also hold true in this connection and how empirical and conceptual OGD research differ in their distribution across the different key topics .

While our systematization and analyses enhance the level of lucidity and understanding with regard to the overall context of OGD, it should be noted that the six identified key topics require further dedicated attention in order to thoroughly interpret and understand the insights of the respective subareas. In this connection, it should further be noted that some of these topics have also been discussed in related research areas, in particular the more general field of open data, which have not been part of our literature review. Future research could synthesize these research streams and examine how they complement our findings. Finally, our comprehensive approach inherently goes at the expense of a detailed examination and discussion of each key topic. Although the majority of literature reviews on OGD have focused on a special key topic, it remains an important task for future studies to scrutinize recent, widely unexplored subtopics in OGD research, such as innovation and value creation.

In conclusion, although OGD has accumulated a substantial body of knowledge over the last decade, the field is still in an emerging stage and calls for further research to provide answers to a variety of important unresolved issues from an IS perspective. This systematic literature review contributes to a comprehensive understanding of OGD and may serve as a suitable reference point and impetus in bridging the gap between OGD and IS research and exploiting the potential of OGD for the digital economy.

(DOCX 39 kb)

Open Access funding enabled and organized by Projekt DEAL.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Bernd W. Wirtz, Email: ed.reyeps-inu@ztriw-sl .

Jan C. Weyerer, Email: ed.reyeps-inu@rereyew .

Marcel Becker, Email: ed.reyeps-inu@rekcebm .

Wilhelm M. Müller, Email: ed.reyeps-inu@relleumw .

• Afful-Dadzie E, Afful-Dadzie A. Open government data in Africa: A preference elicitation analysis of media practitioners. Government Information Quarterly. 2017; 34 (2):244–255. doi: 10.1016/j.giq.2017.02.005. [ CrossRef ] [ Google Scholar ]
• Alexopoulos C, Loukis E, Mouzakitis S, Petychakis M, Charalabidis Y. Analysing the characteristics of open government data sources in Greece. Journal of the Knowldege Economy. 2018; 9 (3):721–753. doi: 10.1007/s13132-015-0298-8. [ CrossRef ] [ Google Scholar ]
• Altayar, M. S. (2018). Motivations for open data adoption: An institutional theory perspective. Government Information Quarterly, 35 (4), 633–643. 10.1016/j.giq.2018.09.006
• Attard J, Orlandi F, Scerri S, Auer S. A systematic review of open government data initiatives. Government Information Quarterly. 2015; 32 (4):399–418. doi: 10.1016/j.giq.2015.07.006. [ CrossRef ] [ Google Scholar ]
• Bachtiar, A., Suhardi, & Muhamad, M. (2020). Literature review of open government data. 2020 International Conference on Information Technology Systems and Innovation (ICITSI) (pp. 329–334).
• Barry E, Bannister F. Barriers to open data release: A view from the top. Information Polity: The International Journal of Government & Democracy in the Information Age. 2014; 19 (1/2):129–152. doi: 10.3233/IP-140327. [ CrossRef ] [ Google Scholar ]
• Berrone, P., Ricart, J., & Carrasco-Farré, C. (2017). The open kimono: Toward a general framework for open data initiatives in cities. California Management Review 59 (1), 39–70. 10.1177/0008125616683703
• Bogdanović-Dinić S, Veljković N, Stoimenov L. How open are public government data? An assessment of seven open data portals. In: Rodríguez-Bolívar MP, editor. Public administration and information technology. Measuring E-government efficiency. Springer New York; 2014. pp. 25–44. [ Google Scholar ]
• Bonina, C., & Eaton, B. (2020). Cultivating open government data platform ecosystems through governance: Lessons from Buenos Aires, Mexico City and Montevideo. Government Information Quarterly, 37 (3). 10.1016/j.giq.2020.101479
• Borgesius FZ, Gray J, van Eechoud M. Open data, privacy, and fair information principles: Towards a balancing framework. Berkeley Technology Law Journal. 2015; 30 (3):2073–2131. doi: 10.15779/Z389S18. [ CrossRef ] [ Google Scholar ]
• Callaway E, Ledford H, Viglione G, Watson T, Witze A. COVID and 2020: An extraordinary year for science. Nature. 2020; 588 (7839):550–552. doi: 10.1038/d41586-020-03437-4. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chatfield, A., & Reddick, C. (2017). A longitudinal cross-sector analysis of open data portal service capability: The case of Australian local governments. Government Information Quarterly, 34 (2), 231–243.
• Chorley, K. M. (2017). The challenges presented to records management by open government data in the public sector in england. Records Management Journal, 27 (2), 149–158. 10.1108/RMJ-09-2016-0034
• Criado JI, Ruvalcaba-Gómez EA, Valenzuela-Mendoza R. Revisiting the open government phenomenon. A Meta-analysis of the international literature. JeDEM - eJournal of eDemocracy and Open Government. 2018; 10 (1):50–81. doi: 10.29379/jedem.v10i1.454. [ CrossRef ] [ Google Scholar ]
• Crusoe J, Melin U. Investigating Open Government Data Barriers. In: Parycek P, Glassey O, Janssen M, Scholl HJ, Tambouris E, Kalampokis E, Virkar S, editors. Lecture notes in computer science. Electronic government. Springer International Publishing; 2018. pp. 169–183. [ Google Scholar ]
• Dawes SS, Vidiasova L, Parkhimovich O. Planning and designing open government data programs: An ecosystem approach. Government Information Quarterly. 2016; 33 (1):15–27. doi: 10.1016/j.giq.2016.01.003. [ CrossRef ] [ Google Scholar ]
• De Kool, D. & Bekkers, V. J. J. M. (2014). The perceived impact of open inspection data on the quality of education in Dutch primary schools: A parent perspective. Social Science Computer Review . 10.1177/0894439314560853
• Fan B, Zhao Y. The moderating effect of external pressure on the relationship between internal organizational factors and the quality of open government data. Government Information Quarterly. 2017; 34 (3):396–405. doi: 10.1016/j.giq.2017.08.006. [ CrossRef ] [ Google Scholar ]
• Francey, A., & Mettler, T. (2021). The effects of open government data: Some stylised facts. Information Polity: The International Journal of Government & Democracy in the Information Age, 1–16. 10.3233/ip-200281.
• Gascó-Hernández, M., Martin, E. G., Reggi, L., Pyo, S., & Luna-Reyes, L. F. (2018). Promoting the use of open government data: Cases of training and engagement. Government Information Quarterly, 35 (2), 233–242. 10.1016/j.giq.2018.01.003
• González-Zapata, F. & Heeks, R. (2015). The multiple meanings of open government data: Understanding different stakeholders and their perspectives. Government Information Quarterly, 32 (4), 441–452. 10.1016/j.giq.2015.09.001
• Gonzálvez-Gallego, N., Nieto-Torrejón, L., & Pérez-Cárceles, M. (2020). Is open data an enabler for trust? Exploring the link and the mediating role of citizen satisfaction. International Journal of Public Administration, 43 (14), 1218–1227. 10.1080/01900692.2019.1668412
• Gusenbauer M. Google scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics. 2019; 118 (1):177–214. doi: 10.1007/s11192-018-2958-5. [ CrossRef ] [ Google Scholar ]
• Haini SI, AB R, Mohd NZ, Zainuddin NM, Ibrahim R. Factors influencing the adoption of open government data in the public sector: A systematic literature review. International Journal on Advanced Science, Engineering and Information Technology. 2020; 10 (2):611. doi: 10.18517/ijaseit.10.2.9488. [ CrossRef ] [ Google Scholar ]
• Hassan, M. I. A., & Twinomurinzi, H. (2018). A systematic literature review of open government data research: Challenges, opportunities and gaps. 2018 Open Innovations Conference (OI) (pp. 299–304).
• Hermanto, A., Solimun, S., Rinaldo Fernandes, A. A., Wahyono, W., & Zulkarnain, Z. (2018). The importance of open government data for the private sector and ngos in indonesia. Digital Policy, Regulation and Governance, 20 (4), 293–309. https://search.ebscohost.com/login.aspx?direct=true&db=eoh&AN=1724283&site=ehost-live
• Hossain MA, Dwivedi YK, Rana NP. State-of-the-art in open data research: Insights from existing literature and a research agenda. Journal of Organizational Computing & Electronic Commerce. 2016; 26 (1/2):14–40. doi: 10.1080/10919392.2015.1124007. [ CrossRef ] [ Google Scholar ]
• Janssen M, Charalabidis Y, Zuiderwijk A. Benefits, adoption barriers and myths of open data and open government. Information Systems Management. 2012; 29 (4):258–268. doi: 10.1080/10580530.2012.716740. [ CrossRef ] [ Google Scholar ]
• Jetzek, T., Avital, M., & Bjørn-Andersen, N. (2013). Generating value from open government data. In R. Baskerville, & M. Chau (Eds.), Proceedings of the 34th International Conference on Information Systems (ICIS 2013) . http://aisel.aisnet.org/cgi/viewcontent.cgi?article=1181&context=icis2013
• Jetzek, T., Avital, M., & Bjørn-Andersen, N. (2014). Data-driven innovation through open government data. Journal of Theoretical and Applied Electronic Commerce Research, 9 (2), 100–120. 10.4067/S0718-18762014000200008
• Jetzek T, Avital M, Bjørn-Andersen N. The sustainable value of open government data. Journal of the Association for Information Systems. 2019; 20 (6):702–734. doi: 10.17705/1jais.00549. [ CrossRef ] [ Google Scholar ]
• Karkin N, Yavuz N. An inquiry for local open government data policy: Is a proactive model of open government data portals possible in Turkey? Current Politics and Economics of the Middle East. 2017; 8 (2):129–150. [ Google Scholar ]
• Kassen M. A promising phenomenon of open data: A case study of the Chicago open data project. Government Information Quarterly. 2013; 30 (4):508–513. doi: 10.1016/j.giq.2013.05.012. [ CrossRef ] [ Google Scholar ]
• Kessler EH, Chakrabarti AK. Innovation speed: A conceptual model of context, antecedents, and outcomes. Academy of Management Review. 1996; 21 (4):1143–1191. doi: 10.5465/amr.1996.9704071866. [ CrossRef ] [ Google Scholar ]
• Khurshid MM, Zakaria NH, Rashid A, Kazmi R, Shafique MN, Nazir Ahmad M. Analyzing diffusion patterns of big open data as policy innovation in public sector. Computers & Electrical Engineering. 2019; 78 :148–161. doi: 10.1016/j.compeleceng.2019.07.010. [ CrossRef ] [ Google Scholar ]
• Kim H. Interlinking open government data in Korea using Administrative District knowledge graph. Journal of Information Science Theory & Practice (JIStaP) 2018; 6 (1):18–30. doi: 10.1633/JISTaP.2018.6.1.2. [ CrossRef ] [ Google Scholar ]
• Kleiman F, Meijer S, Janssen M. Behavioral factors influencing the opening of government data by civil servants. In : ICEGOV 2020, Proceedings of the 13th International Conference on Theory and Practice of Electronic Governance (pp. 529–534) Association for Computing Machinery; 2020. [ Google Scholar ]
• Kurtz LP, Santos PM, Rover AJ, Zuiderwijk A, Hinnant CC. Open data via websites of Brazilian superior courts of justice: Changes between 2013 and 2017. Information Polity: The International Journal of Government & Democracy in the Information Age. 2019; 24 (2):147–162. doi: 10.3233/IP-180104. [ CrossRef ] [ Google Scholar ]
• Lane PJ, Koka BR, Pathak S. The reification of absorptive capacity: A critical review and rejuvenation of the construct. Academy of Management Review. 2006; 31 (4):833–863. doi: 10.5465/amr.2006.22527456. [ CrossRef ] [ Google Scholar ]
• Lee SY, Díaz-Puente JM, Martin S. The contribution of open government to prosperity of society. International Journal of Public Administration. 2019; 42 (2):144–157. doi: 10.1080/01900692.2017.1405446. [ CrossRef ] [ Google Scholar ]
• Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Medicine. 2009; 6 (7):e1000100. doi: 10.1371/journal.pmed.1000100. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lim, T. C. (2021). Patterns in environmental priorities revealed through government open data portals. Telematics and Informatics, 101678 . 10.1016/j.tele.2021.101678
• Lim WM, Yap S-F, Makkar M. Home sharing in marketing and tourism at a tipping point: What do we know, how do we know, and where should we be heading? Journal of Business Research. 2021; 122 :534–566. doi: 10.1016/j.jbusres.2020.08.051. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Luna-Reyes LF, Najafabadi MM, Zuiderwijk A, Hinnant CC. The US open data initiative: The road ahead. Information Polity: The International Journal of Government & Democracy in the Information Age. 2019; 24 (2):163–182. doi: 10.3233/IP-180106. [ CrossRef ] [ Google Scholar ]
• Máchová R, Lněnička M. Evaluating the quality of open data portals on the National Level. Journal of Theoretical & Applied Electronic Commerce Research. 2017; 12 (1):21–41. doi: 10.4067/S0718-18762017000100003. [ CrossRef ] [ Google Scholar ]
• Magalhaes, G., & Roseira, C. (2020). Open government data and the private sector: An empirical view on business models and value creation. Government Information Quarterly, 37 (3). 10.1016/j.giq.2017.08.004
• Marjanovic, O. & Cecez-Kecmanovic, D. (2017). Exploring the tension between transparency and datification effects of open government IS through the lens of Complex Adaptive Systems. The Journal of Strategic Information Systems, 26 (3), 210–232. 10.1016/j.jsis.2017.07.001
• Marmier A, Mettler T. Developing an index for measuring OGD publisher compliance to good practice standards: Insights from opendata.Swiss. Information Polity: The International Journal of Government & Democracy in the Information Age. 2020; 25 (1):91–110. doi: 10.3233/IP-180120. [ CrossRef ] [ Google Scholar ]
• Martín-Martín A, Orduna-Malea E, Harzing A-W, Delgado López-Cózar E. Can we use Google scholar to identify highly-cited documents? Journal of Informetrics. 2017; 11 (1):152–163. doi: 10.1016/j.joi.2016.11.008. [ CrossRef ] [ Google Scholar ]
• Martín-Martín, A., Thelwall, M., Orduna-Malea, E., & Delgado López-Cózar, E. (2020). Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations' COCI: A multidisciplinary comparison of coverage via citations. Scientometrics, 1–36. 10.1007/s11192-020-03690-4. [ PMC free article ] [ PubMed ]
• McBride K, Aavik G, Toots M, Kalvet T, Krimmer R. How does open government data driven co-creation occur? Six factors and a 'perfect storm'; insights from Chicago's food inspection forecasting model. Government Information Quarterly. 2019; 36 (1):88–97. doi: 10.1016/j.giq.2018.11.006. [ CrossRef ] [ Google Scholar ]
• McNutt, J., Justice, J., Melitski, J., Ahn, M., Siddiqui, S., Carter, D., & Kline, A. (2016). The diffusion of civic technology and open government in the United States. Information Polity, 21 (2), 153–170. 10.3233/IP-160385
• Meijer, A. (2015). Government transparency in historical perspective: From the ancient regime to open data in The Netherlands. International Journal of Public Administration, 38 (3), 189–199. 10.1080/01900692.2014.934837
• Meijer A, Potjer S. Citizen-generated open data: An explorative analysis of 25 cases. Government Information Quarterly. 2018; 35 (4):613–621. doi: 10.1016/j.giq.2018.10.004. [ CrossRef ] [ Google Scholar ]
• Mergel I, Kleibrink A, Sörvik J. Open data outcomes: U.S. cities between product and process innovation. Government Information Quarterly. 2018; 35 (4):622–632. doi: 10.1016/j.giq.2018.09.004. [ CrossRef ] [ Google Scholar ]
• Neto AJA, Neves DF, Santos LC, Junior MCR, do Nascimento, Rogério P. C. Open government data usage overview: A systematic literature mapping. In : EATIS ‘18, Proceedings of the Euro American Conference on Telematics and Information Systems . Association for Computing Machinery; 2018. [ Google Scholar ]
• Novais, T., Albuquerque, J. P. de, & Craveiro, G. S. (2013). An account of research on open government data (2007–2012): A systematic literature review. In Scholl H., Wimmer M.A., Tambouris E., Janssen M., & Macintosh A. (Eds.) (pp. 76–83). Gesellschaft fur Informatik (GI) . Retrieved from https://www.scopus.com/inward/record.uri?eid=2-s2.0-84918551984&partnerID=40&md5=8b722dd21af1f1c3743202eaeaae389d
• Open Knowledge Foundation. (2021). Open definition: Defining open in open data, open content and open knowledge . Retrieved from http://opendefinition.org/
• Parycek, P., Höchtl, J., & Ginner, M. (2014). Open Government Data Implementation Evaluation. Journal of theoretical and applied electronic commerce research, 9 (2), 80–99. 10.4067/S0718-18762014000200007
• Paul J, Benito GRG. A review of research on outward foreign direct investment from emerging countries, including China: What do we know, how do we know and where should we be heading? Asia Pacific Business Review. 2018; 24 (1):90–115. doi: 10.1080/13602381.2017.1357316. [ CrossRef ] [ Google Scholar ]
• Paul J, Criado AR. The art of writing literature review: What do we know and what do we need to know? International Business Review. 2020; 29 (4):101717. doi: 10.1016/j.ibusrev.2020.101717. [ CrossRef ] [ Google Scholar ]
• Piotrowski SJ. The “open government reform” movement. American Review of Public Administration. 2017; 47 (2):155–171. doi: 10.1177/0275074016676575. [ CrossRef ] [ Google Scholar ]
• Purwanto A, Zuiderwijk A, Janssen M. Citizen Engagement With Open Government Data. International Journal of Electronic Government Research. 2020; 16 (3):1–25. doi: 10.4018/IJEGR.2020070101. [ CrossRef ] [ Google Scholar ]
• Raisch S, Birkinshaw J. Organizational ambidexterity: Antecedents, outcomes, and moderators. Journal of Management. 2008; 34 (3):375–409. doi: 10.1177/0149206308316058. [ CrossRef ] [ Google Scholar ]
• Roa, H. N., Loza-Aguirre, E., & Flores, P. (2019). A survey on the problems affecting the development of open government data initiatives. In Teran L., Meier A., & Pincay J. (Eds.) (pp. 157–163). Institute of Electrical and Electronics Engineers Inc. Retrieved from https://www.scopus.com/inward/record.uri?eid=2-s2.0-85068367066&doi=10.1109%2fICEDEG.2019.8734452&partnerID=40&md5=edfd3858943a1342f976f8bc20cf71e9
• Rowe F. What literature review is not: Diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255. doi: 10.1057/ejis.2014.7. [ CrossRef ] [ Google Scholar ]
• Ruijer E, Grimmelikhuijsen S, Meijer A. Open data for democracy: Developing a theoretical framework for open data use. Government Information Quarterly. 2017; 34 (1):45–52. doi: 10.1016/j.giq.2017.01.001. [ CrossRef ] [ Google Scholar ]
• Ruijer E, Martinius E. Researching the democratic impact of open government data: A systematic literature review. Information Polity: The International Journal of Government & Democracy in the Information Age. 2017; 22 (4):233–250. doi: 10.3233/IP-170413. [ CrossRef ] [ Google Scholar ]
• Ruijer E, Meijer A. Open government data as an innovation process: Lessons from a living lab experiment. Public Performance & Management Review. 2020; 43 (3):613–635. doi: 10.1080/15309576.2019.1568884. [ CrossRef ] [ Google Scholar ]
• Safarov, I. (2019). Institutional dimensions of open government data implementation: Evidence from the Netherlands, Sweden, and the UK. Public Performance & Management Review, 42 (2), 305–328. 10.1080/15309576.2018.1438296
• Safarov I, Meijer A, Grimmelikhuijsen S. Utilization of open government data: A systematic literature review of types, conditions, effects and users. Information Polity: The International Journal of Government & Democracy in the Information Age. 2017; 22 (1):1–24. doi: 10.3233/IP-160012. [ CrossRef ] [ Google Scholar ]
• Saldaña J. The coding manual for qualitative researchers (2. ed.) SAGE Publ; 2013. [ Google Scholar ]
• Saxena S. Summarizing the decadal literature in open government data (OGD) research: A systematic review. Foresight. 2018; 20 (6):648–664. doi: 10.1108/FS-07-2018-0074. [ CrossRef ] [ Google Scholar ]
• Saxena S, Janssen M. Examining open government data (OGD) usage in India through UTAUT framework. Foresight. 2017; 19 (4):421–436. doi: 10.1108/FS-02-2017-0003. [ CrossRef ] [ Google Scholar ]
• Sayogo, D. S., Pardo, T. A., & Cook, M. (2014). A framework for benchmarking open government data efforts. 47th Hawaii International Conference on System Sciences (pp. 1896–1905). 10.1109/HICSS.2014.240
• Shao DD, Saxena S. Barriers to open government data (OGD) initiative in Tanzania: Stakeholders' perspectives. Growth and Change. 2019; 50 (1):470–485. doi: 10.1111/grow.12282. [ CrossRef ] [ Google Scholar ]
• Shepherd E, Bunn J, Flinn A, Lomas E, Sexton A, Brimble S, et al. Open government data: Critical information management perspectives. Records Management Journal. 2019; 29 (1):152–167. doi: 10.1108/RMJ-08-2018-0023. [ CrossRef ] [ Google Scholar ]
• Smith, G., & Sandberg, J. (2018). Barriers to innovating with open government data: Exploring experiences across service phases and user types. Information Polity: The International Journal of Government & Democracy in the Information Age, 23 (3), 249–265. 10.3233/IP-170045
• Susha, I., Grönlund, Å., & Janssen, M. (2015). Driving factors of service innovation using open government data: An exploratory study of entrepreneurs in two countries. Information Polity: The International Journal of Government & Democracy in the Information Age, 20 (1), 19–34. 10.3233/IP-150353
• Tai K-T. Open government research over a decade: A systematic review. Government Information Quarterly. 2021; 38 (2):101566. doi: 10.1016/j.giq.2021.101566. [ CrossRef ] [ Google Scholar ]
• Tranfield D, Denyer D, Smart P. Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management. 2003; 14 (3):207–222. doi: 10.1111/1467-8551.00375. [ CrossRef ] [ Google Scholar ]
• Ubaldi B. Open government data: Open government data: towards empirical analysis of open government data initiatives. OECD working papers on public governance (No. 22) Organisation for Economic Cooperation and Development (OECD); 2013. [ Google Scholar ]
• Urquhart C. Grounded theory for qualitative research: A practical guide. SAGE; 2013. [ Google Scholar ]
• Veljković N, Bogdanović-Dinić S, Stoimenov L. Benchmarking open government: An open data perspective. Government Information Quarterly. 2014; 31 (2):278–290. doi: 10.1016/j.giq.2013.10.011. [ CrossRef ] [ Google Scholar ]
• Wang, H.‑J., & Lo, J. (2016). Adoption of open government data among government agencies. Government Information Quarterly, 33 (1), 80–88. 10.1016/j.giq.2015.11.004
• Wang D, Chen C, Richards D. A prioritization-based analysis of local open government data portals: A case study of Chinese province-level governments. Government Information Quarterly. 2018; 35 (4):644–656. doi: 10.1016/j.giq.2018.10.006. [ CrossRef ] [ Google Scholar ]
• Wang Y, Han JH, Beynon-Davies P. Understanding blockchain technology for future supply chains: A systematic literature review and research agenda. Supply Chain Management: An International Journal. 2019; 24 (1):62–84. doi: 10.1108/SCM-03-2018-0148. [ CrossRef ] [ Google Scholar ]
• Wang, V., & Shepherd, D. (2020). Exploring the extent of openness of open government data – a critique of open government datasets in the UK. Government Information Quarterly, 37 (1). 10.1016/j.giq.2019.101405
• Watson RT, Webster J. Analysing the past to prepare for the future: Writing a literature review a roadmap for release 2.0. Journal of Decision Systems. 2020; 29 (3):129–147. doi: 10.1080/12460125.2020.1798591. [ CrossRef ] [ Google Scholar ]
• Webster J, Watson RT. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):xiii–xxiii. doi: 10.2307/4132319. [ CrossRef ] [ Google Scholar ]
• Welch, E., Feeney, M., & Park, C. H. (2016). Determinants of data sharing in U.S. city governments. Government Information Quarterly, 33 (3), 393–403. 10.1016/j.giq.2016.07.002
• Wirtz BW, Birkmeyer S. Open government: Origin, development, and conceptual perspectives. International Journal of Public Administration. 2015; 38 (5):381–396. doi: 10.1080/01900692.2014.942735. [ CrossRef ] [ Google Scholar ]
• Wirtz, B. W., Weyerer, J. C., & Rösch, M. (2018). Citizen and open government: An empirical analysis of antecedents of open government data. International Journal of Public Administration, 41 (4), 308–320. 10.1080/01900692.2016.1263659
• Wirtz BW, Weyerer JC, Rösch M. Open government and citizen participation: An empirical analysis of citizen expectancy towards open government data. International Review of Administrative Sciences. 2019; 85 (3):566–586. doi: 10.1177/0020852317719996. [ CrossRef ] [ Google Scholar ]
• World Wide Web Foundation. (2017). Open data barometer: Global report . Retrieved from https://opendatabarometer.org/doc/4thEdition/ODB-4thEdition-GlobalReport.pdf
• Worthy, B. (2015). The impact of open data in the UK: Complex, unpredictable, and political. Public Administration, 93 (3), 788–805. 10.1111/padm.12166
• Yang, T.-M., Lo, J., & Shiang, J. (2015). To open or not to open? Determinants of open government data. Journal of Information Science, 41 (5), 596–612. 10.1177/0165551515586715
• Yang, T. M., & Wu, Y. J. (2016). Examining the socio-technical determinants influencing government agencies’ open data publication: A study in Taiwan. Government Information Quarterly, 33 (3):378–392. 10.1016/j.giq.2016.05.003
• Young MM. Implementation of digital-era governance: The case of open data in U.S Cities. Public Administration Review. 2020; 80 (2):305–315. doi: 10.1111/puar.13156. [ CrossRef ] [ Google Scholar ]
• Zhao, Y., & Fan, B. (2018). Exploring open government data capacity of government agency: Based on the resource-based theory. Government Information Quarterly, 35 (1), 1–12. 10.1016/j.giq.2018.01.002
• Zhenbin Y, Kankanhalli A, Ha S, Tayi GK. What drives public agencies to participate in open government data initiatives? An innovation resource perspective. Information & Management. 2020; 57 (3):N.PAG-N.PAG. doi: 10.1016/j.im.2019.103179. [ CrossRef ] [ Google Scholar ]
• Zientek LR, Werner JM, Campuzano MV, Nimon K. The use of Google scholar for research and research dissemination. New Horizons in Adult Education and Human Resource Development. 2018; 30 (1):39–46. doi: 10.1002/nha3.20209. [ CrossRef ] [ Google Scholar ]
• Zuiderwijk A, Helbig N, Gil-Garcia J, Janssen M. Innovation through open data: A review of the state-of-the-art and an emerging research agenda. Journal of Theoretical & Applied Electronic Commerce Research. 2014; 9 (2):I–XIII. doi: 10.4067/S0718-18762014000200001. [ CrossRef ] [ Google Scholar ]
• Zuiderwijk, A., Janssen, M., & Dwivedi, Y. K. (2015). Acceptance and use predictors of open data technologies: Drawing upon the unified theory of acceptance and use of technology. Government Information Quarterly, 32 (4), 429–440. 10.1016/j.giq.2015.09.005
• Zuiderwijk A, Janssen M, van de Kaa G, Poulis K. The wicked problem of commercial value creation in open data ecosystems: Policy guidelines for governments. Information Polity: The International Journal of Government & Democracy in the Information Age. 2016; 21 (3):223–236. doi: 10.3233/IP-160391. [ CrossRef ] [ Google Scholar ]