systematic literature review in engineering

Last updated 20/06/24: Online ordering is currently unavailable due to technical issues. We apologise for any delays responding to customers while we resolve this. For further updates please visit our website: https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > Journals
• > Proceedings of the Design Society: International Conference on Engineering Design
• > Volume 1 Issue 1
• > Systematic Literature Reviews: An Introduction

Article contents

Systematic literature reviews: an introduction.

Published online by Cambridge University Press:  26 July 2019

Systematic literature reviews (SRs) are a way of synthesising scientific evidence to answer a particular research question in a way that is transparent and reproducible, while seeking to include all published evidence on the topic and appraising the quality of this evidence. SRs have become a major methodology in disciplines such as public policy research and health sciences. Some have advocated that design research should adopt the method. However, little guidance is available. This paper provides an overview of the SR method, based on the literature in health sciences. Then, the rationale for SRs in design research is explored, and four recent examples of SRs in design research are analysed to illustrate current practice. Foreseen challenges in taking forward the SR method in design research are highlighted, and directions for developing a SR method for design research are proposed. It is concluded that SRs hold potential for design research and could help us in addressing some important issues, but work is needed to define what review methods are appropriate for each type of research question in design research, and to adapt guidance to our own needs and specificities.

This article has been cited by the following publications. This list is generated based on data provided by Crossref .

• Google Scholar

View all Google Scholar citations for this article.

Save article to Kindle

To save this article to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

• Volume 1, Issue 1
• Guillaume Lame
• DOI: https://doi.org/10.1017/dsi.2019.169

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox .

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive .

Reply to: Submit a response

- No HTML tags allowed - Web page URLs will display as text only - Lines and paragraphs break automatically - Attachments, images or tables are not permitted

Your details

Your email address will be used in order to notify you when your comment has been reviewed by the moderator and in case the author(s) of the article or the moderator need to contact you directly.

You have entered the maximum number of contributors

Conflicting interests.

Please list any fees and grants from, employment by, consultancy for, shared ownership in or any close relationship with, at any time over the preceding 36 months, any organisation whose interests may be affected by the publication of the response. Please also list any non-financial associations or interests (personal, professional, political, institutional, religious or other) that a reasonable reader would want to know about in relation to the submitted work. This pertains to all the authors of the piece, their spouses or partners.

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Systematic reviews in Science, Engineering and I.T

Systematic or systematic-like, defining the research question, guidelines and reporting standards, where to search, where to find systematic reviews, useful tools, welcome to the systematic reviews in science, engineering and i.t guide                      .

This guide has information to support researchers conducting systematic reviews in the fields of Science, Engineering and I.T. If you are conducting a systematic review with a health focus, see the Systematic Reviews for Medicine and Health Sciences Library Guide.

While systematic reviews are traditionally undertaken in the area of Health and Medicine, there has been a rise in other disciplines conducting systematic and systematic-like reviews, using the systematic approach to summarize and assess existing evidence to identify gaps and help inform decisions.

There are many types of reviews , and it is important to assess whether your research is best suited to a systematic review or a systematic-like review. A systematic-like review could mean taking a more structured and organised approach to your review or ensuring that your review is transparent and reproducible. Depending on the level of your research, you can determine which type of review and which guidelines are best to follow.

The frameworks below can help you clearly define your research question. A clear and focused question will guide you through each stage of your review, from establishing your inclusion and exclusion criteria, developing your search strategy to composing your findings.

• PICO Patient/Population, Intervention, Control Group/Comparison, Outcome (Health-based)
• PECO Population, Exposure, Comparator, Outcomes (Environmental science)
• PCS Problem, Constraints, Systems (Computer Science)
• PIFTS Product/Process, Impact/Interest, Flow, and Type of lifecycle assessment (Lifecyle assessment)

Guidelines and standards help to ensure that a systematic review provides results that are valid, transparent, and reproducible. In Health and Medicine, the Cochrane Handbook is the gold standard for systematic reviews. The rise of systematic reviews in disciplines other than health has called for new guidelines to be developed. Some of these guidelines are listed below as well as articles by researchers in different fields who have identified the need for discipline specific systematic review guidelines, and which outline examples.

• CEE (Collaboration for Environmental Evidence) Guidelines and Standards for Evidence Synthesis in Environmental Management (Environmental sciences)
• PRISMA 2020 Checklist
• PRISMA for Ecology and Evolutionary Biology (Environmental sciences)
• ROSES: Reporting standards for systematic evidence syntheses (Environmental sciences)
• Systematic Quantitative Literature Review Method for Environmental Science (Griffith University) (Environmental sciences)

Articles containing example guidelines

• Guidelines for a systematic review in systems and automatic engineering
• Guidelines for performing systematic literature teviews in software engineering
• Guidelines for performing systematic reviews in the development of toxicity factors
• Guidelines for systematic review in conservation and environmental management

A protocol is a plan for your research which details every stage of your process. Your protocol should then be registered, so that other researchers are advised that someone is already conducting research on the topic. Refer to the guidelines and reporting standards relevant to your research, as they may contain information on how to develop a protocol.

Developing a protocol

• PRISMA for systematic review protocols

Searching and registering protocols

• Cadima (Environmental sciences)
• OSF registries
• Prospero (Health related outcomes)
• Research Registry

Browse the library database list to find relevant databases

Find the library guide for your subject area

Environmental Sciences

• CEE: Environmental Evidence Library of Evidence Syntheses

Other subject areas:

Search the library including the search term “systematic review” and your subject area to find reviews relevant to your area of study.

Search the library

These tools can assist with producing a Systematic Review.

• Covidence Systematic review management tool. Handles two person title/abstract screening, full text screening, risk of bias assessment and data extraction from included studies. All University of Melbourne staff and students have unlimited access to Covidence.
• Systematic review toolbox A community-driven, searchable, web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. Over 200 tools are included.
• Text Mining for Search Strategy Development library guide Our separate guide to all things related to using text mining for developing your systematic review search strategy. Over 10 tools covered.

Contact the Science and Engineering Library Team

You can contact the science and engineering library team directly:.

Book an online Zoom Research Consultation

Would you like help with expert searching?

Do you want assistance with managing your data or understanding how to maximise your research’s impact?

Is reference management your weak spot?

Librarians can help.

Researchers and graduate research students are eligible for a one-on-one  1-hour online Zoom consultation with one of our subject specialist librarians.

Book an online Zoom consultation

Ask a question on Library Chat

Visit us on social media.

• Last Updated: Apr 29, 2024 1:08 PM
• URL: https://unimelb.libguides.com/sysrev_STEM

Advertisement

A systematic literature review of requirements engineering education

• Original Article
• Open access
• Published: 19 May 2022
• Volume 28 , pages 145–175, ( 2023 )

Cite this article

You have full access to this open access article

• Marian Daun   ORCID: orcid.org/0000-0002-9156-9731 1 ,
• Alicia M. Grubb   ORCID: orcid.org/0000-0002-3552-3165 2 ,
• Viktoria Stenkova   ORCID: orcid.org/0000-0002-4936-1873 1 &
• Bastian Tenbergen   ORCID: orcid.org/0000-0002-0145-4800 3  

9731 Accesses

8 Citations

Explore all metrics

Requirements engineering (RE) has established itself as a core software engineering discipline. It is well acknowledged that good RE leads to higher quality software and considerably reduces the risk of failure or budget-overspending of software development projects. It is of vital importance to train future software engineers in RE and educate future requirements engineers to adequately manage requirements in various projects. To this date, there exists no central concept of what RE education shall comprise. To lay a foundation, we report on a systematic literature review of the field and provide a systematic map describing the current state of RE education. Doing so allows us to describe how the educational landscape has changed over the last decade. Results show that only a few established author collaborations exist and that RE education research is predominantly published in venues other than the top RE research venues (i.e., in venues other than the RE conference and journal). Key trends in RE instruction of the past decade include involvement of real or realistic stakeholders, teaching predominantly elicitation as an RE activity, and increasing student factors such as motivation or communication skills. Finally, we discuss open opportunities in RE education, such as training for security requirements and supply chain risk management, as well as developing a pedagogical foundation grounded in evidence of effective instructional approaches.

Similar content being viewed by others

Requirements Engineering Education: A Systematic Literature Review

Experience-Oriented Approaches for Teaching and Training Requirements Engineering: An Experience Report

Teaching requirements elicitation interviews: an empirical study of learning from mistakes

Avoid common mistakes on your manuscript.

1 Introduction

Requirements engineering (RE) is commonly accepted as the foundation of high-quality software [ 132 ]. Requirements engineering education (REE) must not only deal with teaching students how to specify formal and informal requirements but also how to elicit and negotiate requirements involving multiple sources—particularly human stakeholders. Thus, REE must make students aware of socio-technical challenges and teach human-related aspects, which poses significant challenges for REE in higher education.

Furthermore, students must be adequately prepared to take on industrial challenges [ 172 , 195 ], while incorporating the theoretical concepts underlying RE [ 30 ]. REE is at best an afterthought in many university software engineering curricula [ 179 ], focusing on lecture-style instruction with few or no realistic examples. In many cases [ 64 , 67 ], RE is not instructed in dedicated courses, but instructed as part of a generic software engineering course. The problem with this situation is twofold: on the one hand, graduates only gain a rudimentary understanding of the minimal RE knowledge required by accreditation standards [ 1 ], standard curricula [ 74 , 75 , 90 ], and bodies of knowledge [ 20 ]. On the other hand, the opportunity is lost to give students enough experience to pick the right RE tools for each development project. In consequence, it is left to the industry to adequately train their staff to be effective in RE.

We need a clear understanding of what to teach (i.e., learning objectives), as well as what educational approaches are the most effective, who the learners are, and what learning outcomes to strive for. Herein, we contribute a systematic literature review of the field of Requirements Engineering Education. Our work allows researchers to gain an overview of the current state of the art and provides educators with insights on how to teach which RE technique.

We consider three research goals:

Goal 1: develop a systematic map of the current state of REE research;

Goal 2: report on current practices and their learning outcomes; and

Goal 3: evaluate how REE has changed over the previous decade.

We further elaborate on these goals in Sect.  3 .

This paper is structured as follows. Section  2 discusses meta-studies on the field of REE. Section  3 details our research questions and methodology. In Sects.  4 – 6 , we discuss results pertaining to each of our research goals (goal 1–goal 3). Section  7 concludes this paper.

2 Background and related work

2.1 challenges in mastering requirements engineering.

Requirements Engineering (RE) is a socio-technical, iterative process to elicit, document, and manage the requirements of a system under development [ 56 ]. RE bridges the gap between human users, developers, and managers, i.e., between people with and without software engineering expertise. RE helps to understand what problem needs to be solved by a (software) system. In addition, it helps to discover who needs to be involved in the engineering process (i.e., stakeholders) and how the problem could be solved by exploring trade-offs and alternatives [ 186 ]. RE requires analysis of both the problem space (i.e., context analysis) and solution space (i.e., the intervention). This is accomplished through a variety of requirements discovery or requirements gathering techniques, including eliciting requirements by interviewing stakeholders or by analyzing existing systems, before documenting the requirements in the form of a specification.

For example, interview techniques alone demand careful selection, as stakeholders may respond differently depending on the mode of inquiry. Imagine a focus group for a new mobile app to allow children to self-monitor health symptoms. A focus group consisting of physicians and children might quickly arrive at decisions about the app’s medical goals, but neglect the children’s perspective because in this setting, the children themselves might be too intimidated to contribute. Documentation techniques require similar careful choice. Storyboards, personas, user interface mock-ups, and natural language requirements (constrained or not), are useful to communicate ideas quickly with a broad audience of non-technical stakeholders, but lack precision for safety-related applications. Formal methods are very precise; however, they require substantial technical expertise and are generally unfit for directly communicating design choices and alternative solutions to stakeholders.

Despite excellent work in the field, elicited and documented requirements artifacts are often incomplete, conflict with one another, and/or suffer from other inadequacies [ 55 , 120 ]. The quality of how the RE process is conducted immediately impacts the quality of the requirements, which in turn, impacts the quality of the system under development. The RE process must be iterative and perpetually monitored with regard to elicitation, documentation, and validation, as well as tracing [ 148 ] requirements from their “source” (e.g., stakeholders, but also laws and standards) to their “destination” (e.g., their refinement into more requirements, analysis results, or their implementation into code). These challenges motivate us to investigate the landscape of RE approaches as it relates to education and training.

Mastering Requirements Engineering is not only a monumental task for the learner, but also for the educator [ 39 ]. On one hand, the theory behind concepts, techniques, and ontologies is quite technical and demands a high amount of rote memorization [ 31 ]. On the other hand, most of the RE process is “learning by doing,” i.e., the learners must to experience it for themselves [ 65 ] before being able to appreciate (and with repeated exposure, eventually master) the RE process and develop a “feeling” when certain techniques are preferable over others. This dichotomy requires a carefully calibrated RE curriculum that balances theory instruction and process exposure.

2.2 Studies on the state of requirements engineering education

In a recent REFSQ conference keynote, Footnote 1 Martin Glinz provided a survey spanning the past several decades on RE Education literature. Indeed, over the past 20 years, a series of reports have been published into the state of the art of software engineering education that are more or less concerned with aspects of requirements engineering education. One of the earliest ones by Shaw [ 162 ] came at a time, where software engineering education was mostly done at the graduate level, aiming to prepare future PhD students. Shaw picked up the claim made in [ 183 ], where graduate and postgraduate software engineering education starts too late and should begin at the undergraduate level alongside traditional computer science education. To this end, Shaw identified “forces” impacting the software engineering industry and academia, and derived “aspirations” for higher education in software engineering to strive towards. Shaw took a wider view than RE education alone, and she aspires for software engineering education to include the need for novice software engineers to specialize into roles and sub-fields like requirements engineering, testing, and even safety assessment. Moreover, Shaw suggested that software engineering education takes an experience-based stance to allow the learner to put theory into practice and develop an intuition for the application of techniques.

By 2008 [ 105 ], software engineering curricula became relatively wide-spread at the undergraduate level, and with it came an increased focus on RE education. As pointed out by Regev et al. [ 146 ], undergraduate RE education was slow to address Shaw’s aspirations, due to discrepancies between typical project-based learning in higher education and actual industry experiences. According to Regev et al., academic classroom projects translate poorly to the industry because of their “sterile” nature, which inadequately reflect industrial practices. The authors attributed this discrepancy to the fact that academic projects must be narrowly scoped to be completed within one semester, by a few students who do not have prior knowledge of the application domain. Additionally, instructors must provide the same experiential opportunity regardless of student background and possible arising group conflicts. Regev et al.’s observations are consistent with views previously reported by a series of other authors, including [ 27 , 33 , 54 , 68 , 169 ], and later confirmed with an empirical study by Menon et al. [ 112 ].

Three systematic mapping studies were conducted between 2012 and 2020, which consist of the work by Malik and Zafar [ 105 ], the aforementioned work by Idri, Ouhbi, et al. [ 71 , 135 ], and the work by Cico et al. [ 24 ]. While the mapping studies by Malik and Zafar as well as by Cico et al. take a wide aim on software engineering education at large, the work by Idri, Ouhbi et al, focuses particularly on RE education. Interestingly, Malik and Zafar report that while some of the mapped primary studies are concerned with project-based learning, the vast majority are concerned with educational technology and tools. Moreover, none of the 70 studies mapped by Malik and Zafar could be easily classified into the knowledge area “Requirements Engineering” according to the reference curricula available then (i.e., “Knowledge Area A” in [ 2 ] or “Knowledge Area C” in [ 90 ]). This indicates that REE research was incongruent with reference curricula and software engineering education research largely ignored RE as a topic. The more focused mapping study conducted by Ouhbi et al. [ 71 , 135 ] reveals a similar trend: only 19 out of 79 mapped primary studies mention reference curricula. The vast majority of papers (77%, see [ 135 ]) present solution approaches with mostly graduate or undergraduate students, with only a minority describing some evaluation of existing approaches. Only few primary studies concerned with industrial training or industrial case studies were found (i.e., 16% and 6% or selected studies, respectively). Moreover, while Ouhbi et al. found that 16% of selected primary studies were written with industrial training consultants as co-authors, neither  [ 71 ] nor [ 135 ] report on industry-readiness of learners.

In summary, past studies investigating the state of the art of REE have been conducted and published in loose intervals. As the newest REE-specific study conducted by Ouhbi et al. [ 71 , 135 ] stems from 2012, we expect the field to have evolved in light of the strong evolution of the field driven by new technologies (cf. [ 193 ]). Therefore, in this paper we want to provide an up-to-date investigation of the current state of REE. We investigate how the field has changed since the investigation of Ouhbi et al., and whether needs posed by new technologies have already been considered in REE research. In addition, we derive common practices and provide guidelines for REE synthesized from the found studies, which has not been done so far. Thus, we review educational approaches that foster learning objectives suitable to the requirements-related problem to be instructed.

3 Research method

In this section, we first elaborate on our research goals introduced in Sect.  1 , and introduce the research questions explored in our systematic literature review (SLR). We then describe our SLR methodology in detail, including how we searched for relevant papers, extracted knowledge, and analyzed data.

3.1 Goals and research questions

As mentioned in Sect.  1 , this SLR complements the mapping study by Ouhbi et al. [ 135 ]. Ouhbi et al.’s work mainly investigated the type of contribution, without placing a clear focus on learning outcomes. We, therefore, provide an overview of existing research about the state of REE and its impact on students’ learning outcomes with the study at hand.

We define three overall goals for our study:

Goal 1: Provide a systematic map of the current state of research in requirements engineering education. Such a systematic map helps researchers in relating their own research to the state of the art and educators in selecting existing approaches for application and adaptation to their own needs.

Goal 2: Provide a synthesis of the current practices the studies reported in the systematic map (i.e., goal 1). This helps to identify approaches best suited for specific RE learning outcomes and challenges in teaching RE.

Goal 3: Evaluate how the state of REE has changed over the last decade since the investigation by Ouhbi et al. [ 135 ].

To fulfill these goals, we defined ten detailed research questions, that allow us to assess the state of REE research. The research questions are listed in Table  1 .

To achieve goal 1, our SLR contains a systematic map that adheres to established research questions for systematic maps as defined by Petersen et al. [ 142 ]. These research questions have been adapted to account for research on REE. As commonly done in systematic mapping studies, we are interested in the researchers involved in REE (RQ1-2), the major publications and venues in the area (RQ3-5), and how do authors conduct and describe their research in this area (RQ6-8). In addition, we defined research questions regarding the educational approaches used, learning outcomes addressed, and the RE techniques taught (RQ9-10).

To achieve goal 2, we relate answers from goal 1 with one another. This allows investigating the instructional theories underlying REE, with a focus on learning outcomes. Taking this as a starting point we synthesize the findings, contributions, benefits, and shortcomings of the papers in the so created sets.

To achieve goal 3, we defined the research questions to be investigated on the basis of the research questions used by Ouhbi et al. [ 135 ]. This allows us a fair comparison of our findings—particularly newer publications—with the findings of Ouhbi et al.. Thereby, it can be investigated whether the state of REE research has changed over the last decade.

3.2 Search procedure

The selected search method of an SLR may impact the found results considerably: manual search, database search, and snowball search may result in paper sets with significant disparities [ 21 ]. In order to avoid limiting the scope of investigation to selected venues (like in manual search), or getting “stuck” in local cliques of mutually referencing papers (like in snowball searches), we used a database search to cover the overall spectrum of possible approaches.

In this spirit, we also used broad search terms to lower the risk of missing relevant papers. Our defined search string is as follows:

TITLE-ABS-KEY (“Requirements Engineering” AND “Education”)

For database searches it is common to include synonyms in the search string; however, this was not appropriate in the case of our investigation. We excluded “training” and “learning” from the search string as pilot testing the search string yielded an extra-proportional number of machine learning and artificial intelligence approaches being included in the results, which are beyond the scope of this study. We restrained the string from including the different areas of RE as substitute for the term “requirements engineering.” Doing so would have led to a misrepresentation of the field as many techniques relevant for requirements engineering education are used in other fields. Instead, we wanted to represent what authors believe is requirements engineering education. Thus, we restricted the search to requirements engineering education literature.

In addition, we analyzed the search string using comparison by manual search for selected venues (as suggested as part of the quasi-gold standard, cf. [ 197 ]). Analysis showed high sensitivity of the search string.

We used Scopus for the search because it covers many publishers, including the most common publishers for computer science research (e.g., ACM, IEEE, Elsevier, Springer), and unlike Google Scholar allows filtering non-peer-reviewed publications. The search string was developed based on the literature review’s topic and research questions, as is commonly done in systematic literature reviews [ 89 , 142 ].

3.3 Study selection

The search was conducted by three different researchers who evaluated each paper based on the inclusion and exclusion criteria (see Table  2 ) on their own. We considered papers published at any time until December 31, 2020. Papers were included in the set of relevant papers of the respective literature review if all researchers found the paper relevant and excluded if all found the paper irrelevant. In cases of inconsistent perceptions of the paper’s relevance, the paper was discussed among the researchers until consensus was reached.

Study selection process

Data extraction process

Figure  1 shows the process of step-wise exclusion of studies to derive the final set of included studies. The studies were selected and excluded at different stages.

Automated search using the search string resulted in 671 publications to be considered.

In the first round of exclusion, 391 papers were excluded by Researcher A as they were very obviously of no relevance to the field, were non-peer-reviewed publications, not in English, or for other obvious violations of the exclusion criteria. This left 280 inclusion candidates.

In a second round, two other researchers applied the inclusion and exclusion criteria on the remaining 280 papers individually. The separate application of inclusion and exclusion criteria was chosen to improve the quality of the paper selection process. In cases of differences agreement was reached in later discussions. In this step, Researcher B included 55.2% and excluded 44.8% of the inclusion candidates, while Researcher C included 45.1% and excluded 54.9% of the inclusion candidates.

In this step, the difference in inclusions and exclusions between Researcher B and C were investigated in detail. We identified 105 common inclusions and 59 common papers where both researchers agreed on exclusion. This yielded an inter-rater agreement of 76.92%, \(k=0.5217\) (which is “fair agreement”). As this big difference was surprising, the situation was discussed between all researchers. It was noticed that most differences resulted from papers that were not about RE education in the first place but discussed RE education in the context of SE education or in more general curricula. A close investigation of these papers yielded the understanding that the papers did not provide sufficient detail on the particular aspects of RE education to be included in the study. In total, we excluded 43 REE-related papers that met this criterion. Consequently, after Step 4, 493 papers from the original 671 were excluded, yielding 178 papers as inclusion candidates. Of these, 36 papers (i.e., about 5.4% of the original 671 papers) remained undecided for a last step of conflict resolution.

In the final step, Researcher A investigated the undecided 36 papers, proposed a solution for inclusion and exclusion, and the final decision was reached by discussion among all three researchers. From the remaining 36 undecided papers, ten were included and 26 were excluded.

In summary, we investigated 671 papers, from which we excluded 519 papers. Resulting in the final set of 152 included publications. Footnote 2

3.4 Data extraction

The data extraction process is illustrated in Fig.  2 . To answer research questions RQ1-6, we extracted each paper’s meta-data from Scopus. For RQ7-9, each included paper was read carefully by three different researchers to extract data pertinent to the research questions. For RQ10, we grouped selected studies into common themes for synthesis. We used word-tags pertaining to the content of a study (e.g., “industry-centric,” “motivation,” or “completeness”) and discussed our findings. Where there was disagreement between any two researchers, a third researcher evaluated the paper. The final classification was reached through discussions among all three researchers.

3.5 Quality assessment

Recently, some SLRs assess the quality of included studies (e.g., [ 135 ]), but these assessments lack a common standard. For example, the application of qualitative quality assessment criteria may be seen as difficult and ambiguous particularly when conducted by researchers of diverse backgrounds (e.g., [ 97 ]), and may, therefore, result in the erroneous exclusion of study results from synthesis. In addition, as is the case with our search, SLRs do not require primary data (i.e., papers) to have been published with sufficient transparency and quality for application of further empirical methods. Thus, we follow a commonly suggested quantitative approach to quality assessment by only including publications that have been peer-reviewed. Hence, we elected to have the quality assessment criteria be reflected in the inclusion and exclusion criteria outlined in Table  2 , instead of conducting an additional subjective quality assessment.

3.6 Analysis and classification

In this section, we revisit our research questions RQ1-10 and describe how we applied the classification schemas. Table  3 presents an overview of this information.

For RQ6, we used a commonly accepted classification for research methods provided by Wieringa et al. [ 194 ]. In doing so, we distinguished between evaluation research, proposal of a solution, validation research, philosophical papers, opinion papers, and personal experience papers (see Table  4 ). Each paper was mapped to exactly one category. In some cases, the categorization of papers might not be obvious. For example, it can be difficult to distinguish between a proposal of a solution and validation research because these types differ in terms of completeness and rigor of their evaluation, which may not be fully described. In these cases, the classification was based only on the presentation of the paper. Other evaluation activities that were suggested but not explicitly reported in the paper were not considered. Each paper was then assigned to the category that fit best.

For RQ7, we adapted the scheme proposed by Petersen et al. [ 140 ], which has been reused in other mapping studies (e.g., [ 36 , 52 ]). However, as some papers did not fit well into any of the original categories, we added a category for other contributions. Table  5 lists each contribution type. Each paper was assigned to all categories that apply.

3.7 Validity evaluation

In this section, we discuss aspects of validity according to the classification scheme in [ 141 ], and the measures taken to mitigate these potential threats.

3.7.1 Descriptive validity

Descriptive validity deals with the accurateness and objectivity of an investigation. As threats to descriptive validity are considered more significant in qualitative investigations than in quantitative investigations, we assume that there are no major threats to descriptive validity. We did not use qualitative quality assessment but favored quantitative quality assessment, which supports descriptive validity. Misclassification of papers may have led to threats to descriptive validity for RQ10 in particular. We built our classification to a large extent on existing and accepted classification schemes. We classify papers as intended by the authors (e.g., type of research contribution, educational approach used), which have been substantiated in the peer review process, to avoid threats from misinterpretation. It cannot be completely ruled out that authors and reviewers of one paper might have accepted an erroneous classification. We assume this was rare enough in occurrence to not impact the descriptive validity (i.e., without misrepresenting the field).

3.7.2 Theoretical validity

Theoretical validity concerns whether the research questions can be answered with the study setup. A major threat in this category typically stems from selection bias. To avoid this bias, we defined objective inclusion and exclusion criteria and applied them rigorously. Inclusion and exclusion criteria were applied independently by two different researchers, with a third researcher validating the choices. Also, the classification was done by two researchers independently, again with a third conducting quality assurance. In case conflicts in the inclusion/exclusion or classification of a paper arose between any two researchers, another researcher was involved, and the conflict was solved by discussion among all researchers, switching roles between “classifier” and “validator” in order to help each individual maintain an objective point of view.

3.7.3 Generalizability

Generalizability of the findings deals with the question, whether the set of papers included into the systematic mapping study are representative and do not miss important aspects. Comparison with previous secondary studies on requirements engineering education (see also Sect.  3.8 ) indicates that we did not miss a considerable number of relevant primary studies to be included.

3.7.4 Interpretive validity

Interpretive validity is concerned with the validity of the conclusions drawn. Hence, researcher biases are a considerable threat. To avoid threats to interpretive validity inclusion and exclusion criteria as well as the classification scheme were not applied by one researcher alone. As outlined above, conflicts were resolved by discussion among at least three researchers that investigated the paper independently. This reduces the threat of researcher bias.

3.7.5 Repeatability

To ensure repeatability, we report the search and selection process as well as the inclusion and exclusion criteria in sufficient detail to enable other researchers to verify our work. Moreover, we make our data available online, particularly with regard to RQ10. Additionally, abstaining from applying qualitative exclusion criteria helps improve repeatability. However, it cannot be ruled out that different researchers might have classified some of the papers in some cases into different categories. This is a common threat in systematic mapping studies and systematic literature reviews. Yet, due to the large number of included publications, we are confident that this would not alter the implications of our findings.

3.8 Validity evaluation for goal 3

Regarding goal 3, it is important to ensure that we use common grounds with the study of Ouhbi et al. [ 135 ], since this study serves as a baseline for our comparison of how REE has changed over the past decade.

We identified 36 of the 79 studies selected by Ouhbi et al.. Two of these studies were considered the same contribution in our work (i.e., [ 84 ]) because the two papers were published in the same venue very close to one another. Of the 43 remaining studies reported by Ouhbi et al., we identified 32 that either did not meet our inclusion criteria (e.g., studies with a primary focus on RE education, rather than education at large using RE methods), or meet our exclusion criteria (most commonly studies that are less than four pages long or dealing with RE for engineering education, see Table  2 ). One study was unobtainable to us, but reported in Ouhbi et al. (i.e., [ 139 ], for which, in fact, we were unable to locate any publication record at all). The remaining seven studies identified by Ouhbi et al. were not identified by us using the process described above. These studies are [ 8 , 23 , 42 , 91 , 96 , 199 ] and [ 178 ].

Two of the contributions identified by us are in fact Ouhbi et al.’s work [ 71 , 135 ]. During Step 4 in Sect.  3.3 , we included 50 papers which were published in the time period reported by Ouhbi et al.. Of these, we selected 33 contributions that were not reported by Ouhbi et al.. These papers are [ 3 , 16 , 22 , 28 , 44 , 45 , 49 , 53 , 58 , 59 , 60 , 62 , 83 , 85 , 98 , 115 , 116 , 118 , 122 , 127 , 133 , 134 , 143 , 151 , 152 , 153 , 156 , 166 , 173 , 174 , 175 , 185 , 187 , 188 ] and [ 192 ]. Thus, we have an agreement of 67.19% with the work of Ouhbi et al., which yields a Cohen’s \(\kappa\) of 0.3586 (i.e., fair agreement ) [ 26 ].

When comparing results with Ouhbi et al., search strategy accounts for some of the differences between included studies. We relied on Scopus (as this already covers the established publishers in the field) to search for articles, while Ouhbi et al. used the publishers’ search engines and Google Scholar. We purposefully used a more general search string than Ouhbi et al., as outlined above to investigate what authors believe RE Education shall be concerned with. Additionally, we applied stricter exclusion criteria.

In summary, we found more candidate papers but also excluded more. Like Ouhbi et al., we were interested in metadata about the papers. Yet, they investigated which studies referred to reference curricula, while our investigation focused on educational approaches and learning outcomes regardless of reference curricula, and the change of REE research since Ouhbi et al.’s work. Thus, our work is complementary.

Returning to goal 3, since we have covered sufficient common ground with the work of Ouhbi et al., we can provide valid observations about how the field of REE has evolved over the past decade.

4 Results for goal 1

Automatically generated map of author networks. Red lines indicate connections between authors, who are part of two collaboration groups. The darker the hue, the more co-authored papers (Color figure online)

Fragment of author networks only including those with more than one collaboration

Publications and venues per year. Each bar represents one year, with cumulative counts of publications per year (RQ4) listed at the top of each bar. Bars are sub-divided by type of publication venue (RQ3) to illustrate changes in venue over time

In this section, we present our systematic literature map (goal 1) and explore each research question (RQ1-10) in detail.

4.1 Most active researchers (RQ1)

We begin by exploring who is most involved in REE activities. Table  6 shows the most prolific authors in the area of REE. A total of sixteen authors contributed at least four published papers. Most prolific is Didar Zowghi from the University of Technology Sydney, Australia with nine published papers. As can further be seen, authors regularly involved in REE stem from around the world with a strong focus on Europe. Nine researchers are affiliated with universities from European Union countries: Germany (5), Spain (2), Portugal (1), and Italy (1). Three authors are affiliated with Malaysia, two with Australia, and one with Chile or the United States. Thus, we can see that while 152 articles were selected in our study, the majority of the contributions do not appear to be the primary scientific focus of the publishing scholars (with the exception of the individuals from Table  6 ), as most authors have fewer than two contributions in this field.

4.2 Research networks (RQ2)

Using study metadata, we automatically generated Fig.  3 , which shows the existing networks of authors found in the included studies. This gives a high-level overview of how segmented the efforts are in REE. Rectangles visualize collaborations between individual authors. As can be seen there exists a variety of individual collaborations that are not connected to other authors. Thus, we can assume that the field is rather scattered without collaborations between different author clusters. The coloring indicates the number of collaborations. Most authors participate in only one collaboration (light blue color), the maximum amount of collaborations is four between two authors (dark blue color). To improve readability and further explore existing networks of authors, we isolated networks with more than one collaboration to create a fragment of our map in Fig.  4 . Overall, these findings suggest that most selected studies appear to be separate contributions without a systematic continuation of a research direction. A notable exception is the work by Zowghi, Spoletini, Ferreira, and Bano from recent years, which investigates the use of interviews to practice requirements elicitation [ 13 , 40 , 41 ] and inspections [ 11 ].

4.3 Top venues (RQ3)

Table 7 shows all venues where multiple papers on REE have been published. We found fourteen venues where researchers regularly publish REE research. Yet, there are only five venues where REE seems to be published on a regular basis (i.e., with more than five total publications). The most established venue for REE is the International Workshop on Requirements Engineering Education and Training (REET) with 32 publications. The most established conferences are the IEEE International Conference on Software Engineering Education and Training (CSEE&T, 16) and the IEEE International Requirements Engineering Conference (RE, 10). The most established journal is Requirements Engineering (REJ), yet carries only five publications (of 152 total selected studies). This indicates that so far many early ideas and problem descriptions are elaborated on, with more mature research on REE being rarely addressed in the three most representative venues of requirements engineering research. According to Daneva et al. [ 29 ], these are REJ, RE, and the Working Conference on Requirements Engineering: Foundations for Software Quality (REFSQ), excluding their workshops. Yet, RE and REJ carry only 15 publications (ca. 9.7% of all 152 selected studies), while REFSQ is not represented in this list at all. Moreover, REJ is merely in sixth place (shared with the iStar workshops). In conclusion, REE research seems to be primarily published in education-related venues that are not specific to requirements engineering as well as the REET workshop. We conclude that there may be a missing connection between non-education research in RE and research specific to RE education.

Citation network of articles (dots) being cited by other articles (edges, head pointing to cited article). As can be seen, only six articles are cited four or more times, suggesting no common foundation of RE education literature

4.4 Paper per year (RQ4)

We found an increasing trend of publications over the years. Figure  5 shows the distribution of publications by year and type of venue where a paper was published (see also Sect.  4.3 ). As can be seen, research on REE started slowly in the beginning with only four conference papers between 1988 and 1998, and one journal article. This was followed by a phase from 1999 to 2007, where papers were regularly published, however in small numbers each year, and only in conferences. Since 2007, REE-related workshops have appeared and are in part responsible for the increase in publications reaching a maximum of seventeen publications in 2018. Thus far, 2018 was the year with the largest number of published journal papers. These findings suggest that REE has gained more and more interest over the years and its importance is shown in still increasing publication numbers. More than half of the publications selected in our work were published after the work by Ouhbi et al. [ 135 ]. Ouhbi et al.’s work was conducted in 2012 (almost 10 years ago), which coincided with the beginning of a four-year hiatus of REET. The eighth installment of REET was in 2013 and ninth and tenth installments were in 2018 and 2020, respectively. This in turn coincides with a period of slightly decreased frequency of workshop contributions and contributions at the three top venues for RE-specific research [ 29 , 179 ]: the REFSQ conference, the RE conference, and the Requirements Engineering journal (see also Sect.  4.3 ).

4.5 Top cited publications (RQ5)

We generated a citation network to analyze citation cycles. Figure  6 shows an excerpt of the citation network, i.e., the set of papers that cite other papers from all included studies. Arrow heads point to papers citing another paper (i.e., can be thought of as an “import” relationship). First, it can be seen that only about a third of all selected studies cite any papers within our set of 152 selected papers at all; the two-thirds of papers not citing any other papers have been omitted from Fig.  6 . Second, no paper is cited more than four times (see outgoing arrows in Fig.  6 ). Most papers cite merely one or two other papers and only five papers cites at least as many other papers (see ID 1001, 1005, 1058, 1008, and 1043 in Fig.  6 ). These are typically review papers. For example, the paper with the ID 1001 is the review paper by Ouhbi et al. [ 135 ]. Thus, we can conclude that no considerable citation cycles do exist. This means that neither is there as standard reference for REE accepted by the community. Thus, we found that most (i.e., at least two-thirds of our selected studies) REE research happens in “a vacuum,” without relying heavily on other findings in the field.

4.6 Research methods (RQ6)

We evaluated the papers based on the presented type of research, i.e., its underlying research method. Table  8 (left-hand side) shows the results separated by year. As can be seen the vast majority of papers are either solution proposals or experience reports. In contrast, evaluation research and validation research are only sparsely conducted. This means that while there exists a plethora of approaches aiming at improving REE and a variety of personal experience reports, more thorough empirical investigations of the field either by exploratory evaluation studies or by thoroughly validated solutions are missing. We conclude from this that the maturity of the field must be considered rather low. This is in line with our findings from RQ5, as indications of high overall maturity would be indicated by common, frequently cited references.

4.7 Contributions (RQ7)

For the contributions of the included studies, we mapped the publications according to the classification scheme proposed by Petersen et al. [ 140 ]. Table  8 (right-hand side) shows the results separated by year. Most publications propose a method, followed by tools to be introduced in REE. This is in line with findings by Malik and Zafar [ 105 ] (see also Sect.  2.2 ). In addition, we found a large number of papers classified as “other” . These mostly result from the large number of experience reports, which typically do not propose any kind of contribution in the sense of the categories in [ 140 ]. Nevertheless, we classified them according to their common theme, as shown in Fig.  7 . As can be seen, many papers classified as “other” in Table  8 report on limitations, pitfalls, or constraints, yet without specifying concrete solutions (17 in total). A total of 12 papers are concerned with involving real or realistic stakeholders (e.g., through role playing), while six papers propose a course design (without explicitly proposing it as a solution to a specific problem). Six more papers propose education research case studies and/or examples (often conflating the two terms), while again six studies report on empirical studies with students, surveys, or other types of investigations, yet without validating or evaluating a proposed solution (see Table  8 ). We infer from this that unlike non-education fields of software engineering, REE is fairly diverse, yet centers around proposing specific methods or approaches, or involving specific tools. This is in-line with our finding that most contributions are solution proposals (see RQ6). Although this further indicates low maturity of the field, it also means that a diverse set of contributions and solution avenues exist to teach RE, thereby suggesting a rich (albeit unsystematic) “toolbox” of educational approaches.

“Other” contributions from Table  8

4.8 Keywords (RQ8)

Table 9 shows the ten most frequently used keywords. As can be seen most keywords are basic terms. Beside these keywords, other keywords are used five times or less often. Thus, this indicates that—beside the topic of requirements elicitation—there seems to be no specific area of interest in requirements engineering that education research particularly focuses on. The frequent use of the term “requirements elicitation” indicates that for this specific area of RE there may be a particular interest in how to teach this topic. Yet, other areas of RE may not receive as much attention. This may make it difficult for educators interested in the field to find a solution to an instructional problem they are faced with, without being intimately familiar with the many solution proposals that exist in the field (see RQ6 and RQ7).

4.9 Learners (RQ9)

Figure  8 shows the distribution of the emphasized audience of teaching approaches as stated by the included publications. The vast majority of papers (120) clearly address university students. Of these, three papers consider postgraduate students, 21 focus on graduate students, 44 on undergraduates, and 52 do not further specify the level of the learner. Only 17 papers address teaching industry professionals. Thirteen papers omit the audience (“not mentioned” in Fig.  8 ) or generically speak of “students” (“unknown” in Fig.  8 ). We assume some of these address university education and find this sufficiently obvious that authors do not deem it important to specify this further. One paper places emphasis on RE education at the high-school level and another one investigates RE knowledge in alumni. This seems to show that Shaw’s “aspiration” [ 162 ] was in part answered, as a substantial number of approaches target aspiring software engineers in very early stages (i.e., at the undergraduate level) to instruct role-specific skills related to requirements engineering. Yet, by comparison, industry training is currently not a key focus in REE research.

Type of learners addressed

Topics of interest

4.10 Learning outcomes (RQ10)

To gain insights into what the included studies propose or investigate—and thus on the question what the current state of research in REE deals with from a content-related point of view—we identified broad recurring themes. Figure  9 shows these themes and their frequency.

Teaching requirements engineering activities. Most papers (i.e., 73) are concerned with teaching different RE activities. Recurring activities are elicitation, negotiation, specification, requirements validation, management, and modeling. In addition, specific activities as safety analyses or requirements tracing are concern of some publications.

Teaching soft skills. Forty-nine included studies focus on teaching soft skills when teaching RE. Targeted soft skills are typically closely related to the work profile of a requirements engineer. Papers commonly focus on communication skills, teamwork and collaboration skills, conflict resolution skills, interviewing techniques, or technical writing.

Improving student-related factors. In this category, 32 papers aim at improving the learning of students by increasing student motivation, enthusiasm for the subject matter, coping with overwhelmed students, or aim to improve students’ ability to explore problems and deal with solution uncertainty.

Improving industry readiness. A total of 29 publications aim at improving industry readiness of the students to cope with real RE problems. This is typically done by involving real stakeholders in a course, using or investigating real requirements specifications, or applying industry-realistic examples in the classroom.

Teaching requirements quality. In total 20 papers, focus on improving students’ sensitivity to high-quality requirements. Requirements quality properties mainly include consistency and correctness of requirements and requirements specification documents, but also ambiguity, and completeness.

Raising awareness for integrated RE processes. Although these eleven papers were included as they place particular emphasis on teaching RE, their focus lies on doing so as part of a broader development context, e.g., dealing with real customers’ needs.

Adaptability to professional environments. Nine papers propose specific educational settings to foster professional RE skills. For instance, this includes distributed global settings to mimic spatial separation of teams or teaching computer science students together with students from other disciplines to raise awareness of multidisciplinary issues.

This list shows that teaching requirements engineering activities is only part of what REE is concerned with, as about half of the papers deal with non-core requirements engineering theory.

In summary, the results presented in this section constitute our systematic map, which addresses goal 1. It is notable that since Mary Shaw’s aspiration (i.e., to include more role-specific undergraduate software engineering education, see [ 162 ]) has been answered by the REE community. A vast plethora of approaches have been proposed, especially since 2012 and beyond. Yet, the field suffers from low overall maturity. Most research appears to be solution proposals, without suggesting a continuing research avenue, and without producing a core area of expertise, neither surrounding scholars, nor surrounding methods, nor surrounding specific contributions. Nevertheless, we found that successful requirements engineering instruction encompasses more than theory, i.e., student factors and soft skills, as well as industry-readiness. Therein lie core themes in the papers we have discussed. In the next section, we address goal 2 and discuss the most significant trends pertaining to learning outcomes, as well as the educational approaches to achieve them.

5 Results for goal 2

Next, we explore goal 2 of this paper, which investigates the current practices regarding pedagogical techniques and the learning outcomes they seek to achieve. We initially hoped to distill these practices based on data from validated approaches. However, as can be seen by the results of RQ6 and RQ7 (see Sects.  4.6 and 4.7 , respectively), research contributions are overwhelmingly solution proposals or experience reports, with only 21.7% being evaluation or validation research. While several proposed solutions provide at least minimal quantitative or qualitative evidence as to their efficacy, a systematic replication and investigation of their pedagogical benefits is (unsurprisingly [ 25 , 163 ]) largely missing.

Nevertheless, as can be seen by the results from RQ10 (see Sect.  4.10 ), there are some clear and promising trends. These trends can be summarized into the following topics:

Authenticity and industry-readiness

Teaching RE activities and requirements quality

Student factors and soft skill development

To give further context to our discussion of learning outcomes, we tagged the papers in our mapping based on their educational approach, as explained in Sect.  3.4 shown in Fig.  10 .

Type of educational approach

Publications per year proposing an industry-centric learning outcomes and educational approaches

Studies explicitly instructing RE activities

5.1 Authenticity and industry-readiness

The first trend that we observed is the prominence of work that focuses on industry-readiness and giving students an authentic RE experience. We found that 32 papers (see Fig.  10 ) used an industry-centric educational approach (e.g., by involving external stakeholders from real companies), and 29 papers (see Fig.  9 ) explicitly mention industry-readiness as a learning outcome. Figure  11 shows these studies over time. From this timeline, we can see that this research focus is comparatively young, as half of these approaches have been published in the past 10 years alone.

Further, this trend suggests that the community is moving away from instructor-centric approaches, which focus on rote memorization of theory and individual high-stakes problem solving. Instead, nearly two-thirds (61.84%) of the studies shown in Fig.  10 propose a non-instructor-centric approach to instruct RE. Note that in Fig.  10 , approaches could pertain to more than one category. Nevertheless, we found 94 individual studies. These include the 32 aforementioned industry-centric approaches, as well as student collaboration (29 studies), project-based (13 studies), problem-based (9 studies), and other inquiry-based paradigms (20 studies (e.g., games [ 5 , 50 ] or case studies [ 110 , 182 ]). Of the 32 industry-centric studies, seven studies do so by fostering student collaboration (i.e., [ 27 , 38 , 123 , 125 , 167 , 177 , 180 ]), four do so through project-based instruction (i.e., [ 9 , 16 , 32 , 180 ]), and four through some other form of inquiry-based instruction (i.e., [ 27 , 53 , 168 , 177 ]). Within these 32 studies, two paradigms around stakeholder involvement can be differentiated: on one hand, approaches involve external stakeholders in realistic projects (e.g., [ 48 , 61 , 103 , 137 , 138 ]); while on the other hand, approaches involve the instructor (or some other non-industry representative, e.g., [ 47 , 177 ]) to engage in role-playing to create an industry-realistic project experience (e.g., [ 98 , 125 , 130 , 168 , 176 ]). In both of these paradigms, stakeholders serve as a partner to help students with requirements activities to some degree (see Sect.  5.3 ).

However, achieving industry authenticity is not necessarily done by involving real or mimicked stakeholders alone. Other approaches include using industry-realistic case examples (e.g., [ 30 , 31 , 32 ]) or using geographically distributed teams working on the same project (e.g., [ 9 , 16 , 149 , 152 , 198 ]). These approaches are interesting because they address soft skills in addition to industry-readiness (also see Sect.  5.3 ).

Evidence on the effectiveness of improving industry-authenticity relies on experience reports (e.g., [ 98 , 138 , 171 ]). Quantitative data are mostly available by means of students’ course evaluations (see [ 147 ]), or exam results (see [ 32 ]). Perhaps this is because student evaluations, assignment sheets, and exams are the typical means of assessment of student performance; however, another way of assessing learning outcomes is to measure student performance against industry needs, such as through a graduate alumni surveys of preparedness. This was done in [ 184 ], where researchers found that perceived usefulness of instructed documentation formats (e.g., use cases or glossaries) seem to increase with graduates’ work service and project experience.

In summary, we consider it a positive development that educational approaches have taken a keen focus on improving students’ industry-readiness and are moving away from rote memorization in favor of formative learning. However, many of these approaches aim at doing so without consideration of industry needs. Few approaches report on providing requirements engineering training to practitioners, with the notable exception of Morales-Ramirez et. al’s work in [ 123 ]. For both, more studies providing evidence are desirable.

5.2 Teaching RE activities and requirements quality

Next, we investigate trends in selecting topics to include in RE training. About half of the investigated studies focus on specific RE activities (73 studies in total, see Fig.  9 ). We visualize our selected categories of this breakdown in Fig.  12 . Among the most common are elicitation (39.72% of studies, e.g., [ 53 , 69 , 86 , 102 , 150 , 174 , 176 ]), modeling (28.77% of studies, which includes “modeling syntax” [ 17 , 34 , 66 ] and “process modeling” [ 107 , 159 ]). Eleven studies (15.07%) explicitly aim to instruct the whole RE process, while validation, verification, or quality assurance are a topic in only eight studies (10.96%, e.g., [ 41 , 69 ]), and management in only five studies (6.85%, including “time management,” “project management,” or “process management,” i.e., [ 14 , 38 , 100 , 114 , 121 ]). Surprisingly rarely do studies investigate more rigorous RE activities. For example, we found only three studies that look at security requirements engineering (from a process perspective, not quality perspective, i.e., [ 110 , 111 , 143 ]), three studies investigating formal methods (i.e., [ 44 , 188 , 191 ]), and two studies on requirements tracing (i.e., [ 19 , 116 ]). Safety requirements engineering was merely the elementary instructional focus of a single study (i.e., [ 180 ]).

Looking closer at the most investigated RE activity, elicitation, the vast majority of included studies (i.e., 19 out of 29) do so by means of using interviews as the predominant technique (see Table  10 ). Of the remaining ten papers, six studies did not specifically emphasize any particular elicitation technique, while three used specific technique (e.g., workshops) and one used various elicitation techniques including interviews.

Additionally, we considered which instructional approaches were used in elicitation activities. As shown in Table  11 , papers that used role playing were the most predominant approach. Other approaches included using real stakeholders, games, and tools. The contribution by Sedelmaier and Landes [ 161 ] was particularly noteworthy in this respect, not only because it is one of the few studies that employ a specific pedagogical paradigm (i.e., “competence-oriented didactics” in Table  11 ). While two papers did not specify any approach, we did not find any papers that studied the use of competitor or market analyses.

It is also noteworthy that some requirements engineering activities that could be considered essential (e.g., negotiation or prioritization) are not specifically targeted by RE education at all, as shown in Fig.  12 . While these activities may be subsumed in those studies targeting the “whole process,” the respective authors did not explicitly list all activities they included.

We observed a mismatch between teaching requirements quality properties (i.e., completeness, consistency, and traceability) and work advocating for a project-based learning environment. We found a total of 20 studies that explicitly mention teaching students to be sensitive to requirements quality. Of these, only two also pertain to those included in Sect.  5.1 (i.e., [ 58 , 180 ]). Figure  13 shows the breakdown of quality requirements publications, where studies may target more than one quality. Of the remaining 18 studies (see Fig.  13 ), the predominant focus is on requirements “consistency” (i.e., [ 49 , 62 , 69 , 166 , 174 , 188 , 190 , 191 ]). “Correctness” is targeted by five studies (i.e., [ 7 , 49 , 58 , 87 , 188 ]); however in doing so, studies often conflate formal provability of requirements and the sense of adequacy with regard to stakeholder needs (see [ 55 , 57 ] for a discussion of the difference). Only one included study explicitly mentions “adequacy,” but this is specifically in the context of security requirements [ 131 ]. Similarly, “completeness” is only explicitly targeted by Westphal in [ 191 ], albeit in the context of formal modeling of requirements. Four studies do not limit the educational focus on individual qualities, but rather mention “quality as a whole.” These studies are [ 11 , 41 , 53 ] and [ 59 ]. A total of six studies implicitly target requirements quality (“others” in Fig.  13 ). While they explicitly state the need to instruct sensitivity to high-quality requirements, the educational approach therein is not specifically targeted to requirements artifacts, but rather activities to improve quality in requirements. In this sense, “complexity” or “abstraction” are mentioned by three studies (i.e., [ 37 , 93 , 114 ]). The remaining studies each mention one quality property: traceability [ 60 ], ambiguity [ 174 ], and understandability [ 175 ].

As outlined in Sect.  4.6 , many of the studies we surveyed are “solution proposals” (see Table  8 ). Of these, most advocate for project-centric collaborative approaches and a minority advocate for instructor-centric, theory-heavy instruction. This is consistent with our earlier finding (see Sect.  5.1 ) that most approaches advocating industry-readiness do so in a project-based setting, requiring students to experience the whole RE process, from elicitation to documentation, to management. However, only ten of 73 studies mention a specific RE activity in an industry-realistic setting as opposed to targeting the whole RE process or not mentioning RE activities at all (i.e., [ 28 , 32 , 48 , 50 , 100 , 106 , 126 , 128 , 180 , 184 ]).

In summary, we found very little overlap between studies mentioned in Sect.  5.1 with studies aiming to teach RE activities and a focus on requirements quality. This seems to suggest that by increasing industry-readiness comes at the expense of teaching specific RE activities and requirements quality. However, we do not believe this to be the case. Many of the studies mentioned in Sect.  5.1 aim to convey a feeling of the intricacies of the whole RE process to students, not just individual activities. Moreover, it is the whole process experience which highlights issues such as completeness of requirements through elicitation and documentation, adequacy/correctness of requirements through validation and verification, requirements consistency and the like. However, while not ignored, it seems that these intricacies are at best conveyed implicitly. We did not find any study explicitly investigating how industry-readiness may also foster requirements activity proficiency and sensitivity to requirements quality, and recommend this as an area for future research.

Studies explicitly instructing requirements quality

5.3 Student factors and soft skill development

A third theme we found in our analysis is that many of the included studies emphasize student factors and soft skill development. This means that the focus is on “how” to conduct requirements activities effectively, thereby increasing soft-skills such as communication (as in [ 177 ]) or customer-orientation (as in [ 174 ]) rather than solely teaching “that” requirements elicitation is necessary. We tagged the publications emphasizing soft skills and visualize our results in Fig.  14 . The most frequently addressed soft skills are teamwork, collaboration, or social interaction (30.61% of studies pertaining to soft skills, e.g., [ 16 , 31 , 32 , 66 , 118 , 138 , 147 , 153 , 180 , 187 ]), interviewing skills (24.49%, e.g., [ 12 , 13 , 28 , 35 , 156 , 176 , 198 ]), customer interaction or client-orientation (16.33%, e.g., [ 69 , 118 , 125 , 127 , 157 , 174 ]), and communication (also 16.33% of soft skill studies, e.g., [ 27 , 149 , 152 , 153 , 159 , 168 , 177 ]). Only two studies focused on agile development as a soft skill (namely, [ 67 , 102 ]); however, most studies focusing on collaboration and communication applied a project-centric and industry-realistic (see Sect.  5.1 ) learning environment in conjunction with agile methods. Like in Sect.  5.2 , the overlap to those studies in Sect.  5.1 is fairly low, as only four studies appear to explicitly involve authentic industrial settings to improve students’ soft skills, i.e.,  [ 28 , 67 , 118 , 180 ].

However, as introduced above, most of the contributions whose primary focus is on soft skill development do so in a collaborative and/or project-based setting. Of the studies that apply a formative instruction paradigm (i.e., project-based, problem-based, and/or collaboration-based instruction in Fig.  10 ) and of the 49 studies that aim to improve students’ soft skills (see Fig.  9 ) as their primary learning outcome, the overlap consists of 19 studies (i.e., 30.65% of formative methods studies from Fig.  10 ). These studies mainly focus on communication, interviewing, and team collaboration in project-based settings.

The overlap between the same formative instructional approaches from Fig.  10 and studies specifically aiming to improve student factors is much lower. We identified a total of 12 studies that employ a project-, problem-, or collaboration-based instructional method in combination with the explicit aim of improving student factors. These factors include enthusiasm and motivation (e.g.,  [ 31 , 98 , 108 ]), comprehension and understanding (e.g.,  [ 103 , 117 , 181 ]), learning and retention (e.g.,  [ 16 ]), and introspection (e.g.,  [ 43 , 94 ]), which is a 19.35% overlap with formative approaches.

In total, we found 32 studies that propose a diverse set of pedagogical strategies to improve student factors, which we show in Fig.  15 (note, studies may pertain to more than one student factor). The most commonly addressed student factors are motivation/enthusiasm (11 studies in total i.e.,  [ 5 , 31 , 32 , 34 , 50 , 51 , 94 , 98 , 117 , 138 , 180 ]), understanding/comprehension (8 studies, i.e., [ 44 , 63 , 103 , 117 , 159 , 170 , 181 , 188 ]), retention/learning (7 studies, i.e.,  [ 6 , 15 , 16 , 34 , 66 , 122 , 187 ]), and engagement/ interest (also 7 studies, i.e.,  [ 5 , 31 , 32 , 94 , 117 , 138 , 180 ]). The remaining six studies target a diverse, yet more abstract set of student factors, i.e., “combating students being overwhelmed” [ 190 ], “effort and aggravation” [ 58 ], “review effectiveness” [ 134 ], “acceptance of uncertainty” [ 14 ], “process competency” [ 129 ], and “introspection” into the validation process (i.e.,  [ 11 , 13 , 41 ], which for the purpose of this discussion, we consider one contribution).

Besides formative and industry-centric approaches as outlined above, studies aiming to improve student factors and soft skills propose a diverse set of strategies to fulfill their aim. In particular, the use of games or gamification (e.g., [ 5 , 6 , 34 , 50 , 99 , 156 , 170 , 187 , 196 ]), engaging case examples (e.g., [ 3 , 9 , 30 ]), or using low-stakes assignments (e.g., [ 18 , 28 , 196 ]) are promising approaches that emerge from the literature.

In summary, while proposals for teaching specific RE activities are separate from improving student factors and soft skills (see Sect.  5.2 ), we found that student factors and soft skills are a tangential learning outcome of this work. By comparison, industry-authenticity specifically adopts external stakeholders or role playing as an instructional mechanism in order to improve student motivation and enthusiasm (see Sect.  5.1 ). The REE literature recognizes that soft skills are critical for students’ success in future employment and that student factors are critical in improving student success in requirements engineering. Nevertheless, more work on how to successfully and holistically integrate theory instruction and student success is desirable.

Studies explicitly improving soft skills

Studies explicitly improving student factors

6 Results for goal 3

In this section, we address goal 3 by evaluating how REE has changed over the last decade. To accomplish this goal, we compare our findings to those from Ouhbi et al. [ 71 , 135 ]. In Sect.  3.8 , we compared our literature search methodology and results to those from Ouhbi et al. to contextualize our analysis. In the following subsections, we compare to Ouhbi et al.’s “implications and advice for instructors” and what REE research has contributed since the study concluded. Finally, we identify additional gaps in current REE literature and offer our own conclusions.

6.1 How literature addresses Ouhbi et al.’s “implications”

Following a detailed map of the REE field, Ouhbi et al. provide advice to REE instructors in the form of seven implications derived from their selected studies. In this section, we discuss these implications and contrast them with studies published after the period of investigation reported by Ouhbi et al., which allows us to consider the progress in the field since 2012. Furthermore, we expand on these implications with our own observations and recommendations for REE instructors.

6.1.1 Combating vague requirements

Ouhbi et al. recommend that instructors teach proper problem scoping in order to avoid vague requirements. The authors assert that certain personality traits improve team performance in this respect. Indeed, such a relationship exists [ 164 , 165 , 189 ] and as we have outlined above, student factors such as comprehension, effort, and enthusiasm are explicitly mentioned learning outcomes in 32 of our 152 selected studies. Eighteen of these studies fall into a time frame after Ouhbi et al.’s search completed. While none of these studies mention “vagueness” or “attention to detail” explicitly, several mention “introspection” (e.g., [ 13 ]) or “comprehension” (e.g., [ 117 ]). Unfortunately, “vagueness” or “level of detail” was not mentioned as a learning outcome in any of our selected studies. We conclude from this that instructors have recognized that student factors are crucial in educating students to become effective requirements engineers, yet student factors alone do not yield effective requirements specifications. We recommend instructors consider pedagogical techniques aimed to increase the level of detail and thereby combat vagueness in requirements specifications.

6.1.2 RE tool instruction

With hundreds of RE tools available on the market, Ouhbi et al. make a strong argument for the need to educate students into using these tools effectively. Indeed, 27 of our selected studies deal with tools or advocate using technology to improve learning and instruction. However, the overwhelming majority of these studies propose games (e.g., [ 5 , 50 , 51 , 70 , 156 , 170 , 187 , 196 ]) or simulation tools (e.g., [ 10 , 150 , 154 ]) to teach RE. They do not outline how to use tools during the RE process. A notable exception is [ 87 ], where requirements modeling using tools is taught as well as [ 106 ], which in part investigates the use of tools to conduct validation and verification of requirements. Our results show that RE tool instruction is still lacking, nearly 10 years after the conclusion of Ouhbi et al.’s survey. A focus here should be on industry-typical tools and tools that are likely to produce a tangible benefit to the RE process, for which current industry needs are unknown and must be assessed (see also Sect.  5.1 ). Nevertheless, using the right tools during RE also depends on the company-specific tool chains and may therefore be “on the job training,” rather than something that can (or should) be instructed at university level. Again, an industry perspective is required to answer this question.

6.1.3 Promote requirements modeling, validation and verification, and prototyping

Our results in Sect.  5.2 show that next to elicitation and RE process instruction, the most commonly addressed RE activities are modeling of any kind as well as quality assurance at large (a total of 29 studies, see Fig.  12 ). Most of these studies occurred before 2012 (i.e., when Ouhbi et al. concluded) with the exception of [ 17 , 66 , 87 , 117 , 125 , 184 ] and [ 190 ]. Ouhbi et al. were correct to point out that more instructional focus was required, as these 29 studies made up a mere 19.1% of all our selected studies (compared to 18.4% for “elicitation” alone). We agree that these activities (i.e., modeling and prototyping) should be promoted during RE instruction. Modeling and requirements validation have proven to be a key asset in the requirements engineer’s toolbox to bridge the gap between non-technical and technical stakeholders. Teaching non-technical skills has thus far mostly taken the form of soft skills (see Sect.  5.3 ), but even in this regard, the focus is on communication and interviewing due to the strong overlap with studies that focus on “elicitation” (see Fig.  14 ). Prototyping of requirements specifications has not been emphasized or made a key learning outcome in any of our selected studies. An opportunity here is lost in that students do not benefit from seeing the relationship between “theoretic” requirements specifications and their implementation. While we have reported an activity involving requirements prototyping in one of our selected studies [ 180 ], this was only a minor milestone in a RE project, burdened by other constraints in the timeline of the semester. We recommend practitioners to develop approaches such as [ 108 ] and incorporate requirements implementation as well.

6.1.4 Using industry-realistic projects

As outlined in Sect.  5.1 , delivering an authentic, industry realistic educational experience has consistently been a focus of REE literature since roughly 2005 (see Fig.  11 ). In Ouhbi et al.’s study, the focus was on REE approaches and their relationship to standard curricula, many which require industry-readiness as a student outcome (e.g., [ 1 ]). While this is a positive trend in the past, we concur with Ouhbi et al. that this remains an important educational outcome for future work in REE.

6.1.5 Promote global software development

Ouhbi et al. emphasize the need for REE approaches to meet the demands placed on software development through a consistent move toward distributed teams. In particular, in light of the recent events (i.e., the COVID-19 pandemic), we agree that video conferencing and distributed teamwork have become necessary skills for students, educators, and industry professionals to master, and will likely shape the landscape of software development for the coming years. Teaching effective RE in such a context may be easier going forward because learners may be accustomed to social distancing and working remotely. Nevertheless, only a minority of our selected studies consider distance learning or geographically separated teams, only one of which was published after 2012 (i.e., [ 16 , 108 , 149 , 152 , 198 ]). This must be a focus of REE approaches going forward, and these approaches could build off of the experiences from forced distancing during the COVID-19 pandemic.

6.1.6 Familiarize students with problem solving

Ouhbi et al. highlight the importance of problem solving skills to become effective requirements engineers and recommend REE literature to take a game-based approach to problem solving. While “problem solving” was only explicitly mentioned in one of our included studies (i.e., [ 17 ]) and while games-based instruction or gamification is the focus of several of our selected studies (e.g., [ 5 , 101 , 109 , 187 ]), we argue that these approaches are not the only strategies to teach effective problem solving. In fact, peer-learning [ 27 ], role-playing [ 4 , 98 , 168 ], fostering analytical thinking [ 64 ], and client-orientation [ 157 , 174 ] have been successfully applied to aspects of RE instruction. Problem solving is at the heart of RE. The key caveat seems to be to create a low-stakes environment, where students can “safely fail” (i.e., explore solution alternatives without grade penalty for being wrong or without threatening project success). Approaches that offer low-stakes learning experiences are quite common, both in individual-centered instruction (e.g., [ 113 , 158 ]) and collaborative instruction (e.g., [ 28 , 31 , 119 ]). In fact, 24 of our selected studies can be roughly categorized as employing some form of low-stakes problem solving experience; a trend that should continue in the future.

6.1.7 Use mobile devices as teaching tools

Ouhbi et al. made an argument to use mobile devices and online tools as a vehicle to teach RE. However, Ouhbi et al. did not articulate in what way REE, in particular, benefits from m-learning or e-learning. When examining our selected studies, only three mention some type of online platform or the use of mobile devices to teach RE (i.e., [ 88 , 124 , 144 ]). We conclude from this that the benefits of m-learning and e-learning to REE may still be largely unexplored, beyond the opportunity to prepare students for the challenges of global software development (see Sect.  6.1.5 ).

6.2 Gaps in current RE education literature

While industry-readiness, authenticity, and student soft skill development are important and encouraging trends in REE literature, in the following sections, we highlight the areas that have not received sufficient attention.

6.2.1 Safety and security requirements

Shockingly few studies (i.e., only three: [ 110 , 111 , 143 ]) deal with security requirements and only one study considers safety requirements explicitly [ 180 ]. Since software systems are increasingly entrusted with sensitive information and playing a mission-critical role, it is vital that students are exposed to these considerations at the earliest possible stage during their undergraduate curriculum. Further work is required to understand how to effectively instruct learners on the intricate notions of security requirements and their impact on the system under development. While some studies may incidentally involve safety and security requirements, a systematic educational approach is required.

6.2.2 Supply chain risk management and supplier/integrator relation

Most project-based approaches involving real or realistic stakeholders aim to convey the difficulty of managing conflicting requirements. However, these approaches may prime students towards an attitude of “document and forget” [ 32 ]. Requirements are rarely seen through to their implementation (see “prototyping” in Sect.  6.1 ). Moreover, typical software engineering projects emphasize software construction. The current literature largely ignores the need to systematically explore reuse of off-the-shelf components, the need to critically reflect on adopting components (e.g., libraries), or risk involved when adopting possible security-critical technologies. The decision to adopt a technology and risk its successful integration are inherently RE-related and must be systematically assessed on the basis of requirements. At present, students do not achieve this learning outcome with the approaches reported herein.

6.2.3 Pedagogy in RE education

Systematic application of pedagogy is largely ignored by contemporary REE literature. Merely two approaches make explicit use of Bloom’s taxonomy to guide their instruction [ 19 , 124 ] and only 10.5% of approaches (i.e., [ 4 , 7 , 11 , 19 , 40 , 41 , 53 , 92 , 104 , 112 , 127 , 145 , 155 , 158 , 160 , 161 , 176 ]) consider systematic pedagogy. Yet, with the exception of closely related studies such as [ 11 , 40 , 41 ], there seems to be no common pedagogical framework nor is there a common basis of systematically gathered evidence as to the effectiveness of teaching approaches given learning outcomes. In fact, to the best of our knowledge, the manuscript at hand is the first and thus far only systematic investigation into REE literature and students’ learning outcomes. We therefore declare a call to action for the REE community (and perhaps the software engineering education community at large) to produce a common, evidence-based pedagogical framework. We hope that the work at hand lays a suitable foundation for such an effort.

7 Discussion, conclusions, and future work

In this paper, we presented the results of a systematic literature review into learning outcomes portrayed in Requirements Engineering Education (REE) literature. We have selected 152 primary studies from 1988 to 2020, to provide three contributions: (goal 1) We provide a systematic map of the current state of REE research. (Goal 2) We review the current practices and educational approaches to achieve learning outcomes. (Goal 3) We show how REE has changed in the last decade and which topics remain unexplored in the literature.

Our main findings include the recent trend towards authentic and industry-realistic learning experiences to improve students’ knowledge, predominantly on topics such as requirements elicitation and modeling, but also with regards to students’ soft skills, collaboration, teamwork, and industry-readiness. To accomplish this, current trends involve real or realistic stakeholders and role playing in low-stakes collaborative project-based instruction scenarios. Theory-based instruction plays a subordinate role in REE, suggesting that knowing about theory is less emphasized than effectively applying theory in industry-realistic settings, ideally spanning all parts of the RE process.

Our findings further suggest that areas where REE approaches are currently lacking include instruction of safety and security requirements engineering, as well as supply chain risk management. Moreover, REE presently suffers from a lack of a common pedagogical basis and systematically gathered evidence. While a plethora of successful teaching methods have been proposed (e.g., game-based learning, new frameworks, and educational tools), for the most part, these contributions are in isolation and not part of a systematic attempt to propose methods that are tailored to student outcomes.

We contrast and complement findings from a previous mapping study by Ouhbi et al. [ 135 ]. While Ouhbi et al.’s work focuses on REE approaches and their consideration of standardized curricula, we place emphasis on synthesizing learning outcomes and educational approaches reported in the literature. We also highlight developments in the field since Ouhbi et al.’s study concluded in 2012. In part, we were able to replicate Ouhbi et al.’s results, differ in some findings, and provide additional findings not previously reported.

To our knowledge, a replication of a systematic literature review or mapping study has thus far not yet been completed in the discipline of software engineering. While it was not our aim to replicate Ouhbi et al.’s work, we believe that the work at hand sufficiently highlights areas of overlap. This produces a secondary outcome of our work, i.e., that differences between our findings can be explained by differences in search methodology and as well as rigor in inclusion and exclusion criteria.

In this paper, we lay a foundation for the REE community to produce a rich evidence-based understanding of effective pedagogical approaches. Given the vastness of our data set, we envision future work focusing on qualitative analysis of previous studies to uncover new insights. For example, we found that interviews for elicitation is well studied in the literature. Future work could look at which other elicitation techniques are taught (e.g., questionnaires, analyzing competitors). Similarly, other studies could investigate how requirements quality metrics (e.g., correctness, consistency) are instructed.

In addition to studying the level of learners (see Sect.  4.9 ), future work could study these educational approaches with respect to which approaches are taught as part of introductory, intermediate, or advanced courses in RE and SE, at both the bachelors and master levels. This would give greater insight into the depth of RE curriculum, and would be complementary to initial efforts  [ 64 , 67 ]. Supporting this line of inquiry, we also intend to survey educators to identify best practices and examine whether there are any instructional approaches that could be of relevance for RE education but have not been published.

As already introduced in Sect.  5 , we found 33 papers (i.e., 21.7% of selected studies) with validated approaches, which was insufficient for our intended analysis. Given the importance of evidence as to the effectiveness of pedagogy, we seek to complete an in-depth qualitative analysis of these papers as part of future work in order to provide insights to what works and what does not work. By looking more deeply at RE activities, we can assist new educators in understanding what is recommended.

In addition, as already discussed in the paper (see Sects.  5.1 ,  5.2 , and  6.1.4 ), we found the further research is required to explicitly investigate the relationship between industry-readiness and requirements proficiency among students. Finally, as proposed in Sect.  6.2.1 , we need a systematic educational approach to instruct students on the development and importance of security requirements.

Available at https://2021.refsq.org/details/refsq-2021-papers/3/The-Challenge-s-of-Teaching-Requirements-Engineering .

The data set is available here: https://doi.org/10.35482/csc.003.2021

ABET Engineering Accreditation Commission (2018) Criteria for accrediting engineering programs, 2018–2019

Abran A, Bourque P, Tripp LL (2004) Guide to the software engineering body of knowledge (SWEBOK(R)): version 3.0, 1st edn. IEEE Computer Society Press, Washington, DC

Google Scholar  

Adam S, Doerr J, Eisenbarth M (2009) Lessons learned from best practice-oriented process improvement in requirements engineering—a glance into current industrial RE application. In: Fourth international workshop on requirements engineering education and training, pp 1–5

Al-Ani B, Yusop N (2004) Role-playing, group work and other ambitious teaching methods in a large requirements engineering course. In: Proceedings of 11th IEEE international conference and workshop on the engineering of computer-based systems, pp 299–306

Alami D, Dalpiaz F (2017) A gamified tutorial for learning about security requirements engineering. In: Proceedings of IEEE 25th international requirements engineering conference (RE), pp 418–423

Alexander M, Beatty J (2008) Effective design and use of requirements engineering training games. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET)

Anil GR, Moiz SA (2017) A holistic rubric for assessment of software requirements specification. In: Proceedings of 5th national conference on E-learning and E-learning technologies (ELELTECH), 2017

Armarego J, Minor O (2005) Studio learning of requirements: towards aligning teaching to practitioner needs. In: REET’05 (1st international workshop on RE education and training). REET, 2005

Auriol G, Baron C, Fourniols J-Y (2008) Teaching requirements skills within the context of a physical engineering project. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET)

Babiceanu RF (2014) A software and systems integration framework for teaching requirements engineering. In: Proceedings of 121st ASEE annual conference and exposition, 2014

Bano M, Zowghi D, Ferrari A, Spoletini P (2020) Inspectors academy: pedagogical design for requirements inspection training. In: 2020 IEEE 28th international requirements engineering conference (RE), pp 215–226

Bano M, Zowghi D, Ferrari A, Spoletini P, Donati B (2018) Learning from mistakes: an empirical study of elicitation interviews performed by novices. In: Proceedings of IEEE 26th international requirements engineering conference (RE), pp 182–193

Bano M, Zowghi D, Ferrari A, Spoletini P, Donati B (2019) Teaching requirements elicitation interviews: an empirical study of learning from mistakes. Requir Eng 24(3):259–289

Article   Google Scholar  

Barnes RJ, Gause DC, Way EC (2008) Teaching the unknown and the unknowable in requirements engineering education. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET)

Bennaceur A, Lockerbie J, Horkoff J (2015) On the learnability of i*: experiences from a new teacher. In: 1st International iStar teaching workshop (iStarT 2015), vol 1370, pp 43–48

Berkling K, Geisser M, Hildenbrand T, Rothlauf F (2007) Offshore software development: transferring research findings into the classroom. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol. 4716 LNCS, pp 1–18

Berre AJ, Huang S, Murad H, Alibakhsh H (2018) Teaching modelling for requirements engineering and model-driven software development courses. Comput Sci Educ 28(1):42–64

Berry DM, Kaplan CS (2010) Planned programming problem gotchas as lessons in requirements engineering. In: Proceedings of 5th international workshop on requirements engineering education and training, pp 20–25

Bhowmik T, Niu N, Reese D (2014) Students vs. professionals in assisted requirements tracing: how could we train our students? In: Proceedings of 121st ASEE annual conference and exposition

Bourque P, Fairley RE, IEEE Computer Society (2014) Guide to the software engineering body of knowledge (SWEBOK(R)): version 3.0, 3rd edn. IEEE Computer Society Press, Washington, DC

Brings J, Daun M, Kempe M, Weyer T (2018) On different search methods for systematic literature reviews and maps: experiences from a literature search on validation and verification of emergent behavior. In: Proceedings of the 22nd international conference on evaluation and assessment in software engineering 2018, pp 35–45

Bubenko JA (1995) Challenges in requirements engineering. In: Proceedings of 1995 IEEE international symposium on requirements engineering (RE’95), pp 160–162

Callele D, Makaroff D (2006) Teaching requirements engineering to an unsuspecting audience. SIGCSE Bull 38(1):433–437

Cico O, Jaccheri L, Nguyen-Duc A, Zhang H (2020) Exploring the intersection between software industry and software engineering education—a systematic mapping of software engineering trends. J Syst Softw 172:110736

Cockburn A, Dragicevic P, Besançon L, Gutwin C (2020) Threats of a replication crisis in empirical computer science. Commun ACM 63(8):70–79

Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46

Connor AM, Buchan J, Petrova K (2009) Bridging the research-practice gap in requirements engineering through effective teaching and peer learning. In: Proceedings of sixth international conference on information technology: new generations, pp 678–683

Cybulski JL, Parker C, Segrave S (2006) Touch it, feel it and experience it: developing professional is skills using interview-style experiential simulations. In: Proceedings of 17th Australasian conference on information systems, 2006

Daneva M, Damian D, Marchetto A, Pastor O (2014) Empirical research methodologies and studies in requirements engineering: how far did we come? J Syst Softw 95:1–9

Daun M (2020) Teaching requirements engineering with industry case examples. In: Software engineering Unterricht and Hochschulen (SEUH), 2020

Daun M, Salmon A, Tenbergen B, Weyer T, Pohl K (2014) Industrial case studies in graduate requirements engineering courses: the impact on student motivation. In: Proceedings of IEEE 27th conference on software engineering education and training (CSEE&T), pp 3–12

Daun M, Salmon A, Weyer T, Pohl K, Tenbergen B (2016) Project-based learning with examples from industry in university courses: An experience report from an undergraduate requirements engineering course. In: Proceedings of IEEE 29th international conference on software engineering education and training (CSEE&T), pp 184–193

Davis AM, Hickey AM, Chamillard AT (2005) Moving beyond the classroom: integrating requirements engineering research & education to improve practice

de Pádua Albuquerque Oliveira A, Werneck VMB, do Prado Leite JCS, Cysneiros LM (2015) The monopoly game to teach ERi*c—intentional requirements engineering. In: 1st International iStar teaching workshop (iStarT 2015), vol 1370, pp 49–54

Donati B, Ferrari A, Spoletini P, Gnesi S (2017) Common mistakes of student analysts in requirements elicitation interviews. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 10153. LNCS, pp 148–164

Engström E, Runeson P (2011) Software product line testing—a systematic mapping study. Inf Softw Technol 53(1):2–13

Feldgen M, Clua O (2015) Teaching effective requirements engineering for large-scale software development with scaffolding. In: Proceedings of IEEE frontiers in education conference (FIE), vol 2015-February, 2015

Fernandes JM, Machado RJ, Seidman SB (2009) A requirements engineering and management training course for software development professionals. In: 2009 22nd conference on software engineering education and training, February 2009, pp 20–25

Fernández DM, Franch X, Seyff N, Felderer M, Glinz M, Kalinowski M, Volgelsang A, Wagner S, Bühne S, Lauenroth K (2019) Do we preach what we practice? Investigating the practical relevance of requirements engineering syllabi—the IREB case

Ferrari A, Spoletini P, Bano M, Zowghi D (2019) Learning requirements elicitation interviews with role-playing, self-assessment and peer-review. In: 2019 IEEE 27th international requirements engineering conference (RE), pp 28–39

Ferrari A, Spoletini P, Bano M, Zowghi D (2020) SaPeer and ReverseSaPeer: teaching requirements elicitation interviews with role-playing and role reversal. Requir Eng 25(4):417–438

Ferrari R, Madhavji N (2005) Requirements engineering education for novice software architects. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET), 2005

Ferreira VG, Canedo ED (2019) Using design sprint as a facilitator in active learning for students in the requirements engineering course: An experience report. In: Proceedings of the 34th ACM/SIGAPP symposium on applied computing (SAC19), pp 1852–1859

France RB, Larrondo-Petrie MM (1995) Understanding the role of formal specification techniques in requirements engineering. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 895, pp 207–221

Fuji T (2005) Finding competitive advantage in requirements analysis education. In: 13th IEEE international conference on requirements engineering (RE’05), pp 493–494

Gabrysiak G, Giese H, Seibel A (2011) Why should I help you to teach requirements engineering? In: 2011 6th International workshop on requirements engineering education and training, August 2011, pp 9–13

Gabrysiak G, Giese H, Seibel A, Neumann S (2010) Teaching requirements engineering with virtual stakeholders without software engineering knowledge. In: Proceedings of 5th international workshop on requirements engineering education and training, pp 36–45

Gabrysiak G, Hebig R, Pirl L, Giese H (2013) Cooperating with a non-governmental organization to teach gathering and implementation of requirements. In: 2013 26th International conference on software engineering education and training (CSEE T), May 2013, pp 11–20

Garbers B, Periyasamy K (2006) A light-weight tool for teaching the development and evaluation of requirements documents. In: Annual conference of American Society of Engineering Education (ASEE), 2006

Garcia I, Pacheco C, León A, Calvo-Manzano JA (2019) Experiences of using a game for improving learning in software requirements elicitation. Comput Appl Eng Educ 27(1):249–265

García I, Pacheco C, León A, Calvo-Manzano JA (2020) A serious game for teaching the fundamentals of ISO/IEC/IEEE 29148 systems and software engineering—lifecycle processes—requirements engineering at undergraduate level. Comput Stand Interfaces 67:103377

Garousi V, Mesbah A, Betin-Can A, Mirshokraie S (2013) A systematic mapping study of web application testing. Inf Softw Technol 55(8):1374–1396

Gary KA (2009) Contextual requirements experiences within the software enterprise. In: Proceedings of fourth international workshop on requirements engineering education and training, pp 12–19

Gibson JP (2000) Formal requirements engineering: learning from the students. In: Proceedings 2000 Australian software engineering conference, pp 171–180

Glinz M (2000) Improving the quality of requirements with scenarios

Glinz M (2020) Standard glossary for the certified professional for requirements engineering (CPRE) studies and exam v2.0.0. Technical report, International Requirements Engineering Board e.V., October 2020

Glinz M, Fricker SA (2015) On shared understanding in software engineering: an essay. Comput Sci Res Dev 30(3):363–376

Goldsmith RF (2009) BAs will falter until they learn to discover REAL, business requirements. In: Proceedings of fourth international workshop on requirements engineering education and training (REET ’09), pp 6–11

Gotel O, Kulkarni V, Say M, Scharff C, Sunetnanta T (2009) Distributing responsibilities to engineer better requirements: leveraging knowledge and perspectives for students to learn a key skill. In: Fourth international workshop on requirements engineering education and training, pp 28–37

Gotel OCZ, Morris SJ (2012) Case-based stories for traceability education and training. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET), pp 1–8

Hagel G, Müller-Amthor M, Landes D, Sedelmaier Y (2018) Involving customers in requirements engineering education: mind the tgoals! In: Proceedings of 3rd European conference of software engineering education (ECSEE), pp 113–121

Hasson P, Cooper S (2004) A case study involving the use of Z to aid requirements specification in the software engineering course. In: Proceedings of 17th conference on software engineering education and training, vol 17, pp 84–89

Heimbürger A, Isomöttönen V (2019) Infographics as a reflective assignment method in requirements engineering e-course? In: 2019 IEEE frontiers in education conference (FIE), pp 1–5

Hertz K, Spoletini P (2018) Are requirements engineering courses covering what industry needs? A preliminary analysis of the United States situation. In: IEEE 8th international workshop on requirements engineering education and training (REET), pp 20–23

Hmelo-Silver CE (2004) Problem-based learning: what and how do students learn? Educ Psychol Rev 16:235–266

Horkoff J (2015) Observational studies of new i* Users: challenges and recommendations. In: 1st International iStar teaching workshop (iStarT 2015), vol 1370, pp 13–18

Horkoff J (2018) The influence of agile methods on requirements engineering courses. In: Proceedings of IEEE 8th international workshop on requirements engineering education and training (REET), pp 11–19

Huijs C, Sikkel K, Wieringa R (2005) Mission 2 solution: Requirements engineering education as central theme in the BIT programme

Iacob C, Faily S (2017) Using extreme characters to teach requirements engineering. In: Proceedings of IEEE 30th conference on software engineering education and training (CSEE&T), vol 2017-January, pp 107–111

Ibrahim Z, Soo MC, Soo MT, Aris H (2019) Design and development of a serious game for the teaching of requirements elicitation and analysis. In: 2019 IEEE international conference on engineering, technology and education (TALE), pp 1–8

Idri A, Ouhbi S, Fernández-Aléman JL, Toval A (2012) A survey of requirements engineering education. In: Proceedings of the 2012 IEEE global engineering education conference (EDUCON), pp 1–5

IEEE (1998) IEEE standard for conceptual modeling language syntax and semantics for IDEF1X/Sub 97/ (IDEF/Sub Object/). IEEE Std 1320.2-1998

IEEE (2014) IEEE standard for software quality assurance processes. IEEE Std 730-2014 (Revision of IEEE Std 730-2002), pp 1–138

IEEE & ACM JTFCC, Software Engineering (2004) Curriculum guidelines for undergraduate degree programs in software engineering. Technical report, IEEE & ACM; The Joint Task Force on Computing Curricula

IREB (2020) Certified professional for requirements engineering foundation level syllabus v3.0.1. Technical report, International Requirements Engineering Board e.V., October 2020

ISO/IEC (2002) ISO/IEC 15474-1:2002 information technology—CDIF framework—part 1: overview

ISO/IEC (2008) ISO/IEC 14102:2008 information technology—guideline for the evaluation and selection of CASE tools

ISO/IEC (2009) ISO/IEC 10746-2:2009 information technology—open distributed processing—reference model: foundations

ISO/IEC (2012) ISO/IEC 19500-2:2012 information technology—object management group—common object request broker architecture (CORBA)—part 2: interoperability

ISO/IEC (2014) ISO/IEC 25000:2014 systems and software engineering—systems and software quality requirements and evaluation (SQuaRE)—guide to SQuaRE

ISO/IEC (2015) ISO/IEC 2382:2015 information technology—vocabulary

ISO/IEC/IEEE (2017) ISO/IEC/IEEE Int. standard—systems and software engineering—vocabulary. ISO/IEC/IEEE 24765:2017(E)

Jagielska D, Wernick P, Wood M, Bennett S (2006) How natural is natural language?: how well do computer science students write use cases? In: Companion to the 21st ACM SIGPLAN symposium on object-oriented programming systems, languages, and applications, vol 2006, pp 914–924

Jamaludin NAA, Sahibuddin S, Hidayat NH (2012) Challenges of a project-based learning approach towards requirement engineering. Int J Comput Appl 50(3):66–71

Jiang Y, Li M, He Z, Zhao C (2009) Nine steps to shorten the distance between requirement theory and practice. In: First international workshop on education technology and computer science, vol 3, pp 694–698

Kakeshita T, Yamashita S (2015) A requirement management education support tool for requirement elicitation process of REBOK. In: 3rd International conference on applied computing and information technology/2nd international conference on computational science and intelligence, pp 40–45

Keller K, Neubauer A, Brings J, Daun M (2018) Tool-support to foster model-based requirements engineering for cyber-physical systems. In: Modellierung (workshops), vol 2060, pp 47–56

Kilicay-Ergin N, Laplante PA (2013) An online graduate requirements engineering course. IEEE Trans Educ 56(2):208–216

Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075

Klapholtz D, McDonald J, Pyster A (2009) The graduate software engineering reference curriculum (gswerc). In: 2009 22nd Conference on software engineering education and training, pp 290–291

Knauss E, Schneider K, Stapel K (2008) A game for taking requirements engineering more seriously. In: 2008 Third international workshop on multimedia and enjoyable requirements engineering—beyond mere descriptions and with more fun and games, pp 22–26

Koch M, Landes D (2015) Making means-end-maps workable for recommending teaching methods. In: Proceedings of eighth international i* workshop, vol 1402, pp 85–90

Kramer J (2003) Abstraction—is it teachable? ‘the devil is in the detail’. In: 16th Conference on software engineering education and training, vol 2003-January, p 32

Kurkovsky S, Ludi S, Clark L (2019) Active learning with LEGO for software requirements. In: Proceedings of the 50th ACM technical symposium on computer science education, SIGCSE ’19, New York, NY, USA. Association for Computing Machinery, pp 218–224

Laiq M, Dieste O (2020) Chatbot-based interview simulator: a feasible approach to train novice requirements engineers. In: 2020 10th International workshop on requirements engineering education and training (REET), pp 1–8

Lami G (2005) Teaching requirements engineering in the small: an under-graduate course experience. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET), 2005

Lavallee M, Robillard P-N, Mirsalari R (2013) Performing systematic literature reviews with novices: an iterative approach. IEEE Trans Educ 57(3):175–181

Liang P, De Graaf O (2010) Experiences of using role playing and wiki in requirements engineering course projects. In: Proceedings of 18th international IEEE requirements engineering conference, pp 1–6

Lima T, Campos B, Santos R, Werner C (2012) UbiRE: a game for teaching requirements in the context of ubiquitous systems. In: Proceedings of XXXVIII Conferencia Latinoamericana En Informatica (CLEI), 2012

Liu L, Jin Z (2008) Balancing academic and industrial needs in RE courses. In: Requirements engineering education and training, 2008

Lopes J (2020) Evaluating the students’ experience with a requirements elicitation and communication game. In: Proceedings of 23rd Ibero-American conference on software engineering (CIbSE 2020), ClbSE 2020

Lopez-Lorca A, Burrows R, Sterling L (2018) Teaching motivational models in agile requirements engineering. In: Proceedings of 8th international workshop on requirements engineering education and training (REET), pp 30–39

Ludi S (2007) Introducing accessibility requirements through external stakeholder utilization in an undergraduate requirements engineering course. In: Proceedings of 29th international conference on software engineering (ICSE’07), pp 736–743

Macaulay L, Mylopoulos J (1995) Requirements engineering: an educational dilemma. Autom Softw Eng 2(4):343–351

Malik B, Zafar S (2012) A systematic mapping study on software engineering education. Int J Educ Pedag Sci 6(11):3343–3353

Manohar P, Acharya S, Wu PY, Ansari AA, Schilling WW Jr (2015) Case study based educational tools for teaching software V&V course at undergraduate level. In: 122nd ASEE annual conference and exposition: making value for society, 2015

Marsicano G, Mendes FF, Fernandes MV, Freitas SAAD (2016) An integrated approach to the requirements engineering and process modelling teaching. In: IEEE 29th international conference on software engineering education and training (CSEET), pp 166–174

Marutschke DM, Kryssanov VV, Brockmann P (2020) Teaching distributed requirements engineering: simulation of an offshoring project with geographically separated teams. In: 2020 IEEE 32nd conference on software engineering education and training (CSEE T), pp 1–5

Mayr H (2015) Teaching better requirements engineering using LEGO® serious Play™. In: Proceedings of ACM conference on innovation and technology in computer science education (ITiCSE ’15), pp 126–131

Mead NR, Hough ED (2006) Security requirements engineering for software systems: case studies in support of software engineering education. In: Proceedings of 19th conference on software engineering education and training, vol 2006, pp 149–156

Mead NR, Shoemaker D, Ingalsbe J (2009) Teaching security requirements engineering using SQUARE. In: Proceedings of fourth international workshop on requirements engineering education and training, pp 20–27

Memon RN, Ahmad R, Salim SS (2010) Problems in requirements engineering education. In: Proceedings of 8th international conference on frontiers of information (FIT ’10), 2010

Memon RN, Ahmad R, Salim SS (2013) A direction framework to address problems in requirements engineering education. Malays J Comput Sci 26(4):294–311

Memon RN, Salim SS, Ahmad R (2012) Analysis and classification of problems associated with requirements engineering education: towards an integrated view. Arab J Sci Eng 39(3):1923–1935

Memon RN, Salim SS, Ahmad R (2012) Identifying research gaps in requirements engineering education: an analysis of a conceptual model and survey results. In: IEEE conference on open systems, 2012

Merten T, Schäfer T, Bürsner S (2012) Using RE knowledge to assist automatically during requirement specification. In: Seventh IEEE international workshop on requirements engineering education and training (REET), pp 9–13

Mich L (2014) Teaching requirements analysis: a student project framework to bridge the gap between business analysis and software engineering. In: Proceedings of 8th international workshop on requirements engineering education and training (REET), vol 1217, pp 20–25

Minocha S, Petre M, Roberts D (2008) Using wikis to simulate distributed requirements development in a software engineering course. Int J Eng Educ 24(4):689–704

Mkpojiogu EOC, Hussain A (2017) Assessing students’ performance in software requirements engineering education using scoring rubrics. In: AIP conference proceedings, vol 1891

Mohamed AH (2010) Facilitating tacit-knowledge acquisition within requirements engineering. In: Proceedings of the 10th WSEAS international conference on applied computer science, ACS’10, Stevens Point, Wisconsin, USA, 2010. World Scientific and Engineering Academy and Society (WSEAS), pp 27–32

Mohan S, Chenoweth S (2011) Teaching requirements engineering to undergraduate students. In: Proceedings of 42nd ACM technical symposium on computer science (SIGCSE ’11), pp 141–146

Moody DL, Sindre G (2003) Incorporating quality assurance processes into requirements analysis education. In: Proceedings of the 8th annual conference on innovation and technology in computer science education, vol 8, pp 74–78

Morales-Ramirez I, Alva-Martinez LH (2018) Requirements analysis skills: how to train practitioners? In; IEEE 8th international workshop on requirements engineering education and training (REET), pp 24–29

Moreira F, Ferreira MJ (2016) Teaching and learning requirements engineering based on mobile devices and cloud: a case study. In: Blended learning: concepts, methodologies, tools, and applications, vol 4. IGI Global, pp 1190–1217

Nakamura T, Kai U, Tachikawa Y (2015) Requirements engineering education using expert system and role-play training. In: Proceedings of IEEE international conference on teaching, assessment and learning for engineering (TALE), pp 375–382

Nakatani T (2008) Requirements engineering education for professional engineers. Front Artif Intell Appl 180(1):495–504

Nakatani T, Tsumaki T, Tamai T (2010) Instructional design of a requirements engineering education course for professional engineers. Smart Innov Syst Technol 3:119–151

Nakatani T, Tsumaki T, Tamai T (2010) Requirements engineering education for senior engineers: course design and its evaluation. In: Proceedings of 5th international workshop on requirements engineering education and training, pp 26–35

Nguyen L, Armarego J, Swatman P (2005) Understanding requirements engineering process: a challenge for practice and education. In: 5th International business information management conference, vol 1

Nkamaura T, Tachikawa Y (2017) Requirements engineering education using role-play training. In: Proceedings of 2016 IEEE international conference on teaching, assessment, and learning for engineering (TALE), pp 231–238

Noel R, Munoz R, Becerra C, Villarroel R (2017) Developing competencies for software requirements analysis through project based learning. In: Proceedings of 35th international conference of the Chilean Computer Science Society (SCCC)

Ochodek M, Kopczyńska S (2018) Perceived importance of agile requirements engineering practices—a survey. J Syst Softw 143:29–43

Ogata S, Matsuura S (2012) Training of requirements analysis modeling with UML-based prototype generation tool. In; 5th India software engineering conference, pp 105–108

Ott D, Raschke A (2012) Review improvement by requirements classification at Mercedes-Benz: limits of empirical studies in educational environments. In: Second IEEE international workshop on empirical requirements engineering (EmpiRE), pp 1–8

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A (2015) Requirements engineering education: a systematic mapping study. Requir Eng 20(2):119–138

Paschoal LN, de Oliveira MM, Chicon PMM (2018) A chatterbot sensitive to student’s context to help on software engineering education. In: 2018 XLIV Latin American computer conference (CLEI), pp 839–848

Penzenstadler B, Mahaux M, Heymans P (2013) University meets industry: calling in real stakeholders. In: 2013 26th International conference on software engineering education and training (CSEE T), May 2013, pp 1–10

Penzenstadler B, Richardson D, Karlin B, Cook A, Callele D, Wnuk K (2014) Using non-profit partners to engage students in RE. In: Proceedings of 8th international workshop on requirements engineering education and training (REET 2014), vol 1217, pp 1–10

Periyasamy K, Qin X, He D (2011) A requirements editor for teaching requirements engineering. REET workshop, 2011

Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: 12th International conference on evaluation and assessment in software engineering (EASE) 12, pp 1–10

Petersen K, Gencel C (2013) Worldviews, research methods, and their relationship to validity in empirical software engineering research. In: 2013 Joint conference of the 23rd international workshop on software measurement and the 8th international conference on software process and product measurement. IEEE, pp 81–89

Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. Inf Softw Technol 64:1–18

Pfleeger SL, Pfleeger CP (2009) Harmonizing privacy with security principles and practices. IBM J Res Dev 53(2):6:1–6:12

Prihartini N, Soemitro HL, Hendradjaya B (2017) Identifying aspects of web e-learning in LMS-based for requirement engineering process modeling. In: International conference on data and software engineering (ICoDSE), 2017

Quintanilla Portugal RL, Engiel P, Pivatelli J, Do Prado Leite JCS (2016) Facing the challenges of teaching requirements engineering. In: Proceedings of 38th international conference on software engineering companion (ICSE ’16), pp 461–470

Regev G, Gause DC, Wegmann A (2008) Requirements engineering education in the 21st century, an experiential learning approach. In: 2008 16th IEEE international requirements engineering conference, pp 85–94

Regev G, Gause DC, Wegmann A (2008) Requirements engineering education in the 21st century, an experiential learning approach. In: 16th IEEE international requirements engineering conference, pp 85–94

Rempel P, Mäder P (2015) A quality model for the systematic assessment of requirements traceability. In: 2015 IEEE 23rd international requirements engineering conference (RE), pp 176–185

Romero M, Vizcaíno A, Piattini M (2008) Developing the skills needed for requirement elicitation in global software development. In: Proceedings of the tenth international conference on enterprise information systems, vol DISI, pp 393–396

Romero M, Vizcaíno A, Piattini M (2008) A simulator for education and training in global requirements engineering: a work in progress. In: Proceedings of eighth IEEE international conference on advanced learning technologies, pp 123–125

Romero M, Vizcaíno A, Piattini M (2008) Toward a definition of the competences for global requirements elicitation. In: 13th Annual conference on innovation and technology in computer science education, p 364

Romero M, Vizcaíno A, Piattini M (2008) Towards the definition of a multi-agent simulation environment for education and training in global requirements elicitation. In: Proceedings of 2008 conference on human system interactions, pp 48–53

Rosca D (2003) Developing teamwork and communication skills in a multidisciplinary experiment. In: Proceedings of 33rd annual frontiers in education conference, vol 3, pp S4C14–S4C17

Rosca D (2000) Active/collaborative approach in teaching requirements engineering. In: Proceedings of 30th annual frontiers in education conference, vol 1, pp T2C-9–T2C-12

Rupakheti CR, Hays M, Mohan S, Chenoweth S, Stouder A (2017) On a pursuit for perfecting an undergraduate requirements engineering course. In: Proceedings of IEEE 30th conference on software engineering education and training (CSEE&T), vol 2017-January, pp 97–106

Rusu A, Russell R, Cocco R (2011) Simulating the software engineering interview process using a decision-based serious computer game. In: Proceedings of 16th international conference on computer games (CGAMES), pp 235–239

Scepanovic S, Beus-Dukic L (2015) Teaching requirements engineering: EUROWEB experience. In: Proceedings of European conference on software architecture (ECSAW ’15), vol. 07-11-September-2015

Sedelmaier Y, Landes D (2014) A multi-level didactical approach to build up competencies in requirements engineering. In: Proceedings of 8th international workshop on requirements engineering education and training (REET 2014), vol 1217, pp 26–34

Sedelmaier Y, Landes D (2014) Using business process models to foster competencies in requirements engineering. In: Proceedings of IEEE 27th conference on software engineering education and training (CSEE&T), pp 13–22

Sedelmaier Y, Landes D (2017) Experiences in teaching and learning requirements engineering on a sound didactical basis. In: Proceedings of ACM conference on innovation and technology in computer science education (ITiCSE ’17), vol Part F128680, pp 116–121

Sedelmaier Y, Landes D (2018) Systematic evolution of a learning setting for requirements engineering education based on competence-oriented didactics. In: Proceedings of IEEE global engineering education conference (EDUCON), vol. 2018-April, pp 1062–1070

Shaw M (2000) Software engineering education: a roadmap. In: Proceedings of the conference on the future of software engineering, ICSE ’00, New York, NY, USA, 2000. Association for Computing Machinery, pp 371–380

Shepperd M, Ajienka N, Counsell S (2018) The role and value of replication in empirical software engineering results. Inf Softw Technol 99:120–132

Shuto M, Washizaki H, Kakehi K, Fukazawa Y, Yamato S, Okubo M, Tenbergen B (2017) Relationship between the five factor model personality and learning effectiveness of teams in three information systems education courses. In: 2017 18th IEEE/ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD), pp 167–174

Shuto M, Washizaki H, Fukazawa Y, Yamato S, Okubo M, Tenbergen B (2018) Personality and learning effectiveness of teams in information systems education courses. EAI Endorsed Trans e-Learn 5(17):1–9

Sikkel K, Daneva M (2010) Teaching consistency of UML specifications. In: 5th International workshop on requirements engineering education and training, pp 17–19

Sikkel K, Daneva M (2011) Getting the client into the loop in information system modelling courses. In: Proceedings of 6th international workshop on requirements engineering education and training (REET), pp 1–4

Sindre G (2005) Teaching oral communication techniques in RE by student-student role play: initial experiences. In: 18th Conference on software engineering education and training (CSEET’05), pp 85–94

Smith R, Gotel O (2007) Re-o-poly: a game to introduce lightweight requirements engineering good practices

Soo MT, Aris H (2019) Game-based learning in requirements engineering: an overview. In: Proceedings of 2018 IEEE conference on e-Learning, e-Management and e-Services (IC3e), pp 46–51

Souza AF, Ferreira B, Valentim N, Conte T (2018) An experience report on teaching multiple design thinking techniques to software engineering students. In: XXXII Brazilian symposium on software engineering, pp 220–229

Spichkova M (2019) Industry-oriented project-based learning of software engineering. In: 2019 24th International conference on engineering of complex computer systems (ICECCS), pp 51–60

Suri D (2002) Introducing requirements engineering in an undergraduate engineering curriculum: lessons learnt. In: 2002 American Society for engineering education annual conference and exposition, pp 3175–3183

Suri D, Gassert J (2005) Gathering project requirements: a collaborative and interdisciplinary experience. In: Proceedings of American Society for engineering education annual conference and exposition, pp 6759–6764

Svahnberg M, Gorschek T, Borg A, Sandahl K, Eriksson M, Börster J, Loconsole A (2008) Perspectives on requirements understandability—for whom does the teacher’s bell toll? In: 2008 Requirements engineering education and training (REET)

Svensson RB, Regnell B (2017) Is role playing in Requirements Engineering Education increasing learning outcome? Requir Eng 22(4):475–489

Tachikawa Y, Nakamura T (2017) Education for requirements elicitation using group-work and role-play. In: Proceedings of 2017 IEEE global engineering education conference (EDUCON), pp 780–783

Takako N (2007) Improving engineering mind in eliciting requirements. In: REET workshop, 2007

Tenbergen B, Daun M (2019) Is requirements-engineering research delivering what it promised?: a review of its accomplishments and opportunities after 10 years. IEEE Softw 36(4):6–11

Tenbergen B, Daun M (2019) Industry projects in requirements engineering education: application in a university course in the US and comparison with Germany. In: Proceedings of Hawaii international conference on system sciences 2019, HICSS-53, 2019

Tiwari S, Ameta D, Singh P, Sureka A (2018) Teaching requirements engineering concepts using case-based learning. In: Proceedings of 2nd international workshop on software engineering education for millennials (SEEM’18), pp 8–15

Tiwari S (2020) Impact of CBL on student’s learning and performance: an experience report. In: Proceedings of 13th innovations in software engineering conference (ISEC 2020), ISEC 2020, New York, NY, USA, 2020. Association for Computing Machinery

Tomayko JE (1999) Forging a discipline: an outline history of software engineering education. Ann Softw Eng 6(1–4):3–18

Tort A, Olivé A, Pastor JA (2014) Former students’ views on the usefulness of conceptual modeling education. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 8697, pp 237–246

Tuya J, Garcia-Fanjul J (1999) Teaching requirements analysis by means of student collaboration. In: Proceedings—frontiers in education conference, vol 1, pp 11b4–11–11b4–15

Van Lamsweerde A (2009) Requirements engineering: from system goals to UML models to software, vol 10. Wiley, Chichester

Vega K, Fuks H, Carvalho G (2009) Training in requirements by collaboration: branching stories in second life. In: Proceedings of Simpósio Brasilerio de Sistemas Colaborativos (SBSC 2009), pp 116–122

Von Konsky BR, Robey M, Nair S (2004) Integrating design formalisms in software engineering education. In: Proceedings of 17th conference on software engineering education and training, vol 17, pp 78–83

Washizaki H, Sunaga Y, Shuto M, Kakehi K, Fukazawa Y, Yamato S, Okubo M, Tenbergen B (2017) Combinations of personal characteristic types and learning effectiveness of teams. In: 2017 IEEE 41st annual computer software and applications conference (COMPSAC), vol 1, pp 456–457

Wei B, Delugach HS, Colmenares E, Stringfellow C (2016) A conceptual graphs framework for teaching UML model-based requirements acquisition. In: Proceedings of IEEE 29th international conference on software engineering education and training (CSEET), pp 71–75

Westphal B (2018) An undergraduate requirements engineering curriculum with formal methods. In: Proceedings of IEEE 8th international workshop on requirements engineering education and training (REET), pp 1–10

Wever A, Maiden N (2011) What are the day-to-day factors that are preventing business analysts from effective business analysis? In: 2011 IEEE 19th international requirements engineering conference, pp 293–298

Weyer T, Daun M, Tenbergen B (2020) The changing world and the adapting machine—how digital transformation changes requirements engineering in the embedded and cyber-physical systems industry. IEEE Softw 38(5):83–91

Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng 11(1):102–107

Wohlin C, Regnell B (1999) Achieving industrial relevance in software engineering education. In: Proceedings 12th conference on software engineering education and training (Cat. No.PR00131), pp 16–25

Yasin A, Liu L, Li T, Wang J, Zowghi D (2018) Design and preliminary evaluation of a cyber Security Requirements Education Game (SREG). Inf Softw Technol 95:179–200

Zhang H, Babar MA, Tell P (2011) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637

Zowghi D (2009) Teaching requirements engineering to the Bahá’í students in Iran who are denied of higher education. In: Proceedings of fourth international workshop on requirements engineering education and training (REET ’09), pp 38–48

Zowghi D, Yusop N, Mehboob Z (2007) The role of conducting stakeholder meetings in requirements engineering training. In: International workshop on the requirements engineering education and training. REET workshop, 2007

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

paluno-The Ruhr Institute for Software Technology, University of Duisburg-Essen, 45127, Essen, Germany

Marian Daun & Viktoria Stenkova

Department of Computer Science, Smith College, Northampton, MA, 01063, USA

Alicia M. Grubb

Department of Computer Science, State University of New York, Oswego, NY, 13126, USA

Bastian Tenbergen

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Marian Daun .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Daun, M., Grubb, A.M., Stenkova, V. et al. A systematic literature review of requirements engineering education. Requirements Eng 28 , 145–175 (2023). https://doi.org/10.1007/s00766-022-00381-9

Download citation

Received : 21 May 2021

Accepted : 01 April 2022

Published : 19 May 2022

Issue Date : June 2023

DOI : https://doi.org/10.1007/s00766-022-00381-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Requirements engineering
• Requirements engineering education
• Systematic literature review
• Learning outcomes
• Find a journal
• Publish with us
• Track your research

• Advanced Search

Advances in automated support for requirements engineering: a systematic literature review

https://ror.org/0220mzb33Department of Informatics, King’s College London, WC2R 2LS, Strand, London, UK

https://ror.org/019apvn83Department of Computer Science, Ahmadu Bello University, Zaria, Nigeria

New Citation Alert added!

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

• Publisher Site

Requirements Engineering

Requirements Engineering (RE) has undergone several transitions over the years, from traditional methods to agile approaches emphasising increased automation. In many software development projects, requirements are expressed in natural language and embedded within large volumes of text documents. At the same time, RE activities aim to define software systems' functionalities and constraints. However, manually executing these tasks is time-consuming and prone to errors. Numerous research efforts have proposed tools and technologies for automating RE activities to address this challenge, which are documented in published works. This review aims to examine empirical evidence on automated RE and analyse its impact on the RE sub-domain and software development. To achieve our goal, we conducted a Systematic Literature Review (SLR) following established guidelines for conducting SLRs. We aimed to identify, aggregate, and analyse papers on automated RE published between 1996 and 2022. We outlined the output of the support tool, the RE phase covered, levels of automation, development approach, and evaluation approaches. We identified 85 papers that discussed automated RE from various perspectives and methodologies. The results of this review demonstrate the significance of automated RE for the software development community, which has the potential to shorten development cycles and reduce associated costs. The support tools primarily assist in generating UML models (44.7%) and other activities such as omission of steps, consistency checking, and requirement validation. The analysis phase of RE is the most widely automated phase, with 49.53% of automated tools developed for this purpose. Natural language processing technologies, particularly POS tagging and Parser, are widely employed in developing these support tools. Controlled experimental methods are the most frequently used (48.2%) for evaluating automated RE tools, while user studies are the least employed evaluation method (8.2%). This paper contributes to the existing body of knowledge by providing an updated overview of the research literature, enabling a better understanding of trends and state-of-the-art practices in automated RE for researchers and practitioners. It also paves the way for future research directions in automated requirements engineering.

Recommendations

A systematic literature review of pre-requirements specification traceability.

Requirements traceability (RT) is the ability to link requirements to other software development artifacts. In pre-requirements (pre-RS) traceability, requirements are linked to their origin, such as interviews with stakeholders, meeting protocols,...

The state of the art in automated requirements elicitation

Context: In large software development projects a huge number of unstructured text documents from various stakeholders becomes available and needs to be analyzed and transformed into structured requirements. This elicitation process is known to be time-...

Applications of ontologies in requirements engineering: a systematic review of the literature

There is an increase use of ontology-driven approaches to support requirements engineering (RE) activities, such as elicitation, analysis, specification, validation and management of requirements. However, the RE community still lacks a comprehensive ...

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

• Information
• Contributors

Published in

© Crown 2024

In-Cooperation

Springer-Verlag

Berlin, Heidelberg

Publication History

• Published: 3 February 2024
• Accepted: 25 November 2023
• Received: 28 July 2023

Author Tags

• Requirements engineering
• Automated RE
• Systematic literature review
• research-article

Funding Sources

Other metrics.

• Bibliometrics
• Citations 0

Article Metrics

• 0 Total Citations View Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0

This publication has not been cited yet

Digital Edition

View this article in digital edition.

Share this Publication link

https://dl.acm.org/doi/abs/10.1007/s00766-023-00411-0

Share on Social Media

• 0 References

Export Citations

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

• Corpus ID: 270619474

A Systematic Literature Review on the Use of Machine Learning in Software Engineering

• Nyaga Fred , I. O. Temkin
• Published 19 June 2024
• Computer Science

Figures and Tables from this paper

83 References

A survey on large language models for code generation, insights into deep learning refactoring: bridging the gap between practices and expectations, a survey of deep learning based software refactoring, unraveling code clone dynamics in deep learning frameworks, towards understanding the impact of code modifications on software quality metrics, a survey of source code representations for machine learning-based cybersecurity tasks, a systematic literature review on explainability for machine/deep learning-based software engineering research, a comparative analysis of large language models for code documentation generation, ai in software engineering: a survey on project management applications, codegen2: lessons for training llms on programming and natural languages, related papers.

Showing 1 through 3 of 0 Related Papers

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

How-to conduct a systematic literature review: A quick guide for computer science research

Angela carrera-rivera.

a Faculty of Engineering, Mondragon University

William Ochoa

Felix larrinaga.

b Design Innovation Center(DBZ), Mondragon University

Associated Data

• No data was used for the research described in the article.

Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:

• • Clearly defined strategies to follow for a systematic literature review in computer science research, and
• • Algorithmic method to tackle a systematic literature review.

Graphical abstract

Specifications table

Method details

A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.

Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.

Define PICOC and synonyms

The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.

Planning Step 1 “Defining PICOC keywords and synonyms”.

Formulate research questions

Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).

Research questions examples.

Select digital library sources

The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.

Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.

Define inclusion and exclusion criteria

Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.

Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.

Define the Quality Assessment (QA) checklist

Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.

Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.

Define the “Data Extraction” form

The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .

Planning Step 6 “Define data extraction form”. Examples of fields.

The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.

After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.

Build digital library search strings

Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:

(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”

Gather studies

Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.

Example of Advanced search on Scopus.

In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.

Study Selection and Refinement

The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.

In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.

Performing quality assessment (QA) in Parsif.al.

Data extraction

Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.

Example of data extraction form using Parsif.al.

The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.

Analysis and Report

The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.

The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.

Techniques for bibliometric analysis and examples.

Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple  data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.

[1] Keyword co-relationship analysis using clusterization in vos viewer.

This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.

Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.

The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].

SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.

CRediT author statement

Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.

Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.

1 https://parsif.al/

Data Availability

Beyond the monolith: A systematic review of the literature on Latiné/x/a/o students in engineering using a liberative approach

Usage metrics.

• Engineering practice and education
• Engineering

A Systematic Literature Review on the Use of Machine Learning in Software Engineering

• Fred, Nyaga
• Temkin, I. O.

Software engineering (SE) is a dynamic field that involves multiple phases all of which are necessary to develop sustainable software systems. Machine learning (ML), a branch of artificial intelligence (AI), has drawn a lot of attention in recent years thanks to its ability to analyze massive volumes of data and extract useful patterns from data. Several studies have focused on examining, categorising, and assessing the application of ML in SE processes. We conducted a literature review on primary studies to address this gap. The study was carried out following the objective and the research questions to explore the current state of the art in applying machine learning techniques in software engineering processes. The review identifies the key areas within software engineering where ML has been applied, including software quality assurance, software maintenance, software comprehension, and software documentation. It also highlights the specific ML techniques that have been leveraged in these domains, such as supervised learning, unsupervised learning, and deep learning. Keywords: machine learning, deep learning, software engineering, natural language processing, source code

• Computer Science - Software Engineering;
• Computer Science - Machine Learning

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Author Biographies
• Google Scholar
• on Google Scholar

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

Article Versions Notes

Tonni, G.; Palicelli, A.; Bassi, M.C.; Torricelli, F.; Vacca, I.; Aguzzoli, L.; Mandato, V.D. Gastrointestinal Stromal Tumors (GISTs) Mimicking Primary Ovarian Tumors or Metastasizing to the Ovaries: A Systematic Literature Review. Cancers 2024 , 16 , 2305. https://doi.org/10.3390/cancers16132305

Tonni G, Palicelli A, Bassi MC, Torricelli F, Vacca I, Aguzzoli L, Mandato VD. Gastrointestinal Stromal Tumors (GISTs) Mimicking Primary Ovarian Tumors or Metastasizing to the Ovaries: A Systematic Literature Review. Cancers . 2024; 16(13):2305. https://doi.org/10.3390/cancers16132305

Tonni, Gabriele, Andrea Palicelli, Maria Chiara Bassi, Federica Torricelli, Ilaria Vacca, Lorenzo Aguzzoli, and Vincenzo Dario Mandato. 2024. "Gastrointestinal Stromal Tumors (GISTs) Mimicking Primary Ovarian Tumors or Metastasizing to the Ovaries: A Systematic Literature Review" Cancers 16, no. 13: 2305. https://doi.org/10.3390/cancers16132305

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals