latest research papers on software quality assurance

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Development of a maturity model for software quality assurance practices, 1. introduction, paper organization, 2. related work, 2.1. software quality assurance overview, 2.2. maturity model’s overview, 2.2.1. business process maturity model (bpmm), 2.2.2. capability maturity model (cmm), 2.2.3. information quality management maturity model (iqm3), 2.2.4. complex product systems (cops) maturity model, 2.2.5. organizational change readiness maturity model (ocrmm), 2.2.6. service systems maturity model, 3. research methodology, 3.1. hierarchal decision modeling (hdm).

• With the definition of HDM, we established the model perspectives and factors for assessing SQA practices’ maturity. The objective of this research is to develop and maturity model to assess SQA practices. The perspectives identified are the testing perspective, requirements validation perspective, technology perspective, software quality management control perspective, and organization and culture perspective with 25 factors under these perspectives.
• Create a survey using Qualtrics, first for the validation phase then for the quantification phase to collect the inputs from the experts.
• Use the Pairwise Comparison tool to determine the weights for perspectives and factors from the survey responses.
• Considering the relative importance of the various perspectives and factors in determining SQA. The perspectives and factors with the highest weights are viewed as being the most important to SQA. Perspectives and factors are ranked against one another to determine which would be preferred based on experts’ points of view and their responses.
• Develop the desirability curves to understand the dynamics of each factor that shows the different levels where companies can fall into. Desirability curves are used to calculate the final maturity score.
• Then, we apply the model to assess the maturity of the two well-established telecommunication companies and calculate the maturity levels of each case. Using the factors’ weights and the company’s weights in the desirability curves, the final maturity score is calculated.

3.2. Desirability Curves

3.3. maturity score, 3.4. inconsistency and disagreement, 3.5. expert panel selection and formation, 3.6. expert panel formation, 4. research model development and results.

• Testing Perspective
• Requirements validation Perspective
• Technology Perspective
• Software quality management control Perspective
• Organization and Culture Perspective

4.1. Proposed Maturity Model

4.2. result: validation and quantification, 4.2.1. validation phase, 4.2.2. quantification phase.

Click here to enlarge figure

5. Case Studies

5.1. telecommunication industry in saudi arabia, 5.2. results of the case studies, 6. discussion, 7. conclusions, author contributions, data availability statement, acknowledgments, conflicts of interest.

• Zhao, Y.; Hu, Y.; Gong, J. Research on International Standardization of Software Quality and Software Testing. In Proceedings of the 2021 IEEE/ACIS 20th International Fall Conference on Computer and Information Science (ICIS Fall), Xi’an, China, 13–15 October 2021; pp. 56–62. [ Google Scholar ] [ CrossRef ]
• Wong, W.Y.; Hai Sam, T.; Too, C.W.; Fong Pok, W. Software Quality Assurance Plan: Setting Quality Assurance Checkpoints within the Project Life Cycle and System Development Life Cycle. In Proceedings of the 2022 IEEE 18th International Colloquium on Signal Processing & Applications (CSPA), Selangor, Malaysia, 12 May 2022; pp. 214–219. [ Google Scholar ] [ CrossRef ]
• Ibarra, S.; Munoz, M. Support tool for software quality assurance in software development. In Proceedings of the 2018 7th International Conference On Software Process Improvement (CIMPS), Guadalajara, Mexico, 17–19 October 2018; pp. 13–19. [ Google Scholar ] [ CrossRef ]
• Atoum, I.; Baklizi, M.K.; Alsmadi, I.; Otoom, A.A.; Alhersh, T.; Ababneh, J.; Almalki, J.; Alshahrani, S.M. Challenges of Software Requirements Quality Assurance and Validation: A Systematic Literature Review. IEEE Access 2021 , 9 , 137613–137634. [ Google Scholar ] [ CrossRef ]
• Poth, A.; Meyer, B.; Schlicht, P.; Riel, A. Quality Assurance for Machine Learning—An approach to function and system safeguarding. In Proceedings of the 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS), Macau, China, 11–14 December 2020; pp. 22–29. [ Google Scholar ] [ CrossRef ]
• Poth, A.; Kottke, M.; Riel, A. Evaluation of Agile Team Work Quality. In Agile Processes in Software Engineering and Extreme Programming–Workshops ; Paasivaara, M., Kruchten, P., Eds.; Lecture Notes in Business Information Processing; Springer International Publishing: Cham, Switzerland, 2020; Volume 396, pp. 101–110. [ Google Scholar ] [ CrossRef ]
• Sabev, P.; Grigorova, K. A Survey on State of Software Quality Assurance in Bulgaria. In Proceedings of the 20th International Conference on Computer Systems and Technologies, Ruse, Bulgaria, 21–22 June 2019; pp. 124–130. [ Google Scholar ] [ CrossRef ]
• Saleem, G.; Azam, F.; Younus, M.U.; Ahmed, N.; Li, Y. Quality assurance of web services: A systematic literature review. In Proceedings of the 2016 2nd IEEE International Conference on Computer and Communications (ICCC), Chengdu, China, 14–17 October 2016; pp. 1391–1396. [ Google Scholar ] [ CrossRef ]
• Kettunen, P. Bringing Total Quality in to Software Teams: A Frame for Higher Performance. In Lean Enterprise Software and Systems ; Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J., Eds.; Lecture Notes in Business Information Processing; Springer: Berlin/Heidelberg, Germany, 2013; Volume 167, pp. 48–64. [ Google Scholar ] [ CrossRef ]
• López, L.; Burgués, X.; Martínez-Fernández, S.; Vollmer, A.M.; Behutiye, W.; Karhapää, P.; Franch, X.; Rodríguez, P.; Oivo, M. Quality measurement in agile and rapid software development: A systematic mapping. J. Syst. Softw. 2022 , 186 , 111187. [ Google Scholar ] [ CrossRef ]
• Sophocleous, R.; Kapitsaki, G.M. Examining the Current State of System Testing Methodologies in Quality Assurance. In Agile Processes in Software Engineering and Extreme Programming ; Stray, V., Hoda, R., Paasivaara, M., Kruchten, P., Eds.; Lecture Notes in Business Information Processing; Springer International Publishing: Cham, Switzerland, 2020; Volume 383, pp. 240–249. [ Google Scholar ] [ CrossRef ]
• Reine De Reanzi, S.; Ranjit Jeba Thangaiah, P. A survey on software test automation return on investment, in organizations predominantly from Bengaluru, India. Int. J. Eng. Bus. Manag. 2021 , 13 , 184797902110620. [ Google Scholar ] [ CrossRef ]
• Tran, H.K.V.; Unterkalmsteiner, M.; Börstler, J.; Ali, N. bin Assessing test artifact quality—A tertiary study. Inf. Softw. Technol. 2021 , 139 , 106620. [ Google Scholar ] [ CrossRef ]
• Shen, P.; Ding, X.; Ren, W.; Yang, C. Research on Software Quality Assurance Based on Software Quality Standards and Technology Management. In Proceedings of the 2018 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), Busan, Republic of Korea, 27–29 June 2018; pp. 385–390. [ Google Scholar ] [ CrossRef ]
• Reddy, M.P.; Reddy, K.L.R. Policies, Processes, Procedures and Measurement in Software Quality Assurance: A State of Art Survey. Int. J. Innov. Sci. Eng. Technol. 2017 , 4 , 8. [ Google Scholar ]
• Arcos-Medina, G.; Mauricio, D. Aspects of software quality applied to the process of agile software development: A systematic literature review. Int. J. Syst. Assur. Eng. Manag. 2019 , 10 , 867–897. [ Google Scholar ] [ CrossRef ]
• Sun, L.; Nazir, S.; Hussain, A. Multicriteria Decision Making to Continuous Software Improvement Based on Quality Management, Assurance, and Metrics. Sci. Program. 2021 , 2021 , 9953618. [ Google Scholar ] [ CrossRef ]
• Gonen, B.; Sawant, D. Significance of Agile Software Development and SQA Powered by Automation. In Proceedings of the 2020 3rd International Conference on Information and Computer Technologies (ICICT), San Jose, CA, USA, 9–12 March 2020; pp. 7–11. [ Google Scholar ] [ CrossRef ]
• Mishra, A.; Otaiwi, Z. DevOps and software quality: A systematic mapping. Comput. Sci. Rev. 2020 , 38 , 100308. [ Google Scholar ] [ CrossRef ]
• Panichella, A. Beyond Unit-Testing in Search-Based Test Case Generation: Challenges and Opportunities. In Proceedings of the 2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST), Motreal, QC, Canada, 26–27 May 2019; pp. 7–8. [ Google Scholar ] [ CrossRef ]
• Backlund, F.; Chronéer, D.; Sundqvist, E. Project Management Maturity Models—A Critical Review. Procedia-Soc. Behav. Sci. 2014 , 119 , 837–846. [ Google Scholar ] [ CrossRef ]
• Lee, J.; Lee, D.; Kang, S. An Overview of the Business Process Maturity Model (BPMM). In Advances in Web and Network Technologies, and Information Management ; Chang, K.C.-C., Wang, W., Chen, L., Ellis, C.A., Hsu, C.-H., Tsoi, A.C., Wang, H., Eds.; In Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2007; pp. 384–395. [ Google Scholar ] [ CrossRef ]
• Tarhan, A.; Turetken, O.; Reijers, H.A. Business process maturity models: A systematic literature review. Inf. Softw. Technol. 2016 , 75 , 122–134. [ Google Scholar ] [ CrossRef ]
• Strutt, J.E.; Sharp, J.V.; Terry, E.; Miles, R. Capability maturity models for offshore organisational management. Environ. Int. 2006 , 32 , 1094–1105. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Paulk, M. Capability Maturity Model for Software. In Encyclopedia of Software Engineering ; Marciniak, J.J., Ed.; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2002; p. sof589. [ Google Scholar ] [ CrossRef ]
• Caballero, I.; Caro, A.; Calero, C.; Piattini, M. IQM3: Information Quality Management Maturity Model. J. Univers. Comput. Sci. 2008 , 14 , 29. [ Google Scholar ] [ CrossRef ]
• Kim, S.; Pérez-Castillo, R.; Caballero, I.; Lee, D. Organizational process maturity model for IoT data quality management. J. Ind. Inf. Integr. 2022 , 26 , 100256. [ Google Scholar ] [ CrossRef ]
• Yeo, K.T.; Ren, Y. Risk management capability maturity model for complex product systems (CoPS) projects. Syst. Eng. 2009 , 12 , 275–294. [ Google Scholar ] [ CrossRef ]
• Zephir, O.; Minel, S.; Chapotot, E. A maturity model to assess organisational readiness for change. Int. J. Technol. Manag. 2011 , 55 , 286. [ Google Scholar ] [ CrossRef ]
• Skelsey, D.; King, D.; Sidhu, R.; Smith, R.; Perkins, C.; Change Management Institute; APMG International. The Effective Change Manager: The Change Management Body of Knowledge ; Kogan Page: London, UK, 2014. [ Google Scholar ]
• Neff, A.A.; Hamel, F.; Herz, T.P.; Uebernickel, F.; Brenner, W.; vom Brocke, J. Developing a maturity model for service systems in heavy equipment manufacturing enterprises. Inf. Manage. 2014 , 51 , 895–911. [ Google Scholar ] [ CrossRef ]
• Rapaccini, M.; Saccani, N.; Pezzotta, G.; Burger, T.; Ganz, W. Service development in product-service systems: A maturity model. Serv. Ind. J. 2013 , 33 , 300–319. [ Google Scholar ] [ CrossRef ]
• Alzahrani, S.; Daim, T.U. Evaluation of the Cryptocurrency Adoption Decision Using Hierarchical Decision Modeling (HDM). In Proceedings of the 2019 Portland International Conference on Management of Engineering and Technology (PICMET), Portland, OR, USA, 25–29 August 2019; pp. 1–7. [ Google Scholar ] [ CrossRef ]
• Lavoie, J.R.; Daim, T. Towards the assessment of technology transfer capabilities: An action research-enhanced HDM model. Technol. Soc. 2020 , 60 , 101217. [ Google Scholar ] [ CrossRef ]
• van Blommestein, K.C.; Daim, T.U. Residential energy efficient device adoption in South Africa. Sustain. Energy Technol. Assess. 2013 , 1 , 13–27. [ Google Scholar ] [ CrossRef ]
• Cleland, D.I.; Kocaoglu, D.F. Engineering Management ; McGraw-Hill series in industrial engineering and management science; McGraw-Hill: New York, NY, USA, 1981. [ Google Scholar ]
• Kocaoglu, D.F. A participative approach to program evaluation. IEEE Trans. Eng. Manag. 1983 , EM-30 , 112–118. [ Google Scholar ] [ CrossRef ]
• Turan, T.; Amer, M.; Tibbot, P.; Almasri, M.; Fayez, F.A.; Graham, S. Use of Hierarchal Decision Modeling (HDM) for selection of graduate school for master of science degree program in engineering. In Proceedings of the PICMET ’09-2009 Portland International Conference on Management of Engineering Technology, Portland, OR, USA, 2–6 August 2009; pp. 535–549. [ Google Scholar ] [ CrossRef ]
• Barham, H. Development of a Readiness Assessment Model for Evaluating Big Data Projects: Case Study of Smart City in Oregon, USA. Ph.D. Thesis, Portland State University, Portland, OR, USA, 2019. [ Google Scholar ] [ CrossRef ]
• Barham, H.; Daim, T.U. The use of readiness assessment for big data projects. Sustain. Cities Soc. 2020 , 60 , 102233. [ Google Scholar ] [ CrossRef ]
• Hogaboam, L.; Ragel, B.; Daim, T. Development of a Hierarchical Decision Model (HDM) for health technology assessment (HTA) to design and implement a new patient care database for low back pain. In Proceedings of the PICMET ’14 Conference: Portland International Center for Management of Engineering and Technology, Infrastructure and Service Integration, Kanazawa, Japan, 27–31 July 2014; pp. 3511–3517. [ Google Scholar ]
• Alanazi, H.A.; Daim, T.U.; Kocaoglu, D.F. Identify the best alternatives to help the diffusion of teleconsultation by using the Hierarchical Decision Model (HDM). In Proceedings of the 2015 Portland International Conference on Management of Engineering and Technology (PICMET), Portland, OR, USA, 2–6 August 2015; pp. 422–432. [ Google Scholar ] [ CrossRef ]
• Phan, K. Innovation Measurement: A Decision Framework to Determine Innovativeness of a Company. Ph.D. Thesis, Portland State University, Portland, OR, USA, 2013. [ Google Scholar ] [ CrossRef ]
• Alzahrani, S.; Daim, T.U. Technology Adoption: Case of Cryptocurrency. In Recent Developments in Individual and Organizational Adoption of ICTs ; Yildiz, O., Ed.; IGI Global: Hershey, PA, USA, 2021; pp. 96–119. [ Google Scholar ] [ CrossRef ]
• Lavoie, J. A Scoring Model to Assess Organizations’ Technology Transfer Capabilities: The Case of a Power Utility in the Northwest USA. Ph.D. Thesis, Portland State University, Portland, OR, USA, 2019. [ Google Scholar ] [ CrossRef ]
• Estep, J. Development of a Technology Transfer Score for Evaluating Research Proposals: Case Study of Demand Response Technologies in the Pacific Northwest. Ph.D. Thesis, Portland State University, Portland, OR, USA, 2017. Available online: https://pdxscholar.library.pdx.edu/open_access_etds/3479 (accessed on 18 December 2022).
• Gibson, E. A Measurement System for Science and Engineering Research Center Performance Evaluation. In Proceedings of the 2016 Portland International Conference on Management of Engineering and Technology (PICMET), Honolulu, HI, USA, 4–8 September 2016. [ Google Scholar ] [ CrossRef ]
• Abotah, R. Evaluation of Energy Policy Instruments for the Adoption of Renewable Energy: Case of Wind Energy in the Pacific Northwest U.S. Ph.D. Thesis, Portland State University, Portland, OR, USA, 2014. [ Google Scholar ] [ CrossRef ]
• Estep, J.; Daim, T. A framework for technology transfer potential assessment. In Proceedings of the 2016 Portland International Conference on Management of Engineering and Technology (PICMET), Honolulu, HI, USA, 4–8 September 2016; pp. 2846–2852. [ Google Scholar ] [ CrossRef ]
• Chen, H.; Kocaoglu, D.F. A sensitivity analysis algorithm for hierarchical decision models. Eur. J. Oper. Res. 2008 , 185 , 266–288. [ Google Scholar ] [ CrossRef ]
• Sabev, P.; Grigorova, K. A Comparative Study of GUI Automated Tools for Software Testing. In Proceedings of the Third International Conference on Advances and Trends in Software Engineering, Lisbon, Portugal, 21–25 February 2016; p. 9. [ Google Scholar ]
• Felderer, M.; Ramler, R. Quality Assurance for AI-Based Systems: Overview and Challenges (Introduction to Interactive Session). In Software Quality: Future Perspectives on Software Engineering Quality ; Winkler, D., Biffl, S., Mendez, D., Wimmer, M., Bergsmann, J., Eds.; Lecture Notes in Business Information Processing; Springer International Publishing: Cham, Switzerland, 2021; Volume 404, pp. 33–42. [ Google Scholar ] [ CrossRef ]
• Tao, C.; Gao, J.; Wang, T. Testing and Quality Validation for AI Software–Perspectives, Issues, and Practices. IEEE Access 2019 , 7 , 120164–120175. [ Google Scholar ] [ CrossRef ]
• Ji, S.; Li, Q.; Cao, W.; Zhang, P.; Muccini, H. Quality Assurance Technologies of Big Data Applications: A Systematic Literature Review. Appl. Sci. 2020 , 10 , 8052. [ Google Scholar ] [ CrossRef ]
• Rashidi, H.; Sadeghzadeh Hemayati, M. Software Quality Models: A Comprehensive Review and Analysis. J. Electr. Comput. Eng. Innov. 2018 , 6 , 59–76. [ Google Scholar ] [ CrossRef ]
• Rashid, J.; Nisar, M.W. How to Improve a Software Quality Assurance in Software Development—A Survey. Int. J. Comput. Sci. Inf. Secur. 2016 , 14 , 11. [ Google Scholar ]
• Lee, M.-C. Software Quality Factors and Software Quality Metrics to Enhance Software Quality Assurance. Br. J. Appl. Sci. Technol. 2014 , 4 , 3069–3095. [ Google Scholar ] [ CrossRef ]
• Gan, M.; Yucel, Z.; Monden, A. Improvement and Evaluation of Data Consistency Metric CIL for Software Engineering Data Sets. IEEE Access 2022 , 10 , 70053–70067. [ Google Scholar ] [ CrossRef ]
• Thota, M.K.; Shajin, F.H.; Rajesh, P. Survey on software defect prediction techniques. Int. J. Appl. Sci. Eng. 2020 , 17 , 331–344. [ Google Scholar ] [ CrossRef ]
• Jaskolka, J.; Hamid, B.; Kokaly, S. Software Design Trends Supporting Multiconcern Assurance. IEEE Softw. 2022 , 39 , 22–26. [ Google Scholar ] [ CrossRef ]
• Nazir, M. Software Quality Assurance and Android Application Development: A Comparison among Traditional and Agile Methodology. Res. J. Comput. Sci. Inf. Technol. 2020 , 4 , 1–29. [ Google Scholar ] [ CrossRef ]
• Slaughter, A.E.; Permann, C.J.; Miller, J.M.; Alger, B.K.; Novascone, S.R. Continuous Integration, In-Code Documentation, and Automation for Nuclear Quality Assurance Conformance. Nucl. Technol. 2021 , 207 , 923–930. [ Google Scholar ] [ CrossRef ]
• Khan, S.U.; Khan, A.W.; Khan, F.; Khan, M.A.; Whangbo, T.K. Critical Success Factors of Component-Based Software Outsourcing Development From Vendors’ Perspective: A Systematic Literature Review. IEEE Access 2022 , 10 , 1650–1658. [ Google Scholar ] [ CrossRef ]
• Huang, F.; Strigini, L. HEDF: A Method for Early Forecasting Software Defects Based on Human Error Mechanisms. IEEE Access 2023 , 11 , 3626–3652. [ Google Scholar ] [ CrossRef ]
• Lee, T.; Nam, J.; Han, D.; Kim, S.; Peter In, H. Developer Micro Interaction Metrics for Software Defect Prediction. IEEE Trans. Softw. Eng. 2016 , 42 , 1015–1035. [ Google Scholar ] [ CrossRef ]
• Tomar, A. The Survey of Metrices on Software Quality Assurance and Reuse. In Proceedings of the National Conference on Innovative Paradigms in Engineering & Technology (NCIPET-2013), Nagpur, India, 17 February 2013; p. 5. [ Google Scholar ]
• Heimicke, J.; Kaiser, S.; Albers, A. Agile product development: An analysis of acceptance and added value in practice. Procedia CIRP 2021 , 100 , 768–773. [ Google Scholar ] [ CrossRef ]
• Teah, T.-S.; Wong, W.-Y.; Beh, H.-C. The Practical Implication of Software Quality Assurance of Change Control Management: Why Overall IT Project Activities Matters? In Proceedings of the 2019 IEEE 7th Conference on Systems, Process and Control (ICSPC), Melaka, Malaysia, 13–14 December 2019; pp. 131–136. [ Google Scholar ] [ CrossRef ]

Share and Cite

Al MohamadSaleh, A.; Alzahrani, S. Development of a Maturity Model for Software Quality Assurance Practices. Systems 2023 , 11 , 464. https://doi.org/10.3390/systems11090464

Al MohamadSaleh A, Alzahrani S. Development of a Maturity Model for Software Quality Assurance Practices. Systems . 2023; 11(9):464. https://doi.org/10.3390/systems11090464

Al MohamadSaleh, Ahmad, and Saeed Alzahrani. 2023. "Development of a Maturity Model for Software Quality Assurance Practices" Systems 11, no. 9: 464. https://doi.org/10.3390/systems11090464

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

arXiv's Accessibility Forum starts next month!

Help | Advanced Search

Computer Science > Software Engineering

Title: quality assurance challenges for machine learning software applications during software development life cycle phases.

Abstract: In the past decades, the revolutionary advances of Machine Learning (ML) have shown a rapid adoption of ML models into software systems of diverse types. Such Machine Learning Software Applications (MLSAs) are gaining importance in our daily lives. As such, the Quality Assurance (QA) of MLSAs is of paramount importance. Several research efforts are dedicated to determining the specific challenges we can face while adopting ML models into software systems. However, we are aware of no research that offered a holistic view of the distribution of those ML quality assurance challenges across the various phases of software development life cycles (SDLC). This paper conducts an in-depth literature review of a large volume of research papers that focused on the quality assurance of ML models. We developed a taxonomy of MLSA quality assurance issues by mapping the various ML adoption challenges across different phases of SDLC. We provide recommendations and research opportunities to improve SDLC practices based on the taxonomy. This mapping can help prioritize quality assurance efforts of MLSAs where the adoption of ML models can be considered crucial.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

1 blog link

Dblp - cs bibliography, bibtex formatted citation.

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• DOI: 10.1002/0471722324
• Corpus ID: 216026878

Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement

• Published 1 February 2005
• Computer Science, Engineering

85 Citations

Techniques for quantitative analysis of software quality throughout the sdlc: the swebok guide coverage, a review report on software quality measurement and estimation, elevating software quality assurance to an art form: a deep dive into cutting-edge qa methodologies, precision testing, and the pursuit of flawless software systems, quality assurance frameworks: analyzing effectiveness in software development lifecycle, measuring software testing efficiency using two-way assessment technique, integration of agile software development and robust design methodology in optimization of software defect parameters, ensuring compliance and security: integrating quality assurance into software development, empirical evaluation of automated test suite generation and optimization, rules of software quality assurance to prevent and reduce software failures in medical devices: therac-25 case study.

• Highly Influenced
• 10 Excerpts

Reuse Strategy based on Quality Certification of Reusable Components

Related papers.

Showing 1 through 3 of 0 Related Papers

quality assurance Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Quality Assurance Information System-The Case of the TEI of Athens

Systematic assessment of data quality and quality assurance/quality control (qa/qc) of current research on microplastics in biosolids and agricultural soils, sistem penjaminan mutu internal (spmi).

Abstract: The purpose of this research is to look at the educational achievements of students through an internal quality assurance system and as a tool to achieve and maintain school progress. Research, with a quantative approach. The data obtained is obtained through interview techniques, observations, and library studies. The results of the study were analyzed by using data reduction, presentation of data and drawing conclusions. The findings of the meaning of the importance of SPMI are implemented in elementary school educational institutions. The study was conducted at one of SMAN 3 Wajo's schools. The results of this study show that: (1) SPMI which is carried out continuously contributes to the acquisition of superior accreditation ratings. (2) The SPMI cycle that is carried out in its entirety has guided the course of various tasks from school stakeholders. (3) Quality culture can be created through the implementation of SPMI.Keywords: Internal Quality Assurance System; Quality of SMAN 3 Wajo School

Sigma metrics in quality control- An innovative tool

The clinical laboratory in today’s world is a rapidly evolving field which faces a constant pressure to produce quick and reliable results. Sigma metric is a new tool which helps to reduce process variability, quantitate the approximate number of analytical errors, and evaluate and guide for better quality control (QC) practices.To analyze sigma metrics of 16 biochemistry analytes using ERBA XL 200 Biochemistry analyzer, interpret parameter performance, compare analyzer performance with other Middle East studies and modify existing QC practices.This study was undertaken at a clinical laboratory for a period of 12 months from January to December 2020 for the following analytes: albumin (ALB), alanine amino transferase (SGPT), aspartate amino transferase (SGOT), alkaline phosphatase (ALKP), bilirubin total (BIL T), bilirubin direct (BIL D), calcium (CAL), cholesterol (CHOL), creatinine (CREAT), gamma glutamyl transferase (GGT), glucose (GLUC), high density lipoprotein (HDL), triglyceride (TG), total protein (PROT), uric acid (UA) and urea. The Coefficient of variance (CV%) and Bias % were calculated from internal quality control (IQC) and external quality assurance scheme (EQAS) records respectively. Total allowable error (TEa) was obtained using guidelines Clinical Laboratories Improvement Act guidelines (CLIA). Sigma metrics was calculated using CV%, Bias% and TEa for the above parameters. It was found that 5 analytes in level 1 and 8 analytes in level 2 had greater than 6 sigma performance indicating world class quality. Cholesterol, glucose (level 1 and 2) and creatinine level 1 showed >4 sigma performance i.e acceptable performance. Urea (both levels) and GGT (level 1) showed <3 sigma and were therefore identified as the problem analytes. Sigma metrics helps to assess analytic methodologies and can serve as an important self assessment tool for quality assurance in the clinical laboratory. Sigma metric evaluation in this study helped to evaluate the quality of several analytes and also categorize them from high performing to problematic analytes, indicating the utility of this tool. In conclusion, parameters showing lesser than 3 sigma need strict monitoring and modification of quality control procedure with change in method if necessary.

Quality assurance for on‐table adaptive magnetic resonance guided radiation therapy: A software tool to complement secondary dose calculation and failure modes discovered in clinical routine

Editorial comment: factors impacting us-lirads visualization scores—optimizing future quality assurance and standards, the association of laryngeal position on videolaryngoscopy and time taken to intubate using spatial point pattern analysis of prospectively collected quality assurance data, the impact of policy changes, dedicated funding and implementation support on early intervention programs for psychosis.

Introduction Early intervention services for psychosis (EIS) are associated with improved clinical and economic outcomes. In Quebec, clinicians led the development of EIS from the late 1980s until 2017 when the provincial government announced EIS-specific funding, implementation support and provincial standards. This provides an interesting context to understand the impacts of policy commitments on EIS. Our primary objective was to describe the implementation of EIS three years after this increased political involvement. Methods This cross-sectional descriptive study was conducted in 2020 through a 161-question online survey, modeled after our team's earlier surveys, on the following themes: program characteristics, accessibility, program operations, clinical services, training/supervision, and quality assurance. Descriptive statistics were performed. When relevant, we compared data on programs founded before and after 2017. Results Twenty-eight of 33 existing EIS completed the survey. Between 2016 and 2020, the proportion of Quebec's population having access to EIS rose from 46% to 88%; >1,300 yearly admissions were reported by surveyed EIS, surpassing governments’ epidemiological estimates. Most programs set accessibility targets; adopted inclusive intake criteria and an open referral policy; engaged in education of referral sources. A wide range of biopsychosocial interventions and assertive outreach were offered by interdisciplinary teams. Administrative/organisational components were less widely implemented, such as clinical/administrative data collection, respecting recommended patient-to-case manager ratios and quality assurance. Conclusion Increased governmental implementation support including dedicated funding led to widespread implementation of good-quality, accessible EIS. Though some differences were found between programs founded before and after 2017, there was no overall discernible impact of year of implementation. Persisting challenges to collecting data may impede monitoring, data-informed decision-making, and quality improvement. Maintaining fidelity and meeting provincial standards may prove challenging as programs mature and adapt to their catchment area's specificities and as caseloads increase. Governmental incidence estimates may need recalculation considering recent epidemiological data.

Current Status of Quality Assurance Scheme in Selected Undergraduate Medical Colleges of Bangladesh

This descriptive cross sectional study was carried out to determine the current status of Quality Assurance Scheme in undergraduate medical colleges of Bangladesh. This study was carried out in eight (four Government and four Non- Government) medical colleges in Bangladesh over a period from July 2015 to June 2016. The present study had an interview schedule with open question for college authority and another interview schedule with open question for head of department of medical college. Study revealed that 87.5% of college had Quality Assurance Scheme (QAS) in their college, 75% of college authority had regular meeting of academic coordination committee in their college, 50% of college had active Medical Education Unit in their college, 87.5% of college authority said positively on publication of journal in their college. In the present study researchers interviewed 53 heads of department with open question about distribution, collection of personal review form, submission with recommendation to the academic co-coordinator, and annual review meeting of faculty development. The researchers revealed from the interviews that there is total absence of this practice which is directed in national guidelines and tools for Quality Assurance Scheme (QAS) for medical colleges of Bangladesh. Bangladesh Journal of Medical Education Vol.13(1) January 2022: 33-39

AN APPLICATION OF CADASTRAL FABRIC SYSTEM IN IMPROVING POSITIONAL ACCURACY OF CADASTRAL DATABASES IN MALAYSIA

Abstract. Cadastral fabric is perceived as a feasible solution to improve the speed, efficiency and quality of the cadastral measurement data to implement Positional Accuracy Improvement (PAI) and to support Coordinated Cadastral System (CCS) and Dynamic Coordinated Cadastral System (DCCS) in Malaysia. In light of this, this study aims to propose a system to upgrade the positional accuracy of the existing cadastral system through the utilisation of the cadastral fabric system. A comprehensive investigation on the capability of the proposed system is carried out. A total of four evaluation aspects is incorporated in the study to investigate the feasibility and capability of the software, viz. performance of geodetic least squares adjustment, quality assurance techniques, supporting functions, and user friendliness. This study utilises secondary data obtained from the Department of Surveying and Mapping Malaysia (DSMM). The test area is coded as Block B21701 which is located in Selangor, Malaysia. Results show that least square adjustment for the entire network is completed in a timely manner. Various quality assurance techniques are implementable, namely error ellipses, magnitude of correction vectors and adjustment trajectory, as well as inspection of adjusted online bearings. In addition, the system supports coordinate versioning, coordinates of various datum or projection. Last but not least, user friendliness of the system is identified through the software interface, interaction and automation functions. With that, it is concluded that the proposed system is highly feasible and capable to create a Cadastral Fabric to improve the positional accuracy of existing cadastral system used in Malaysia.

Export Citation Format

Share document.

• Open access
• Published: 04 November 2016

An empirical evaluation of software quality assurance practices and challenges in a developing country: a comparison of Nigeria and Turkey

• Olaperi Yeside Sowunmi 1 ,
• Sanjay Misra   ORCID: orcid.org/0000-0002-3556-9331 1 , 2 ,
• Luis Fernandez-Sanz 3 ,
• Broderick Crawford 4 &
• Ricardo Soto 4  

SpringerPlus volume  5 , Article number:  1921 ( 2016 ) Cite this article

6490 Accesses

4 Citations

Metrics details

The importance of quality assurance in the software development process cannot be overemphasized because its adoption results in high reliability and easy maintenance of the software system and other software products. Software quality assurance includes different activities such as quality control, quality management, quality standards, quality planning, process standardization and improvement amongst others. The aim of this work is to further investigate the software quality assurance practices of practitioners in Nigeria. While our previous work covered areas on quality planning, adherence to standardized processes and the inherent challenges, this work has been extended to include quality control, software process improvement and international quality standard organization membership. It also makes comparison based on a similar study carried out in Turkey. The goal is to generate more robust findings that can properly support decision making by the software community. The qualitative research approach, specifically, the use of questionnaire research instruments was applied to acquire data from software practitioners.

In addition to the previous results, it was observed that quality assurance practices are quite neglected and this can be the cause of low patronage. Moreover, software practitioners are neither aware of international standards organizations or the required process improvement techniques; as such their claimed standards are not aligned to those of accredited bodies, and are only limited to their local experience and knowledge, which makes it questionable. The comparison with Turkey also yielded similar findings, making the results typical of developing countries. The research instrument used was tested for internal consistency using the Cronbach’s alpha, and it was proved reliable.

For the software industry in developing countries to grow strong and be a viable source of external revenue, software assurance practices have to be taken seriously because its effect is evident in the final product. Moreover, quality frameworks and tools which require minimum time and cost are highly needed in these countries.

Software application packages are usually developed under stringent conditions of time and cost while in a bid to satisfy the requirements of the users. Despite these conditions, such application or system packages must still satisfy functional and non-functional attributes such as maintainability, reliability, dependability, security, availability and other ‘ilities’ as specified. The only assurance of achieving positive results at all these fronts is by adhering to software quality assurance and management processes. To ensure that bugs and flaws in software products are identified and removed, it is necessary to adhere to software quality standards. This would prevent a number of flaws before the implementation and deployment of the application.

Software quality assurance is imperative for a software organization’s success. It ensures the quality of the software while ensuring that it is fully functional and well documented for easy maintenance. It goes beyond testing the application but also includes the monitoring and control of the entire software development processes and products (Scarpino 2011 ).

Software engineering is the application of a systematic, disciplined, and quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software (IEEE Standard 1990 ). According to Sommerville ( 2007 ), it is an engineering discipline that is concerned with all aspects of software production from the early stages of system specification to maintaining the system after it has gone into use. As with other engineering fields, quality practices are necessary to attain success in the process.

Globally, this field of engineering is growing rapidly, and becoming more structured than ever. At the moment, it is more relevant than it has ever been in history. Software products are ubiquitous and changing the face of businesses globally. They are being used to monitor and deploy government infrastructures, to manage financial portfolios, carry out medical procedures, build and control real-time and mission critical systems that cannot afford to fail. Although standards in the field are not yet as pronounced and enforced as in other engineering disciplines; there are best practices and already proven quality techniques that should be taken with austerity, if quality software is to be developed.

Different software quality techniques have been developed including software testing, code reviews, process improvements, risk management, configuration and change management amongst others. These activities can be executed both manually and automatically with the aid of specialized tools.

However, this universal growth in the field is not so evident in developing countries; a negative trend is observed instead. It is indeed saddening to know that only 10% of the software products used in Nigeria are built by indigenous companies; the larger percentage comes from other countries, specifically India. It is on record that Nigeria loses an average of $1 billion dollars to software importation annually; between 1995 and 2008, N23 billion was spent on the purchase of foreign software, and in 2012 alone, over N59bn was transferred out in purchase and maintenance of software; even the government invests so much in foreign software. Nigeria has been noted to be one of the major importers of software products in sub-Saharan Africa (Nwogbo 2010 ; Nigerian Local Content Development Board 2012 ; The Ministerial Committee on ICT Policy Harmonization 2012 ) An explanation to all these might be that the local companies are not producing quality software products.

The quality of a software product is determined by how much the product meets the customer’s requirements, how much the product performs to specifications and the number of defects in it. It is well known, that high quality products are always patronized to the detriment of substandard ones. Therefore, a need for the assessment of the software development practices in indigenous companies in Nigeria, in a bid to unravel the cause of its low patronage and recommend possible solutions to move the industry forward is what motivated this research.

A prior research of the present work was presented in a conference (Sowunmi and Misra 2015 ). This work is an extension of the conference paper, including three new research questions and thus, more revealing findings. A thorough assessment of the overall software quality assurance and management of software organizations has been carried out and comparisons have been made with similar research in Turkey. The research questions investigated in totality are:

RQ1 Do software companies in Nigeria engage in software quality planning?

RQ2 Do software companies in Nigeria follow certified standardized processes and procedures?

RQ3 Do software companies in Nigeria engage in software quality control i.e. measure/test their software product against standards using metrics?

RQ4 Do software companies in Nigeria improve on their processes over time?

RQ5 Are software companies in Nigeria certified by international organizations?

RQ6 What are the challenges inhibiting the adoption of quality practices?

The instrument used to collect data was the questionnaire and the data collected was used to answer the research questions.

The next section presents a literature survey of previous works carried out in the area of software quality management and assurance, followed closely by the detailed methodology that this research work employed. The results of the findings are then be presented, followed by the discussion of results, recommendations, and conclusions.

Background and literature survey

In this section we provide the background, concept and fundamentals of the software quality and quality assurance. The various work done in this area are also summarized in this section.

Quality was first introduced formally by Bell Laboratories in 1916, and it gradually permeated into software production in the 1970s when military applications where being built (Lewis 2004 ). The term quality in the software engineering field does not apply as in other engineering disciplines such as manufacturing, in that it is not confined to predefined specifications; in this case, it should be tailored towards specific customer requirements and organizational standards (Sommerville 2007 ). Quality in the language of software engineering as discussed by Lewis ( 2004 ) means ‘meeting requirements’ and ‘fitness for use’. This implies that the software meets the requirements of the users as stated in the requirements specification, and it does exactly what the user needs. This definition makes the requirements engineering process and the resulting documentation very important, since the quality system revolves around it. Quality is considered a vital requirement for software products, a business essential, a competitive necessity, and a survival issue for the software industry (Murugesan 1994 ). It is a complex concept that is ambiguous and can be difficult to measure. Strong quality focus is emerging in all phases of the software development lifecycle with increasing emphasis on product quality, process maturity, and continual process improvements.

Quality management

Quality management entails all planned systematic activities and processes for creating, controlling and assuring quality. It is not just a task, but it is a habit that needs to be ingrained into a company’s culture (Ebert and Dumke 2011 ). It also aims to monitor and refine the development process, based on the assumption that the quality of the development process directly affects the quality of the delivered product.

• Software quality assurance

There are different definitions for the term software quality assurance (SQA), some of them are stated below:

Software quality assurance, is a well-defined, repeatable process that is integrated with project management and the software development lifecycles to review internal control mechanisms and assure adherence to software standards and procedures. The objective of the process is to assure conformance to requirements, reduce risk, assess internal controls and improve quality while conforming to the stated schedule and budget constraints (Owens and Khazanchi 2009 ).

Software quality assurance is the planned and systematic approach to the evaluation of the quality of and adherence to software product standards, processes and procedures (Agarwal et al. 2007 ). It includes the process of assuring that standards and procedures are followed throughout the software lifecycle.

Software quality assurance is a process itself which envelopes the entire project and software development life cycle. It is not to be confined to the last stage of software development, or as a means of measuring the produced software. It should begin at the very onset of the project, and span through to the end or retirement of the software itself. This is because quality cannot be added to a finished product, at this stage it can only be patched; SQA is therefore a continuous process and assessment (Thayer and Fairley 1997 ).

It was reported in (Owens and Khazanchi 2009 ) that SQA consists of phases and various activities, which should be carried out by a SQA team of skilled professionals independent of the software development team. They proposed and described an SQA process framework as consisting of the following phases:

SQA initiation before the commencement of a project, the SQA team is notified of it, and necessary quality control and audit processes are defined.

SQA planning the goals and objectives of the software quality assurance plan are defined; quality processes or procedures to be followed, standards and metrics to be used, reviews and audits to be carried out are decided.

Requirements assurance validation of requirements to ensure testability, feasibility and completeness.

Design assurance verification of design against requirements, and ensuring that the planned methodologies are being used.

Development assurance making certain that the development team is following the stated development process and coding standards.

Testing assurance verifying that adequate testing has been carried out and defects nave been tracked, recorded and corrected.

Implementation assurance providing assurance that the necessary implementation steps have been completed prior to and after implementation.

SQA closing this entails confirming that the necessary project closing activities, post project review and formal documentation of lessons learnt have been completed.

The term software quality assurance is generally used interchangeably with software quality management, likewise in this work.

Quality planning

This is the process where a specific quality plan is developed for particular project. It involves a selection of organizational standards that are specific to the software project in question and the development process to be used. It also specifies how the quality assessment process will be carried out. It helps to evaluate the project at its end, by checking whether the plan and all quality milestones are achieved.

Quality control

This is the process of monitoring the software development process and checking the product or deliverables (such as the design model or code) to make sure the quality plan and organizational standards and procedures are being followed by the development team. Quality control encompasses a set of software engineering actions that help to ensure that each work product meets its quality goals (Pressman 2010 ). It can be carried out using automated software assessment or by a quality review team. It often involves measurements using software metrics. Any compromise to quality standards that is detected is documented and forwarded to the appropriate personnel for correction. Methods that can be used include design and code walkthroughs, review, testing, inspection and performance checks.

The software quality assurance team

Every member of the overall project team is responsible for maintaining quality in the project, not withstanding, there is still a dire need for a dedicated team committed to the purpose of quality assurance. In previous years, quality assurance was the responsibility of whoever built the product, but that is not so anymore. This team should comprise of people separate from the development team. They assess the product from the customer’s point of view. Their responsibilities include testing, review of documentation (development plans, testing plans, project plan) for completeness and adherence to standards, periodic inspections, reviews and audits (Godbole 2004 ).

Costs and benefits of software quality assurance

The need for software quality assurance cannot be overemphasized. A lack of it has been shown to be one of the major causes of software project failure. It plays a very vital role in the software life cycle process and can substantially increase the chance of a project’s success. It also helps to mitigate potential risks (Owens and Khazanchi 2009 ).

Regardless of the tools, techniques and experience of the development team, failure to give heed to software quality can result in exceeding the allocated time and budget for the project, failure to meet project objectives, poor customer satisfaction and excessive rework.

Software quality is not achieved by chance; a product does not just attain the specified requirements by sheer luck. It is the result of deliberate actions and steps which cost time, money and effort. While ensuring quality has a cost, lack of quality has a cost too. The cost of quality can be divided into three: cost of prevention, cost of appraisal and cost of failure. Costs of prevention include costs to plan and coordinate activities in the SQA process; appraisal costs include cost of measuring the product such as testing, review and metrics evaluations while cost of failure include cost to correct an error, or rework a process due to defect. Failure costs can be internal based on defects detected before shipment to the client or external, based on defects detected have deploying at the client’s site (Pressman 2010 ).

In the long run, quality management decreases production costs because the sooner a defect is located and corrected, the less costly it will be. While the initial costs can be very substantial, it cannot be compared to the adverse effects of losing a customer, a bad reputation, or going out of business. The costs of prevention are easier to bear, than the cost of failure (Lewis 2004 ).

Challenges inhibiting implementation of software quality assurance

Software companies frequently face many difficult challenges in their attempt to deliver high-quality software and strife to achieve customer satisfaction (Elgebeely 2013 ). From different literatures, possible factors that can impair software quality management include: impatient management, strict deadlines, developer ego, extra cost required (e.g. for the purchase of tools), bureaucracy, inadequate tools that can help to automate the process, low level of acquaintance and knowledge of the process, lack of organizational training on quality standards, inexistent framework for quality management in the organization, disapproval by top management, contrary beliefs and opinion, and previous futility of the process.

Pitfalls in SQA

From the literature review, a number of general pitfalls practiced by software organizations in a an attempt to ensure quality were identified and discussed in this sub-section.

Software organizations tend to rush into implementing a software quality assurance process without a prior establishment of functional software quality assurance practices within individual departments (Scarpino 2011 ). Ideally, the reverse is supposed to be the case, quality assurance needs to be enforced first at the departmental level before an encompassing overall process at the top level.

Some software organizations avoid enforcing quality assurance processes in an attempt to ‘cut cost’ and ‘save time’. This is wrong because research has shown that bugs are cheaper to identify and correct during development than after release or deployment at the client’s site (Drake 1996 ).

Software organizations need to observe and improve their SQA processes from time to time. When an established SQA process or activity is being applied for different projects, the suitability and effectiveness of the process should be monitored for future improvements. However, due to some factors this is not usually implemented and improvements are not made.

Evading some already established processes and/or not adhering strictly to the specified order. Each stage or activity in a SQA process is necessary and essential for the overall effectiveness of the entire process. The results of the overall process cannot be relied upon if the sequence of steps laid down is not duly followed.

Mix-up of roles is another issue. A number of organization mixup roles of personnel in executing some tasks. For example, a development manager closing bugs in the bugs repository after they have been fixed rather than a QA team member, members of the development team managing the requirements document, a developer who also serves as a support staff. All these might make void the essence of the process.

SQA should not be seen as the sole responsibility of the SQA team, but a responsibility of everyone involved in any activity in the entire software development lifecycle. Every worker should be thoroughly informed of what is expected in ensuring quality in whatever role they take part in. Moreover, SQA is much more than testing and should not be delayed until the latter end of the project, rather it should be incorporated right from its inception.

Related works

Generally, quality management processes are not strictly adhered to by software companies, and this reduces the overall quality of the software produced. Several research have been carried out with respect to quality implementations in the development processes of software organizations.

Drake ( 1996 ) presented a case study that showed the benefits of ‘applied quality assurance and code-level measurement activities’. The case study presented a software package that had a time-line of 6 months for development, integration and delivery. Due to the tight schedule, throughout the development period, QA activities such as code inspections, walkthroughs, process control and testing were neglected. At the end of the project, the users considered it unacceptable because it took about 4–5 h to perform its critical function. After 2 weeks of an attempt to fix the code, the senior developer realized that the code needed to be reengineered. After about 6 weeks, the new code was ready and that critical section took only few seconds. Due to lack of enforcement of quality, more time and effort was eventually spent.

Laporte et al. ( 2012 ), reported the results of a research that measured the cost of software quality. The results from analyzing over 1100 software tasks that spanned about 88,000 h showed that software quality accounts for about 33% of overall project cost—cost of evaluation accounting for the highest (21%), cost of correcting anomalies was next with 10% and then cost of the prevention, the least, at 2%. It cannot be overemphasized that it pays off to carryout preventive measures of ensuring software quality rather than corrective measures.

Researchers have also worked on the impact of organizational factors on quality. Nagappan et al. ( 2008 ), carried out a research to provide empirical evidence to validate that organizational factors affect software quality. The authors developed a metric for measurement and applied it to data from Windows Vista. Their results showed that of a truth organizational factors affect failure-proneness, even above metrics like churn, dependencies, complexity. Lavallée and Robillard ( 2015 ) also carried out a study to determine how organizational factors affect working conditions of software developers and in turn the quality of software produced. It was observed that decisions made under pressure due to certain organizational factors such as structure of the organization had a negative effect on software quality. The study was carried out via non-participant observation during weekly meetings of an in-house development team of a large telecommunication company over a period of 10 months. Organizational factors including budget protection, scope protection, organizational politics, human resource planning issues and undue pressure from management and senior developers negatively affected the quality of the software products.

Even for companies which implement SQA practices, different issues impede the success and full realization of the benefits of the process. Scarpino ( 2011 ), conducted a software quality assurance evaluation on a software organization that develops software for mobile data synchronization and manages software systems. The research which focused on a particular organization was conducted via face to face interviews at the organization. The findings from the research revealed that the organization was more into software testing rather than an entire software quality process. The research revealed a number of issues within the organization: the organization’s test case steps were too bulky, the test case layout was not directly related to functional specifications, e-communication was employed instead of physical communication between members of the QA team and the developers to analyze test activity, lack of involvement of the QA group at the initiation of a change, lack of efficient use of test case and defect repositories (they were not being used as knowledge bases with other relevant departments; the bug tracking tool (Bugzilla) and the test case repository were not being used as expected) mixup of roles between the development manager and the QA team, as well as insufficient communication between the technical, QA and development team.

Scarpino and Kovacs ( 2008 ) also researched on the adverse effects of implementing a SQA tool without prior establishment of a software quality process for the organization. An organization that implemented an SQA tool was used for this study. The data was collected via interviews and open observational analysis by an external consultant and an internal QA expert. The following were the findings: team members to use the tool were not given adequate training and assistance, there was no clear documentation of how the system would fit into the company’s software development life cycle, the short time and a lack of initial communication with members of the team led to high resistance towards the implementation of the tool. The tool itself was not properly reviewed to verify that it offered all the company’s expectations. The researchers also noticed an inconsistent review of the implementation progress of the tool.

More specifically, assessment of software quality practices of organizations have also been carried out. An empirical study was carried out in (Pusatli and Misra 2011a ) to evaluate the proper implementation of measurement and metric programs in software companies in an area in Turkey. From their research, they observed a common reluctance and lack of interest in utilizing measurements/metrics despite the fact that they are well known in the industry. They also discovered that internationally recognized standards such as ISO and CMMI are only followed if they are explicitly specified as a project’s requirements.

An assessment of the implementations of quality standards in the software industry of Turkey was also carried out (Pusatli and Misra 2011b ). They found out that even organizations that have the ISO and CMMI certificates do not follow the prescribed directives of this organization after obtaining the certificates. They found out the companies do not see quality issues as primary, some don’t even know the names of common quality standards; they believe acquiring the standards are just for ‘show-off’ and that they do not necessarily influence the quality of the products, neither do they make the customers happy which is their priority.

Within the context of developing countries, specifically in Nigeria, similar work has also been done.

Soriyan and Heeks ( 2004 ) performed a comprehensive study of the Nigerian software industry. Their study cut across a general profile of the industry, reviewing location and ownership of the firms, their personal and job descriptions. The study also covered the type of customers they provide services for, as well as the products and services rendered, not leaving out the processes and methods engaged in executing projects. As a result, an expansive picture of the general state of the software industry in Nigeria at the time of the study was presented. However, the study only gave a general profile on the industry without focus or emphasis on its SQA practices.

A group of researchers also investigated the state of software engineering ethics in Nigeria. They observed nonchalance, dispassion and mass negligence on the issue. They also showed with the aid of a case study, that the ACM/IEEE software engineering code of ethics when applied to software development project helps to resolve ethical dilemmas (Ume and Chukwurah 2012 ).

A research to feel the pulse of software professionals in Nigeria on their perceptions of the software inspection as a software quality assurance activity was carried out in (Akinola et al. 2009 ). The authors used a structured questionnaire research instrument for their work. They found out that software inspection is highly neglected in most organization’s software development process, as they consider it a waste of time.

Olalekan ( 2005 ) reported a discourse on the state of the software industry in Nigeria. The research highlighted ‘process compromise’, ‘resistance to measurement’ and poor training of students at the higher education institutions as some of the problems befalling the industry. However, the authors only adduced reasons for its mature state, no empirical investigation was carried out.

More closely related is the work by (Aregbesola et al. 2011 ) who carried out an assessment of how and to what extent software organizations in Nigeria follow organizational processes. Their survey revealed that the companies do not have proper documentation of their organizational software processes and they only apply implicit in-house methods. Using the Software Engineering Institute (SEI) CMMI, model and the SEI Maturity Questionnaire, they measured requirement management, software project planning, software project tracking and oversight, software subcontract management, SQA, and software configuration management. Based on the software process maturity assessment and capability assessment of the industry, the Nigerian software industry is only at the SEI CMMI maturity level 1, while it toggled between 0 and 1 in key process areas.

All these works individually assessed only a part of the entire software quality management process. This research on the other hand takes another dimension, as it seeks to assess the entire processes involved in software quality management and not just a part of it. It also goes beyond that to identify the challenges inhibiting the practice of software quality which the reviewed research works did not assess, this is to discover the peculiarities in the environment that contribute to the current state, so that suitable solutions can be proffered. Moreover, a comparison with the state of the industry in Turkey is made based on the report from a previous research.

Research methodology

The quantitative research method was applied in this research. The survey technique was used and the qualitative data obtained was analysed using descriptive statistics. A thorough literature review of the activities involved in software quality assurance management was embarked to develop the research questions and the research instrument, a closed-ended questionnaire. The questions were reviewed, validated and verified by a software quality professional and a statistician to ascertain the suitability of the questions. A pilot survey was then conducted to ensure that respondents have the correct understanding of the questions.

The questionnaires were then distributed to stakeholders in software development in Lagos being the hub of software activities in Nigeria, and the home to nearly 50% of all software firms in Nigeria (Soriyan and Heeks 2004 ). The data collected was collated and analyzed.

Furthermore, the internal validity for different sections of the questionnaire was measured using the Cronbach’s alpha. This coefficient was calculated using IBM’s SPSS (Statistical Package for the Social Sciences). The results are discussed, and based on the findings, conclusions made. Figure  1 illustrates the research methodology.

This section details the full results of the entire work. The results of the additional research questions are included. A total of 86 questionnaires were analysed. To estimate the reliability of the research instrument, its internal consistency was measured using standardized Cronbach’s alpha which is also known as the coefficient alpha. This was calculated on different sections of the questionnaire, because they measured separate entities of the SQA and also had different Likert scales. For the section that measures quality control and standards, the cronbach alpha was 0.734, for the section that measured quality planning, the cronbach alphas was 0.689 while it was 0.809 for the section that measured the challenges. In the interpretation of cronbach alpha, 0.00 means no consistency, 1.0 means perfect consistency, and any value from 0.70 implies acceptable consistency, as such we can conclude that the research instrument is internally consistent, therefore reliable.

The analysis of the data gathered is as follows:

On quality standards Table  1 and Fig.  2 report the findings. 11.6% of the respondents reported that their organizations did not observe quality standards while only 2.3% said they have no idea of what quality standards are.

Quality standards

Only 33.7% do not have a SQA team that is separate from the development team, and 30.1% either do not have a SQA team or know about such a team.

Results on quality planning are reported in Table  2 and Fig.  3 . A total 22.1% respondents reported that they rarely or never carry out quality planning activities, while only 36 respondents of the 86 reported that they always carry out risk management activities.

As seen in Table  3 and Fig.  4 , quality control and measurement activities are carried out, but only 22% reported that they employ an external review team on their projects. However, periodic reviews, software testing and code walkthroughs are judiciously carried out.

Quality control and measurement

On process improvement activities, 75.6% reported that they improve their processes based on metrics from the previous project, however, this has not been certified by any organization. From the first round of the survey as 57% do not even have an idea of the CMMI, and only 16% are registered under the ISO 9000 assurance models. From the second round of the survey, from the additional questions included, 86.3% are not aware of international or national software standards, and as such are not planning to adopt any.

From the data gathered from respondents, one can ascertain that challenges are being faced at attempts to adhere to software quality assurance practices. Out of the 10 challenges highlighted, the most prominent ones identified include: strict deadlines 72%, extra cost required 46%, inadequate manpower 45.3%, and bureaucracy of the process 40.7%. Full details are given in Table  4 and Fig.  5 .

Sixty-nine of the eighty-six respondents i.e. 80.2% were male while only 19.8% (17) were female. The organizations were of varying staff strength but mostly between 5 and 15.

From the results, some of the striking findings include the following: 13.9% of the respondents either do not have any idea of, or do not practice software quality standards.

Quality standards being major ingredients of quality software is still not understood even in the smallest measure by some practitioners. This implies that in their software development projects, quality standards are not maintained or considered at all.

33.7% of respondents do not have a separate SQA team. As important as a SQA team is in a software development organization, more than half of the respondents do not have one. This implies that no form of quality check is made on software packages before they are shipped to the customers except those made by the developers. This is very risky as it usually takes another eye to identify a bug or potential risk in a software application. 35% do not even have a SQA team at all, or do not have an idea of what a SQA team is.

59.3% do not carry out quality planning always. This means that at the onset of software projects, the quality expectations of the software products are not clearly spelt out. This makes it difficult to determine at the end of the day if the quality attained is what was expected.

Risk management activities in software quality assurance has less than 30% awareness on the part of practitioners of software, this is not a positive one, because it shows that potential risks are not taken care of ahead of time. It they eventually occur; they can really destabilize the team or even crash the project.

81% do not carry out external reviews; that is, they do not subject their software development to scrutiny by parties that are not a part of the organization.

Though some aspects of software quality assurance are taken care of, there is no evidence that certified standardized processes and procedures, are followed.

A good percentage attested to the fact that they adhere to quality standards and control, however, a considerable number are yet to align to this, as such need to be sensitized. Adequate reviews are not being carried out due to the absence of a separate SQA team and an external review team by most organizations. It is not efficient to have those who worked on a project to also review it. A majority of the respondents were not aware of the CMMI as 57% said they had no idea about it at all, and only a very few are registered to the ISO 9000 quality assurance model or any software quality standard organization.

While the practitioners claim to be following software standards, these standards are only based on their level of their knowledge and not aligned to industry standards, as such they might not be yield the best of results.

For the result on challenges, top on the list of barriers was strict deadlines which means that when the time to market is very close, a lot of steps to ensure standards are bypassed. Contrary beliefs and opinion, developer ego, bureaucracy involved in the process and the extra cost involved are other major inhibiting factors. From the first round of the survey we also find that inadequate planning and manpower are also inhibitors.

A majority of the software developers that work in this organizations partake in a minimum of 3 phases of the software lifecycle, this shows that the same set of people are involved in different aspects of a project simultaneously which is not a very good practice because, a likely error committed might not be discovered.

This section presents the comparison with a similar study conducted in Turkey (Pusatli and Misra 2011a ) which was conducted to determine the level of adherence of small and medium scale software enterprises to quality standards.

With respect to compliance with international standards organizations, similar results were obtained. Our study showed that more than 50% of software practitioners are not aware of these standards. The research in Turkey indicates greater awareness of these standards but they are only pursued when they are explicitly required for a project or a job at hand, otherwise, they are seen as long-term goals. Reasons for not taking up the CMMI certification given by some respondents include that it slows the development process and it is not so efficient in practice for small software companies. However, it was observed by the researchers that some software companies that attain the certification only have it as a label and do not follow the regulations afterwards.

Just in line with one of the major challenging inhibitors observed in Nigeria “Strict deadlines”, the review in Turkey identified the same challenge and revealed that the main aim of practitioners is usually to complete and deliver a project within the tight timeline given.

The research in Turkey found that the academic background of the practitioners also limited their knowledge on quality standards, this is because courses on software quality taught in the universities are electives and not compulsory, as such not all graduates of software engineering are grounded in the area.

Other general similarities are discussed quality requirement is not seen as priority, some companies are not aware of quality standards and tools that exist to enhance the measurement of quality. Financial constraints hinder quality, e.g. the cost of hiring extra hands to constitute the SQA team or a professional SQA expert.

This study was conducted in the South Western part of Nigeria only, specifically Lagos, because it has been identified as the hub of the industry in Nigeria, however, this research can be extended to other parts of the country. Moreover, the comparison made with Turkey was based on a previous research, and no new empirical investigation was carried out.

Recommendation

Having discussed the results and findings, the following are recommended.

The Institute of Software Practitioners of Nigeria (ISPON) should sensitize its members on the importance of adherence to quality standards and practices, because a number of firms see it as an extra process with extra cost attached and no remuneration. They should be informed that while enforcing quality might seem expensive at the onset, it is actually cheaper, because not conforming might be costlier in the long run.

Furthermore ISPON can establish a set of quality standards to act as a guide nationally. These standards should be adopted from existing internationally acclaimed standards but made suit the peculiarities of the Nigerian software industry. The institute should not just formulate the standards, but ensure that software practitioners adhere strictly to them.

Software practitioners in Nigeria should also be informed of international institutes and standards organizations that exist to govern and accredit software practices, because it was observed that a vast majority do not even know these organizations as important as they are. They should not just be informed, but also thrive to get accredited by them. This will set the industry in the global stage and make them fit for large and international projects, because they serve as requirements for most of them.

Software companies should of necessity set up a SQA team which ideally should be separate from the development team. They should not partake in any other phase of development, so that they can be properly positioned to identify flaws in the software and other products. Members of the development team should be adequately trained not only on the technical aspects, but also on the quality standards of the organization, and regulatory bodies in the industry both nationally and internationally.

Furthermore, automatic static analysis (ASA) can be employed by these firms since they have limited resources, (in terms of finance, manpower and experienced personnel), and need to make efficient use of these resources.

Automatic static analysis has been proven to be effective, capable of detecting major flaws in program codes, while requiring little effort. They can be incorporated into their existing QA processes, to make it stronger and more reliable. It will help to save the time expended on manual code walkthroughs, and uncover errors usually overlooked mistakenly in the manual process. It will also save cost because some open source packages are actually available for use e.g. ConQAT.

A good awareness of the difference in the cost of ensuring quality during development, before delivery as against after delivery to clients will let organizations see that they can save a lot of time and stress by ensuring quality, because the general notion is that it is not so necessary.

Institute of Software Practitioners of Nigeria and individual organizations should also organize or sponsor their members or employees to attend non-vendor specific conferences and also research in new ways and tools that can help to improve quality efficiently.

Organizational factors that affect productivity such as culture and structure should be properly reviewed and re-defined where necessary, to enhance adherence to software quality processes, because research has shown that they are related and should not be neglected.

Software quality assurance tools can be implemented to reduce the time and effort of team members on quality assurance; however, they should only be implemented after verifying that the tool suits the organization’s SQA process and there would be adequate training of personnel to use it.

Research in the area of software quality should also be sponsored and higher institutions should make software quality assurance a major/compulsory course for students specializing in software engineering.

The research has assessed the overall software quality assurance practices of practitioners in a developing country. The research which was spurred by the need to reduce the level of importation of software into Nigeria and increase the level of patronage of indigenous software organizations has unveiled some potential reasons for the current state of the industry. Recommendations have been made to tackle the current menace and improve quality software practices which if adhered to would lead to the production of quality software packages that would be patronized and stand the test of time.

Agarwal R, Nayak P, Malarvizhi M, Suresh P, Modi N (2007) Virtual quality assurance facilitation model. In: International conference on global software engineering (ICGSE 2007). IEEE, Munich, pp 51–59

Akinola SO, Osofisan AO, Akinkunmi BO (2009) Industry perception of the software inspection process: Nigeria Software Industry as a case study. Afr J Comput ICT 2(2):3–11

Google Scholar  

Aregbesola K, Akinkunmi BO, Akinola OS (2011) Process maturity assessment of the Nigerian Software Industry. Int J Adv Eng Technol 1:10–25

Drake T (1996) Measuring software quality: a case study. Computer (Long Beach Calif) 29:78–87

Ebert C, Dumke R (2011) Software Measurement. Springer, New York

MATH   Google Scholar  

Elgebeely AR (2013) Software quality challenges and practice recommendations. In: IBM. http://www.ibm.com/developerworks/rational/library/software-quality-challenges-practice-recommendations/ . Accessed 15 Nov 2014

Godbole NS (2004) Software quality assurance: principles and practice. Alpha Science International Limited, Oxford

IEEE Standard (1990) IEEE Standard Glossary of Software Engineering Terminology

Laporte CY, Nabil B, Mikel D (2012) Measuring the cost of software quality of a large software project at bombardier transportation: a case study. Softw Qual Manag 14(3):14–31

Lavallée M, Robillard PN (2015) Why good developers write bad code: an observational case study of the impacts of organizational factors on software quality. Software engineering (ICSE), 2015 IEEE/ACM 37th IEEE international conference on. IEEE, Florence, pp 677–687

Chapter   Google Scholar  

Lewis WE (2004) Software testing and continuous quality improvement. Auerbach Publications, Boca Raton

Book   MATH   Google Scholar  

Murugesan S (1994) Attitude towards testing: a key contributor to software quality. In: Proceeding of 1st international conference on software testing, reliability and quality assurance. IEEE, New Delhi, pp 111–115. http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=526375

Nagappan N, Murphy B, Basili V (2008) The influence of organizational structure on software quality: an empirical case study. Proceedings of the 30th international conference on Software engineering. ACM, Chicago, pp 521–530

Nigerian Local Content Development Board (2012) A communique issued at the end of the nigerian local content summit held at the hotel presidential, port harcourt on the 25th and 26th june, 2012. Presented in the workshop operationalizing a development agenda for local content. http://tandicebsolutions.com/rokdownloads/nigerian_local_content_summit/summitcommunique.pdf

Nwogbo K (2010) On Ispon Software Bazaar. Niger Commun Week. http://www.nigeriacommunicationsweek.com.ng/editorial/on-ispon-software-bazaar . Accessed 30 Sept 2016

Olalekan AS (2005) Conducting empirical software engineering research in Nigeria: the posing problems. Proceedings of the 27th international conference on software engineering. ACM, New York, pp 633–634

Owens DM, Khazanchi D (2009) Software quality assurance. Handbook of research on technology project management, planning, and operations. IGI Global, Pennsylvania, pp 245–263

Pressman R (2010) Software engineering: a practitioners approach. Mc-Graw Hill, New York

Pusatli OT, Misra S (2011a) Software measurement activities in small and medium enterprises: an empirical asessment. Acta Polytech Hungarica 8:21–42

Pusatli OT, Misra S (2011b) A discussion on assuring software quality in small and medium. Tech Gaz 18:447–452

Scarpino JJ (2011) An analysis of an enterprise mobility software company—managing software quality and maintaining a competitive edge in flunctuating periods of corporate growth: a case study. Issues Inf Syst 12:7–15

Scarpino J, Kovacs P (2008) Software quality assurance tool’s implementation: a case study. J Int Assoc Comput Inf Syst 9:146–152

Sommerville I (2007) Software engineering. Addison-Wesley, England

Soriyan A, Heeks R (2004) A profile of Nigeria’s software industry. Manchester, UK

Sowunmi OY, Misra S (2015) An empirical evaluation of software quality assurance practices and challenges in a developing country. In: IEEE international conference on computer and information technology; Ubiquitous computing and communications; dependable, autonomic and secure computing; pervasive intelligence and computing (CIT/IUCC/DASC/PICOM). IEEE, Liverpool, pp 867–871

Thayer RH, Fairley RE (1997) The silver bullets of Software Engineering. In: Software Engineering Project Management, Wiley, New Jersey

The Ministerial Committee on ICT Policy Harmonization (2012) National ICT Policy. http://www.researchictafrica.net/countries/nigeria/Nigeria_National_ICT_Policy_(draft)_2012.pdf . Accessed 15 Nov 2014

Ume A, Chukwurah J (2012) Underscoring software engineering ethics in Nigeria’s fast growing information and communications technology. Asian Trans Comput 2:21–30

Download references

Authors’ contributions

OYS is the Phd students and done maximum work under the supervision of SM. SM is main supervisor of OYS and working with her since last four 2 for completion of the work. LFS Luis is Software quality assurance specialist and advisor in several IT companies in EU. He helps us in revising the questionnaire and improving the quality of the paper. BCL abrin and Ricardo Soto- are co researchers with our software engineering cluster in CU. They both contributed a lot for improving the manuscript (reviewed and added valuable contributions) since the beginning of the work. All authors read and approved the final manuscript.

Acknowledgements

We are thankful to Mr. Adewole Adewumi of Computer and Information Science Department for his valuable suggestions and comments for improvement of the work/paper. One of the author Olaperi is also thankful to Head of CIS department for providing resources and support during the work.

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and affiliations.

Covenant University, Ota, Nigeria

Olaperi Yeside Sowunmi & Sanjay Misra

Atilim University, Ankara, Turkey

Sanjay Misra

University of Alcala, Madrid, Spain

Luis Fernandez-Sanz

Pontificia Universidad Católica de Valparaíso, Valparaiso, Chile

Broderick Crawford & Ricardo Soto

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sanjay Misra .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Sowunmi, O.Y., Misra, S., Fernandez-Sanz, L. et al. An empirical evaluation of software quality assurance practices and challenges in a developing country: a comparison of Nigeria and Turkey. SpringerPlus 5 , 1921 (2016). https://doi.org/10.1186/s40064-016-3575-5

Download citation

Received : 02 June 2016

Accepted : 18 October 2016

Published : 04 November 2016

DOI : https://doi.org/10.1186/s40064-016-3575-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Software quality
• Software quality challenges

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Software Quality for AI: Where We Are Now?

• Conference paper
• First Online: 06 January 2021
• Cite this conference paper

• Valentina Lenarduzzi 11 ,
• Francesco Lomio 12 ,
• Sergio Moreschini 12 ,
• Davide Taibi 12 &
• Damian Andrew Tamburri 13  

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 404))

Included in the following conference series:

• International Conference on Software Quality

1871 Accesses

18 Citations

Artificial Intelligence is getting more and more popular, being adopted in a large number of applications and technology we use on a daily basis. However, a large number of Artificial Intelligence applications are produced by developers without proper training on software quality practices or processes, and in general, lack in-depth knowledge regarding software engineering processes. The main reason is due to the fact that the machine-learning engineer profession has been born very recently, and currently there is a very limited number of training or guidelines on issues (such as code quality or testing) for machine learning and applications using machine learning code. In this work, we aim at highlighting the main software quality issues of Artificial Intelligence systems, with a central focus on machine learning code, based on the experience of our four research groups. Moreover, we aim at defining a shared research road map, that we would like to discuss and to follow in collaboration with the workshop participants. As a result, the software quality of AI-enabled systems is often poorly tested and of very low quality.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Systematic literature review on software quality for AI-based software

Synergies Between Artificial Intelligence and Software Engineering: Evolution and Trends

AI-Oriented Software Engineering (AIOSE): Challenges, Opportunities, and New Directions

Informatics Europe https://www.informatics-europe.org .

ACM Europe Council https://europe.acm.org .

The Networked European Software and Services Initiative - NESSI http://www.nessi-europe.com .

https://twitter.com/vale_lenarduzzi/status/1295055334264975360 . Last access: 28 August 2020.

TensorFlow version compatibility. https://www.tensorflow.org/guide/versions

Compatible Versions of PyTorch/Libtorch with Cuda 10.0 (2019). https://discuss.pytorch.org/t/compatible-versions-of-pytorch-libtorch-with-cuda-10-0/58506 . Accessed 11 July 2020

Machine Learning Glossary, Google Developers (2019). https://developers.google.com/machine-learning/glossary . Accessed 28 Aug 2020

Pytorch Lightning. The lightweight PyTorch wrapper for ML researchers (2019). https://github.com/PyTorchLightning/pytorch-lightning . Accessed 11 July 2020

Tensorflow 1.11.0 incompatible with keras2.2.2? (2019). https://github.com/tensorflow/tensorflow/issues/22601 . Accessed 11 July 2020

Avgeriou, P., Kruchten, P., Ozkaya, I., Seaman, C.: Managing technical debt in software engineering (Dagstuhl seminar 16162). Dagstuhl Reports 6 (2016)

Google Scholar  

Avgeriou, P., et al.: An overview and comparison of technical debt measurement tools. IEEE Softw. (2021)

Bradley, A.P.: The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recogn. 30 (7), 1145–1159 (1997)

Article   Google Scholar  

Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M., Pill, R.: Using meta ethnography to synthesise qualitative research: a worked example. J. Health Serv. Res. Policy 7 (4), 209–215 (2002). http://www.ncbi.nlm.nih.gov/pubmed/12425780

Chen, T.Y.: Metamorphic testing: a simple method for alleviating the test oracle problem. In: Proceedings of the 10th International Workshop on Automation of Software Test, AST 2015, pp. 53–54. IEEE Press (2015)

Cohen, G., Afshar, S., Tapson, J., Van Schaik, A.: EMNIST: extending MNIST to handwritten letters. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921–2926. IEEE (2017)

Commission, E.: WHITE PAPER On Artificial Intelligence - A European approach to excellence and trust (2020). https://ec.europa.eu/info/sites/info/files/commission-white-paper-artificial-intelligence-feb2020_en.pdf?utm_source=CleverReach&utm_medium=email&utm_campaign=23-02-2020+Instituts-Journal+07%2F20%3A+Wo+waren+Sie%3F+Es+ging+um+Sie%21&utm_content=Mailing_11823061 . Accessed 09 July 2020

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

Kästner, C., Kang, E.: Teaching software engineering for AI-enabled systems. arXiv preprint arXiv:2001.06691 (2020)

Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence, IJCAI 1995, vol. 2, pp. 1137–1143. Morgan Kaufmann Publishers Inc., San Francisco (1995)

Larus, J., et al.: When computers decide: European recommendations on machine-learned automated decision making (2018)

LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86 (11), 2278–2324 (1998)

Lenarduzzi, V., Sillitti, A., Taibi, D.: A survey on code analysis tools for software maintenance prediction. In: Ciancarini, P., Mazzara, M., Messina, A., Sillitti, A., Succi, G. (eds.) SEDA 2018. AISC, vol. 925, pp. 165–175. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-14687-0_15

Chapter   Google Scholar  

Lwakatare, L.E., Raj, A., Bosch, J., Olsson, H.H., Crnkovic, I.: A taxonomy of software engineering challenges for machine learning systems: an empirical investigation. In: Kruchten, P., Fraser, S., Coallier, F. (eds.) XP 2019. LNBIP, vol. 355, pp. 227–243. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19034-7_14

Dhaval, M.: How to perform Quality Assurance for Machine Learning models? (2018). https://medium.com/datadriveninvestor/how-to-perform-quality-assurance-for-ml-models-cef77bbbcfb . Accessed 09 July 2020

Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (2010). [u.a.]. http://www.amazon.com/Machine-Learning-Tom-M-Mitchell/dp/0070428077

MATH   Google Scholar  

Murphy, C., Kaiser, G.E., Arias, M.: A framework for quality assurance of machine learning applications. Columbia University Computer Science Technical reports, CUCS-034-06 (2006)

NESSI: Software and Artificial Intelligence (2019). http://www.nessi-europe.com/files/NESSI%20-%20Software%20and%20AI%20-%20issue%201.pdf . Accessed 09 July 2020

Radhakrishnan, V.: How to perform Quality Assurance for Machine Learning models? (2019). https://blog.sasken.com/quality-assurance-for-machine-learning-models-part-1-why-quality-assurance-is-critical-for-machine-learning-models . Accessed 09 July 2020

van Rossum, G., Warsaw, B., Coghlan, N.: PEP 8 - Style Guide for Python Code. https://www.python.org/dev/peps/pep-0008/

Rushby, J.: Quality measures and assurance for AI (artificial intelligence) software. Technical report (1988)

Russell, S.J., Norvig, P.: Artificial Intelligence - A Modern Approach: The Intelligent Agent Book. Prentice Hall Series in Artificial Intelligence. Prentice Hall, Upper Saddle River (1995)

Sculley, D., et al.: Hidden technical debt in machine learning systems. In: Advances in Neural Information Processing Systems, pp. 2503–2511 (2015)

Wang, J., Li, L., Zeller, A.: Better code, better sharing: on the need of analyzing Jupyter notebooks (2019)

Zhang, J.M., Harman, M., Ma, L., Liu, Y.: Machine learning testing: survey, landscapes and horizons. IEEE Trans. Softw. Eng. PP , 1 (2020)

Ören, T.I.: Quality assurance paradigms for artificial intelligence in modelling and simulation. Simulation 48 (4), 149–151 (1987)

Download references

Acknowledgement

We would like to thank the software engineering and AI community and in particular Taher Ahmed Ghaleb, Steffen Herbold, Idan Huji, Marcos Kalinowski, Christian Kästner, Janet Siegmund and Daniel Strüber for helping us on the definition of the glossary of AI and SW Engineering terms Footnote 4 .

Author information

Authors and affiliations.

LUT University, Lahti, Finland

Valentina Lenarduzzi

Tampere University, Tampere, Finland

Francesco Lomio, Sergio Moreschini & Davide Taibi

Eindhoven University of Technology - JADS, ’s-Hertogenbosch, The Netherlands

Damian Andrew Tamburri

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Valentina Lenarduzzi .

Editor information

Editors and affiliations.

TU Wien, Vienna, Austria

Dietmar Winkler

Stefan Biffl

Blekinge Institute of Technology, Karlskrona, Sweden

Daniel Mendez

Johannes Kepler University Linz, Linz, Austria

Manuel Wimmer

Software Quality Lab GmbH, Linz, Austria

Johannes Bergsmann

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Cite this paper.

Lenarduzzi, V., Lomio, F., Moreschini, S., Taibi, D., Tamburri, D.A. (2021). Software Quality for AI: Where We Are Now?. In: Winkler, D., Biffl, S., Mendez, D., Wimmer, M., Bergsmann, J. (eds) Software Quality: Future Perspectives on Software Engineering Quality. SWQD 2021. Lecture Notes in Business Information Processing, vol 404. Springer, Cham. https://doi.org/10.1007/978-3-030-65854-0_4

Download citation

DOI : https://doi.org/10.1007/978-3-030-65854-0_4

Published : 06 January 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-65853-3

Online ISBN : 978-3-030-65854-0

eBook Packages : Computer Science Computer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research