literature review for comparative study

A comparative review may, e.g., require you to examine two schools of thought, two issues, or the positions taken by two persons. You may create a hierarchy of issues and sub-issues to compare and contrast, as suggested by the following general plan.

This model lists 3 options for structuring the body of the review. In all cases, you are expected to deal with the similarities ( compare ) and then with the differences ( contrast ): Introduction, Body, & Conclusion

Literature Review Example 3 offers an excellent example of  a comparative review [ Language and gender ]. This was written by Alastair Pennycook for his undergraduate students as a model of (among other things) of how to structure a  review of the literature - for an example of the above structure.

• Methodology
• Open access
• Published: 04 May 2016

Using qualitative comparative analysis in a systematic review of a complex intervention

• Leila Kahwati 1 ,
• Sara Jacobs 1 ,
• Heather Kane 1 ,
• Megan Lewis 1 ,
• Meera Viswanathan 1 &
• Carol E. Golin 2  

Systematic Reviews volume  5 , Article number:  82 ( 2016 ) Cite this article

6825 Accesses

27 Citations

15 Altmetric

Metrics details

Systematic reviews evaluating complex interventions often encounter substantial clinical heterogeneity in intervention components and implementation features making synthesis challenging. Qualitative comparative analysis (QCA) is a non-probabilistic method that uses mathematical set theory to study complex phenomena; it has been proposed as a potential method to complement traditional evidence synthesis in reviews of complex interventions to identify key intervention components or implementation features that might explain effectiveness or ineffectiveness. The objective of this study was to describe our approach in detail and examine the suitability of using QCA within the context of a systematic review.

We used data from a completed systematic review of behavioral interventions to improve medication adherence to conduct two substantive analyses using QCA. The first analysis sought to identify combinations of nine behavior change techniques/components (BCTs) found among effective interventions, and the second analysis sought to identify combinations of five implementation features (e.g., agent, target, mode, time span, exposure) found among effective interventions. For each substantive analysis, we reframed the review’s research questions to be designed for use with QCA, calibrated sets (i.e., transformed raw data into data used in analysis), and identified the necessary and/or sufficient combinations of BCTs and implementation features found in effective interventions.

Our application of QCA for each substantive analysis is described in detail. We extended the original review findings by identifying seven combinations of BCTs and four combinations of implementation features that were sufficient for improving adherence. We found reasonable alignment between several systematic review steps and processes used in QCA except that typical approaches to study abstraction for some intervention components and features did not support a robust calibration for QCA.

Conclusions

QCA was suitable for use within a systematic review of medication adherence interventions and offered insights beyond the single dimension stratifications used in the original completed review. Future prospective use of QCA during a review is needed to determine the optimal way to efficiently integrate QCA into existing approaches to evidence synthesis of complex interventions.

Peer Review reports

Systematic reviews evaluating complex or multicomponent interventions often encounter substantial clinical heterogeneity in intervention components, settings, and populations studied, which often contribute to heterogeneity of effect size. Complex interventions are those that include multiple components that often but do not necessarily interact with each other [ 1 – 4 ]. The UK Medical Research Council suggests that characteristics such as the number and difficulty of behaviors required by those delivering or receiving the intervention, the number and variability of targeted outcomes, and the degree of flexibility of tailoring of the intervention all contribute to an intervention’s complexity [ 5 ]. In addition to the number of components an intervention has, complexity can also refer to properties of the system in which an intervention is implemented, such as setting, number of actors involved, and intervention target characteristics [ 6 , 7 ]. Further, an intervention may employ multiple and varied implementation strategies [ 7 ]. As a result of these myriad sources of potential variation, complex interventions with a common underlying purpose may differ quite substantially from each other in form or function when implemented.

Accordingly, systematic review investigators face substantial methodological challenges to synthesizing bodies of evidence comprised of complex interventions [ 7 ]. Estimating summary effects via quantitative synthesis is often not possible because of heterogeneity. Reviewers may ignore underlying variation by only addressing an overall question of effectiveness (e.g., do these types of interventions work?), or reviewers may stratify the synthesis based on one or more aspects of variation, such as a specific intervention component, outcome, population, or setting [ 7 ]. However, multicomponent interventions with interdependent components may not be suitable for separation into distinct components, and assumptions about linear and additive effects of multiple components may not be valid [ 8 ]. Methods that can systematically explore heterogeneity based on an assumption of causal complexity and that can provide an analytic link between heterogeneity and outcomes would offer an enhancement to current systematic review methods.

Qualitative comparative analysis (QCA) is a case-oriented method to study complex phenomena originating from the comparative social sciences [ 9 ]; it has been proposed as a potential method for synthesizing evidence within systematic reviews [ 7 , 10 ]. QCA uses mathematical set theory, which is the branch of mathematical logic that studies the properties of sets, to examine set relationships between combinations of condition sets (cf., explanatory variables) present among cases and an outcome set (cf., dependent variable). QCA can be useful for identifying complex (i.e., non-linear, non-additive) causal patterns that variable-oriented methods may miss [ 9 , 11 , 12 ]. Applying QCA within the context of a systematic review may enhance review findings for policy-makers and practitioners by systematically evaluating sources of heterogeneity that influence the success (or failure) of an intervention using an approach that preserves each study’s unique combination of intervention components or other features. How to apply QCA within the context of a systematic review and the suitability of the method for this context is not definitively known because few actual applications exist [ 13 , 14 ]. Based on our experience conducting systematic reviews and our experience using QCA in primary research applications, we postulated that using QCA could offer additional insights within a systematic review of a complex intervention beyond traditional synthesis.

In this paper, we describe using QCA within a systematic review and examine its suitability for use within this context. We used data from an Agency for Healthcare Quality and Research (AHRQ)-sponsored review of interventions to improve medication adherence that was recently completed by members of our study team (M.V., C.G.) [ 15 , 16 ]. Medication adherence is a complex behavior with multiple determinants that vary among individuals [ 17 ]. Interventions to improve adherence often involve combinations of behavior change techniques (BCTs), such as interventions to improve self-efficacy or change attitudes. They often use different delivery modes (e.g., telephone vs. in-person) and agents (e.g., physicians, nurses, non-licensed staff) over various intervals of time and at different intensities. Further, interventions may be designed to influence patient adherence through interventions targeted at the practitioner or healthcare system level in addition to patient-directed components. We chose this review to use with QCA because the heterogeneity among interventions and outcomes seemed amenable to exploration through a configural lens and because we had access to all of the raw data and institutional knowledge associated with the review.

We turned to QCA because too much clinical heterogeneity had precluded a meta-analysis and meta-regression. Further, the completed review did not attempt mixed-treatment comparisons because of heterogeneity in the usual-care comparators [ 18 ]. However, all of the aforementioned approaches are correlational in nature, based on the assumption that one true distribution of effect exists and that trial-level covariates independently and additively contribute to variation from the true effect. QCA is not a substitute for these quantitative approaches to synthesis when they are appropriate, but these methods may rarely be appropriate for complex interventions because of the underlying assumptions upon which they are based. Thus, QCA offers a systematic approach to potentially unpacking intervention variability and relationship to an outcome when the phenomena under investigation can be characterized as complex.

We conducted two substantive analyses using QCA using data that was collected as part of a completed review. The first analysis sought to identify which combinations of patient-directed BCTs used across the body of evidence were necessary and/or sufficient for improving medication adherence, and findings from this analysis are presented in detail in a companion paper in this issue [ 19 ]. The second analysis sought to identify which combinations of implementation features (e.g., agent, mode) used across the body of evidence were necessary and/or sufficient for improving medication adherence. In the present paper, we discuss the methodologic approach applied to both analyses and highlight the added value and challenges we identified through its application in a systematic review.

Overview of QCA

Consistent with a case-oriented approach, QCA was originally developed for use with a small to medium number of cases ( N  = 10 to 50), allowing researchers to preserve the iterative nature of the data collection, analysis, and interpretation that stems from familiarity with the cases, a hallmark of qualitative research. More recently, QCA has been used for applications involving larger sample sizes [ 12 ]. Used within a systematic review context, each individual study within the review represents a case.

QCA preserves the holistic nature of each case throughout the analysis by not deconstructing the case into its component variables for analysis. Unlike variable-oriented methods that are based on probabilistic assumptions, QCA uses data from empiric cases to identify set relationships, which can be interpreted as relationships of “necessity” or “sufficiency” that often characterize causally complex phenomena. These relationships are depicted as a solution that uses Boolean operators, such as “AND,” “OR,” and “NOT,” to formulate verbal statements of the relationship between explanatory variables (i.e., conditions in QCA terminology) and an outcome. The solution generated by QCA is analogous to the expression of a correlational relationship among variables using a regression equation; though unlike probabilistic methods, solutions do not offer an estimate of precision, likelihood of finding results due to chance, nor can they be used for statistical hypothesis testing. A truth table is the analytic device used in QCA, and software is used to conduct most analyses [ 12 , 20 ]. A detailed methodological description of QCA, a hypothetical example of an analysis, and a glossary of terms related to QCA is provided as supplementary online material (Additional file 1 ).

Application of QCA to the completed review

Members of our study team (M.V., C.G.) conducted the completed review using methods associated with the AHRQ Effective Health Care Program (available at http://www.ncbi.nlm.nih.gov/books/NBK47095/ ). The completed review was limited to US studies in adults with chronic conditions, excluding patients with HIV/AIDS, severe mental illness, and substance abuse because these conditions often require specialized interventions not applicable to general medical populations [ 15 , 16 ]. Of 4124 citations identified in the completed review, 758 full-text articles were screened for eligibility. Of the 67 low- or medium-risk of bias studies included, 62 were randomized clinical trials and five were observational studies. Included studies were conducted among patient populations with ten different clinical conditions. Seven studies included populations with more than one clinical condition. Study authors did not use consistent language or a standard taxonomy to describe intervention type; thus, the review team developed categories of intervention types. Examples included “education with behavioral support,” “health coaching,” “medication monitoring and reminders,” “shared decision-making or decision aids,” “case management,” and “collaborative care.” Because of heterogeneity of populations and intervention types, a quantitative synthesis was not possible. The primary organizing framework for the qualitative synthesis was clinical conditions (e.g., hypertension, diabetes). Within each of the ten clinical conditions, adherence outcomes were synthesized by intervention type. For example, a low strength of evidence grade for benefit was assigned for the use of case management interventions among patients with diabetes based on evidence from three RCTs. Overall, this approach resulted in 40 strata, each of which was assigned a strength of evidence grade based on the one to five studies falling within the stratum. The completed review’s analytic framework, key questions, and a summary of the results are provided as supplementary online material (Additional file 2 ). In brief, this review found the most consistent evidence for effectiveness across clinical conditions for interventions that included case management and educational interventions.

We developed an approach to using QCA within the context of a systematic review based on existing standards of good practice for conducting QCA and our experience using the method in non-systematic review applications [ 21 – 23 ]. This approach is depicted in Fig.  1 , and although the figure depicts this approach as sequential, in practice, iterative specification and analysis is typical and consistent with qualitative research approaches.

QCA approach used in this analysis. Adapted from Kane et al. [ 22 ]

We will use the elements of Fig.  1 to summarize our process of using QCA with systematic review data.

Specify configural research questions

As indicated in Fig.  1 , we first specified a configural research question, which is a question designed to identify the combinations of conditions that produce an outcome. For each substantive analysis, we specified a single question that combined two of the completed review’s key questions. These were key question 1: “Among patients with chronic diseases with self-administered medication prescribed by a provider, what is the comparative effectiveness of interventions aimed at patients, providers, systems, and combinations of audiences in improving medication adherence?” and key question 3: “How do medication-adherence intervention characteristics vary?” Further, we specified both of the configural research questions to reflect causal asymmetry. The re-specified research question for the first QCA was “What combinations of behavioral change techniques are present in studies demonstrating improved medication adherence?” and for the second QCA was “What combinations of implementation features, such as agent, target, mode, span, and exposure are present in studies demonstrating improved medication adherence?”

Identify studies for use in analysis

We defined studies included in the systematic review as the cases for each analysis. Based on how we operationalized the research questions, we excluded seven of the 67 studies from the completed review from both analyses as they were focused on policy or system level interventions and not relevant to the conditions (BCTs and implementation features) that we were interested in exploring. We found that the process used for study selection in a typical systematic review of interventions, which defines inclusion and exclusion criteria using the PICOTS framework (patient, intervention, comparator, outcome, timing, and setting), ensured that the cases included in the QCA were similar enough to be comparable, yet still offered enough diversity in intervention design to enable understanding heterogeneity of effect. Further, this approach provides an explicit and detailed rationale for the selection (or non-selection) of cases, which is a standard of good practice for conducting QCA [ 21 ].

Specify and calibrate condition sets and outcome set

Because one of our study aims was to assess the suitability of using QCA in a systematic review context, we used a completed review to determine whether data typically abstracted during a review would be acceptable to use with QCA. Thus, our initial approach was to rely on the review’s completed data abstraction files and published evidence tables. However, we adjusted our approach during the course of the analyses to verify and supplement previously abstracted data as we needed additional information not collected during the original review process.

Set calibration refers to the process of assigning a numeric value between 0 and 1 based on data collected from or about the case for each condition set and outcome set included in an analysis. These values are referred to as set membership values and represent the degree to which the case belongs to each of the sets in the analysis. Researchers typically define the rubric that determines what set membership value to assign based on existing theory or information external to the cases at hand. Qualitative and/or quantitative data collected from a case is evaluated against the calibration rubric to determine the specific set membership value that should be assigned to the case. In a crisp-set (cf, binary) calibration scheme, cases are either assigned values of “1” (fully in the set) or “0” (fully out of the set). For example, when trying to establish whether an adherence intervention belongs to the set of studies that are “theory-based,” one could examine whether the intervention designers described and cited specific behavioral theories that were used to develop the intervention; if so, the study would be assigned a 1, and if not, the study would be assigned a 0. Non-binary calibration schemes are also possible and are described in more detail in the online supplementary material (Additional file 1 ).

Studies in the completed review used a variety of medication adherence outcomes measured at various time points based on self-report, prescription fills, or medication event monitoring systems (“smart” medication bottles). Some studies used more than one measure of adherence. We reviewed abstracted data and original studies and determined that we would consider studies to be fully in the set of studies with improved adherence if at least one measure of adherence demonstrated a statistically significant improvement as compared to a usual-care comparison group. We chose this calibration rubric because of the lack of a common adherence measure across studies. We considered using a fuzzy-set calibration rubric, which allows for set membership values between 0 and 1; but, the panoply of adherence measures used both within and across studies and the lack of external standards for defining differences in degree of adherence (e.g., “very much improved adherence” from “slightly improved adherence” from “slightly not improved adherence”) proved too challenging.

Condition sets used in each analysis are summarized in Table  1 . The abstracted data and evidence tables that described the BCTs and implementation features used in studies generally provided inadequate information to enable us to calibrate condition sets; thus, we went back to original study publications to obtain more detail and to clarify ambiguous data abstraction entries for nearly all studies.

The BCTs abstracted during the completed review were determined and defined a priori by the review team and derived from a previous meta-analysis of medication adherence interventions and a published taxonomy of BCTs [ 24 , 25 ]. One study reviewer captured a study’s use of each BCT as “yes” or “no” or “unclear” based on information available in the published intervention description, and this was confirmed by a second reviewer. Thus, studies could be identified as using multiple BCTs. To studies that used a BCT, we assigned a set membership value of 1 for that BCT, and we assigned studies that did not use a BCT, or for which use of the BCT was unclear, a set membership value of 0. We also conducted sensitivity analyses with an alternate rubric that calibrated “unclear” as BCT use.

A challenge we encountered for the first analysis was the large number (12) of BCTs identified during abstraction in the completed review. With this many conditions, we were concerned about limited diversity that would result by including too many condition sets for the fixed number of studies (60). We winnowed the number of included condition sets to nine by eliminating three BCTs that were used by fewer than three studies. We attempted to further reduce the number of BCTs included in analysis by combining two BCTs to create a macrocondition, a typical strategy in QCA to reduce the number of included condition sets. However, we found the BCTs too conceptually distinct to combine into a single macrocondition. Thus, we could not implement a QCA standard of good practice with respect to keeping the number of condition sets relative to the number of cases at a reasonable level [ 21 ].

For the second analysis, which evaluated implementation features, we specified condition set-based implementation features that the completed review authors determined a priori and captured during study abstraction. These features, listed in Table  1 , included intervention agent , target , span of intervention over time, mode of delivery, and intervention exposure . Information about these characteristics was captured by the review team using unstructured abstraction fields. For three of the condition sets, target, agent, and mode, the review team collapsed abstracted data into multivalue, mutually exclusive, categories for descriptive reporting of intervention characteristics.

We evaluated whether the multivalue categorical groupings for target , agent , and mod e could be further collapsed into dichotomous categories for a crisp-set calibration rubric. For target, the review team used information from the published description to assign each study to one of three categories: patient-only, combination of patient and provider, combination of patient and provider and system. For our analysis, we decided that the inclusion of a provider or system target, in addition to targeting the patient, was a key distinction as provider and system interventions would require additional training, infrastructure, and expense. Thus, we considered a study as “fully in” for the target condition set if the intervention targeted a provider or system in addition to a patient. Studies targeting only patients were considered “fully out” of the set. Similarly for mode , we first evaluated the completed review’s categorical groupings before deciding that a key design feature relevant to policy-makers and practitioners would be whether the intervention was delivered in-person versus some other mode (e.g., telephone, virtual, automated) because of secular trends in virtual care, convenience to patients, and perhaps lower costs. We developed two alternatives to accommodate interventions with mixed modes, where some of the intervention was delivered in person and some delivered by phone or virtually. For calibration of the agent condition set, we considered studies that used licensed health care professionals (e.g., nurse, physician, pharmacist) as fully in, and studies that used agents described as research assistants, health coaches, or other non-licensed types of staff as fully out.

The calibration of the final two condition sets in the second analysis, time span of intervention and intensity of exposure , exemplified the iterative back and forth between theory and empirical information from the cases at hand that is a QCA standard of good practice [ 21 ]. Study abstractors captured raw data about these two condition sets in an unstructured format during the review. We first transformed the raw data into standardized numeric values such that time span was represented in “weeks” from beginning to end of the intervention and the total time spent exposed to the intervention was represented in “minutes.” Because exposure information in some studies lacked detail, we made assumptions regarding average length of a clinic visit, telephone contact, or time spent exposed to an automated intervention when it was not specifically provided. For simplicity in interpretation, we chose to calibrate span and exposure with crisp sets. We contemplated various thresholds guided by the following considerations:

Select the calibration threshold with some knowledge of the range of values represented within our studies to avoid setting it too high or too low such that most studies would be in or out of the set.

Incorporate our substantive experience with behavioral interventions regarding what would be considered a threshold for a longer span or a higher exposure, but convey the condition sets using their numeric threshold value rather than terms such as low or high to mitigate concerns over the inherent arbitrariness of wherever we placed the threshold (e.g., span >12 weeks is “in,” rather than “long span” is “in”).

Test alternative thresholds in sensitivity analyses to assess the robustness of our findings with respect to the placement of the calibration threshold.

Ultimately, our main analysis used a calibration threshold of greater than or equal to 12 weeks as fully in the span condition set and a threshold of greater than or equal to 120 min as fully in the exposure condition set. In sensitivity analyses, we evaluated a span threshold of 6 weeks and two exposure thresholds, 60 and 240 min. We identified some differences in findings, and all supplemental analyses were made available as appendices to the main substantive analysis to support transparency and demonstrate the sensitivity of findings to changes in calibration thresholds.

Construct and analyze the truth table

For each analysis, we transformed the raw data matrix of set membership values into a truth table, which places studies with the exact same configuration of set membership values for condition sets into the same truth table row. The number of logically possible truth table rows in an analysis is equal to 2 k , where k is equal to the number of included condition sets; thus, the truth table for the first analysis contained 512 (i.e., 2 9 ) rows and the table for the second analysis contained 32 rows (i.e., 2 5 ). In both analyses, some of the truth table’s logically possible configurations were not present in any studies so these rows are “empty” of any empiric cases and are called logical remainders. The truth table is the analytic device in QCA for determining which configurations of condition sets consistently demonstrate the outcome. If all studies within a truth table row demonstrate improved adherence, then that row is coded as fully in or 1 with a consistency of 100 %. Rarely do real-world phenomena exhibit perfect consistency. In QCA, rows with a consistency of less than 100 % (also referred to as contradictory rows) can still be coded as 1 and included in sufficiency analyses if row consistency is above a prespecified level. Different thresholds for consistency can be used based on the nature of the research question, data quality, and number of cases, but typical thresholds are between 75 and 90 % [ 21 ].

Using the truth table created for each analysis, we identified set relationships between condition sets and configurations of condition sets and the outcome set. As described in the supplemental online materials (Additional file 1 ), superset relationships between condition sets and an outcome set can be interpreted as indicating necessary conditions. Similarly subset relationships between condition sets and an outcome set can be interpreted as indicating sufficient conditions. We used Stata Version 13 (StataCorp, College Station, TX) to create 2 × 2 contingency tables using set membership values for each condition set and the outcome set. Data from these tables are interpreted through a set-theoretic lens, meaning that the proportions produced by the table are interpreted as the consistency of each condition as a necessary condition for the outcome (% of cases in the outcome set that are also in the condition set) or as a sufficient condition for the outcome (% of cases in the condition set that are also in the outcome set). In the first analysis, we identified one BCT (techniques that increase knowledge) as individually necessary and one BCT (techniques that increase self-efficacy) as individually sufficient; in the second analysis, we did not identify any individually necessary or sufficient conditions.

Though an assessment of individually necessary or sufficient conditions is the initial analytic step, it is the evaluation of configurations of condition sets that allows QCA to offer powerful insights into complex causal patterns. For a configuration of condition sets to be necessary, it would need to be consistently present among all studies with the outcome of “improved medication adherence.” We did not identify two or more individual necessary condition sets in either analysis, and because formal logic prescribes that no configuration can be considered necessary unless each individual component condition set is necessary, we quickly discerned that we would not need an assessment of necessary configurations.

We used fsQCA version 2.5 to conduct sufficiency analyses for configurations [ 26 ]. In crisp-set QCA, the configuration of set membership values in each row of the truth table where the outcome set is 1 represents as expression of sufficiency. In other words, if the outcome is consistently present among cases within the row, then that unique combination of condition sets (i.e., presence or absence of conditions in a crisp-set scheme) is a sufficient pathway to the outcome. If multiple truth table rows consistently demonstrate the outcome, then multiple sufficient pathways are present (i.e., an equifinal solution). The most complex expressions of sufficiency can be taken directly from truth table rows; however, these statements are often unwieldy in the number of conditions and operator terms (ANDs, ORs, NOTs), which makes them difficult to interpret. These expressions can be logically minimized to simpler expressions with fewer terms and operators that are still logically consistent with the more complex expression, but easier to interpret.

The fsQCA software uses the Quine-McCluskey algorithm to perform this minimization procedure. The basis of this minimization procedure is that if two truth table rows with the same outcome differ in set membership value of only one condition set, then that condition set is irrelevant for producing the outcome in that row and can be eliminated. The two rows can be merged resulting in a simpler expression of sufficiency. This algorithm is repeated such that all truth table rows are compared and reduced until no further simplification is possible. In actuality, three variants of the minimization procedure are used to produce three variants of a solution, the conservative, the intermediate, and the parsimonious solutions. These three solutions are all logically consistent with each other but represent different degrees of parsimony and differ with respect to whether logical remainders are used as part of the minimization procedure.

Ultimately, we identified seven sufficient configurations in the intermediate solution for the first analysis and four sufficient configurations for the second analysis. A summary of these results is in Tables  2 and 3 . We computed parameters of fit to describe how well the set relationships we identified deviate from a perfect set relationship (i.e., consistency) and how well the solutions identified explain the outcome across all empiric cases included (i.e., coverage). See the online supplementary materials (Additional file 1 ) for additional information regarding parameters of fit.

Make sense of the results

We examined the studies covered by configurations in the identified solutions to narratively describe how these solutions were represented within a study and across studies for each analysis. The process of relating solution findings back to the studies was instructive for identifying the need for adjustments in condition set calibration. This process also helped us to think beyond numeric coverage levels when considering the relevance of the various configurations to the outcome that we identified. For example, in the first analysis, we found configurations that included the absence of various BCTs to be less interpretable than configurations mostly characterized by the presence of BCTs since interventions are not typically designed to explicitly exclude a BCT. Similarly, the process of re-reviewing the studies in light of the solutions they exemplified allowed us to reconsider the relevance of the knowledge BCT condition set, which we had identified as individually necessary. This condition was present in 57 of the 60 studies we used for the QCA and was generally exhibited within studies as providing patients with information about their disease, the medication used to treat, and benefits and side effects of treatment. Thus, membership in the knowledge BCT set was heavily skewed, and knowledge would likely be a necessary condition of whatever outcome set we defined, a concept described by QCA experts as a “trivial” necessary condition [ 12 ]. Lastly, in keeping with standards of good QCA practice, we repeated all analyses for the set of studies ( N  = 26) not demonstrating improved adherence [ 19 ].

We used QCA within a systematic review to identify combinations of BCTs and combinations of implementation features found among effective medication adherence interventions. The 40 strength of evidence grades in the completed review provided readers with a synthesis of the magnitude and direction of effect for 40 small groups of studies, each group characterized by the same clinical condition and type of intervention [ 16 ]. The QCA results we identified complement the completed review findings by synthesizing across the boundaries of clinical condition and typology to identify combinations of BCTs and implementation features present among the entire set of effective interventions. The QCA findings are not a replacement for the findings in the completed review; rather, they provide additional insights based on configurational questions. Configurational questions are often not formulated as review key questions or the evidence is deemed insufficient to answer such questions for a variety of reasons—for example, lack of trials with direct comparisons of various different intervention features. Yet, “what is the recipe for effectiveness?” is often the information that practitioners and policy-makers want to know when complex interventions and their outcomes are heterogeneous.

We judged QCA to be suitable for use within systematic reviews based on the similarity of processes that are already part of a typical evidence synthesis. In Table  4 , we provide our assessment of the alignment between systematic review and QCA steps, specifically the identification of studies/cases to include, data collection, study/case assessment, analysis, and presentation of findings. Our retrospective application of the method was inefficient, requiring re-review of the original studies at various steps in the process. However, a retrospective approach was invaluable for identifying challenges and steps that might be required beyond a typical review process in order to apply QCA. Although we identified alignment at a number of steps, how best to present findings within the review deserves further prospective evaluation.

The alignment between systematic review processes and QCA at the study/case assessment step deserves highlighting because of the importance of this step for fidelity to standards of good QCA practice [ 21 ]. The distinction between the abstraction tasks of transcribing information from studies into evidence tables and making judgments about the use of various BCTs or implementation features based on information in the studies was not well defined during the original review. Calibration of sets for QCA requires a clear rubric for making set membership value assignments and a mechanism for recording the rationale for the assignment, similar to the approach used for risk of bias assessments. Making set membership value assignments in tandem with data abstraction may be efficient; however, calibration rubrics cannot always be determined a priori, and the familiarity with studies gained through abstraction may be helpful for finalizing the rubric. Even the most robust calibration processes may not overcome the paucity of information about intervention components, implementation features available in published study reports. We believe this may be the biggest challenge to applying QCA and encountered this issue in both our substantive analyses. Ultimately, enough information about the study needs to be available to support the set membership value assignment, though sensitivity analyses could mitigate the impact of missing information.

We identified several other applications of QCA within systematic reviews. To date, all applications of QCA to systematic reviews have been published and presented in separate manuscripts, and not as part of the main evidence report. Using data from a subset of studies in a review of community engagement interventions for public health and health promotion, Thomas and Brunton et al. applied QCA to identify which combinations of community engagement methods directed toward pregnant or new mothers were effective for promoting breastfeeding [ 13 , 27 ]. Although this study had limited diversity and low solution coverage, the investigators could derive additional meaning from the analysis that went beyond the initial qualitative synthesis. We agree with these authors’ assertions about the challenge of finding the right balance between parsimony and complexity when defining condition sets. Candy et al. used QCA with a completed Cochrane systematic review to explore relationships between what patients identify as important components of interventions to improve medication adherence for chronic clinical conditions with what components are actually represented within effective interventions [ 14 ]. The authors discuss the challenge with the selection and processing of data that is far removed from its primary source by the time it appears in a systematic review, a challenge we also acknowledge and had not previously encountered in our use of QCA within primary research studies. We concur with the observations of both study authors regarding the lack of intervention detail reported in primary studies limiting the robust application of QCA within a systematic review context.

Our experience is limited to conducting two analyses within the same completed systematic review. Whether QCA is feasible and adds value within reviews that include a smaller or larger numbers of studies or a review that includes many different outcomes or studies where interventions are complex but do not have easily discernible components is uncertain. The extent to which this method could be applied to other systematic reviews of complex interventions is determined by a number of factors, some based on requirements of the method itself. For example, variability in the outcome is essential to this method; we selected the medication adherence review to apply QCA in part because studies in the review included interventions with demonstrated effectiveness and interventions where effectiveness was not demonstrated. Lastly, our study did not evaluate how to present and integrate results from QCA within a traditional qualitative or quantitative review in a way that minimizes the need for an in-depth understanding of the method, yet provides enough transparency for readers to judge the validity and reliability of the findings.

We offer several recommendations for use of this method in systematic reviews. First, ensure some of the review research questions are configural and based on an a priori understanding of the phenomenon under evaluation. Reviews with fewer than ten studies may not be good candidates for QCA because no more than two to three condition sets can be accommodated without creating substantial limited diversity and patterns among condition sets may just as easily identified by “eye-balling.” Finally, we recommend initial calibration rubric design prior to study abstraction for efficiency, but teams should plan to re-specify and re-review studies if needed before making final calibration decisions.

In conclusion, QCA offers systematic reviewers an additional tool for evidence synthesis in reviews of complex interventions. Further prospective use of the method during a review is needed to identify further areas for process alignment, method refinement, and how best to integrate and present results from a QCA into a typical evidence synthesis report.

Abbreviations

Agency for Healthcare Research and Quality

behavioral change technique

human immunodeficiency virus/acquired immunodeficiency syndrome

patient, intervention, comparator, outcome, timing, and setting

qualitative comparative analysis

randomized controlled trial

United Kingdom

Burton C. Heavy tailed distributions of effect sizes in systematic reviews of complex interventions. PLoS ONE. 2012;7(3):e34222. doi: 10.1371/journal.pone.0034222 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694–6.

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21–34. doi: 10.1258/1355819054308530 .

Article   PubMed   Google Scholar  

Rehfuess E, Akl E. Current experience with applying the GRADE approach to public health interventions: an empirical study. BMC Public Health. 2013;13(1):9.

Article   PubMed   PubMed Central   Google Scholar  

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. doi: 10.1136/bmj.a1655 .

Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281–3. doi: 10.1136/bmj.39569.510521.AD .

Guise JM, Chang C, Viswanathan M, Glick S, Treadwell J, Umscheid CA, et al. Agency for Healthcare Research and Quality Evidence-based Practice Center methods for systematically reviewing complex multicomponent health care interventions. J Clin Epidemiol. 2014;67(11):1181–91. doi: 10.1016/j.jclinepi.2014.06.010 .

Alexander JA, Hearld LR. Methods and metrics challenges of delivery-system research. Implementation Science: IS. 2012;7:15. doi: 10.1186/1748-5908-7-15 .

Ragin CC. The comparative method: moving beyond qualitative and quantitative strategies. Berkeley: University of California Press; 1987.

Google Scholar  

Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10(1):45–53.

Ragin CC. Redesigning social inquiry: fuzzy sets and beyond. Chicago: University of Chicago Press; 2008.

Book   Google Scholar  

Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis. Strategies for social inquiry. Cambridge: Cambridge University Press; 2012.

Brunton G, O’Mara-Eves A, Thomas J. The ‘active ingredients’ for successful community engagement with disadvantaged expectant and new mothers: a qualitative comparative analysis. J Adv Nurs. 2014;70(12):2847–60. doi: 10.1111/jan.12441 .

Candy B, King M, Jones L, Oliver S. Using qualitative evidence on patients’ views to help understand variation in effectiveness of complex interventions: a qualitative comparative analysis. Trials. 2013;14:179. doi: 10.1186/1745-6215-14-179 .

Viswanathan M, Golin CE, Jones CD, Ashok M, Blalock S, Wines RCM, Coker-Schwimmer EJL, Grodensky CA, Rosen DL, Yuen A, Sista P, Lohr KN. Medication adherence interventions: comparative effectiveness. Closing the quality gap: revisiting the state of the science. Evidence Report No. 208. (Prepared by RTI International–University of North Carolina Evidence-based Practice Center under Contract No. 290-2007-10056-I.) AHRQ Publication No. 12-E010-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2012. www.effectivehealthcare.ahrq.gov/reports/final.cfm .

Viswanathan M, Golin CE, Jones CD, Ashok M, Blalock SJ, Wines RCM, et al. Interventions to improve adherence to self-administered medications for chronic diseases in the United States: a systematic review. Ann Intern Med. 2012;157(11):785–95. doi: 10.7326/0003-4819-157-11-201212040-00538 .

Osterberg L, Blaschke T. Adherence to medication. N Engl J Med. 2005;353(5):487–97. doi: 10.1056/NEJMra050100 .

Article   CAS   PubMed   Google Scholar  

Li T, Puhan MA, Vedula SS, Singh S, Dickersin K, Ad Hoc Network Meta-analysis Methods Meeting Working G. Network meta-analysis-highly attractive but more methodological research is needed. BMC medicine. 2011;9:79. doi: 10.1186/1741-7015-9-79 .

Kahwati L, Viswanathan M, Golin CE, Kane H, Lewis M and Jacobs S. Identifying configurations of behavior change techniques in effective medication adherence interventions: a qualitative comparative analysis. Sys Rev, 2016, In press.

Thiem A, Duşa A. Boolean minimization in social science research: a review of current software for qualitative comparative analysis (QCA). Soc Sci Comput Rev. 2013;31(4):505–21. doi: 10.1177/0894439313478999 .

Article   Google Scholar  

Schneider CQ, Wagemann C. Standards of good practice in qualitative comparative analysis (QCA). Comp Sociol. 2010;9(3):397–418.

Kane H, Lewis MA, Williams PA, Kahwati LC. Using qualitative comparative analysis to understand and quantify translation and implementation. Translational Behavioral Medicine. 2014;4(2):201–8. doi: 10.1007/s13142-014-0251-6 .

Kahwati LC, Lewis MA, Kane H, Williams PA, Nerz P, Jones KR, et al. Best practices in the Veterans Health Administration’s MOVE! Weight management program. Am J Prev Med. 2011;41(5):457–64. doi: 10.1016/j.amepre.2011.06.047 .

de Bruin M, Viechtbauer W, Schaalma HP, Kok G, Abraham C, Hospers HJ. Standard care impact on effects of highly active antiretroviral therapy adherence interventions: a meta-analysis of randomized controlled trials. Arch Intern Med. 2010;170(3):240–50. doi: 10.1001/archinternmed.2009.536 .

Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol. 2008;27(3):379–87. doi: 10.1037/0278-6133.27.3.379 .

Ragin CC, Drass KA, Davey S. Fuzzy-set/qualitative comparative analysis. 25th ed. Tucson: Department of Sociology, University of Arizona; 2006.

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67. doi: 10.1186/2046-4053-3-67 .

Download references

Acknowledgements

This work was supported by grant 1R03HS022563-01 from the Agency for Healthcare Research and Quality (AHRQ) to Dr. Kahwati. AHRQ had no role in the study design, data collection, analysis, or interpretation of findings. Part of Dr. Golin’s time was supported by the University of North Carolina at Chapel Hill Center for AIDS Research (CFAR), an NIH funded program P30 AI50410.

Author information

Authors and affiliations.

RTI International, 3040 E. Cornwallis Rd., Research Triangle Park, NC, 27709, USA

Leila Kahwati, Sara Jacobs, Heather Kane, Megan Lewis & Meera Viswanathan

Departments of Medicine and Health Behavior, University of North Carolina, Chapel Hill, NC, USA

Carol E. Golin

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Leila Kahwati .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

ML, MV, and LK conceived of the study, and LK secured the funding. LK, ML, MV, HK, and CG designed both analyses, and SJ had substantial intellectual contribution to the revisions for analysis 2. MV and CG contributed to the data collection in the original, completed review. LK and SJ performed the configural analyses. LK drafted the initial manuscript, and all authors critically reviewed the manuscript and approved the final version.

Additional files

Additional file 1:.

Detailed description, example, and glossary. This file provides a detailed description of the historical context of QCA, methodologic foundations, and walks through a hypothetical example analysis to illustrate key features of the methods. A glossary of terms is also included. (PDF 521 kb)

Additional file 2:

Completed review analytic framework, key questions, and summary of findings. This file provides the analytic framework, key questions, and a summary of findings from the completed AHRQ systematic review of interventions to improve medication adherence that was used to examine the suitability of using QCA in a review of a complex intervention. (PDF 156 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Kahwati, L., Jacobs, S., Kane, H. et al. Using qualitative comparative analysis in a systematic review of a complex intervention. Syst Rev 5 , 82 (2016). https://doi.org/10.1186/s13643-016-0256-y

Download citation

Received : 06 August 2015

Accepted : 25 April 2016

Published : 04 May 2016

DOI : https://doi.org/10.1186/s13643-016-0256-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative comparative analysis
• Configurational analyses
• Systematic review methods

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley-Blackwell Online Open

Reporting of test comparisons in diagnostic accuracy studies: A literature review

Yasaman vali.

1 Department of Epidemiology and Data Science, Amsterdam UMC, University of Amsterdam, Amsterdam The Netherlands

Maria Olsen

Mariska m. g. leeflang, patrick m. m. bossuyt, associated data.

Table S1. Search strategy for the overview of comparative accuracy systematic reviews.

Table S2. Reporting items for comparative accuracy studies developed for this literature review.

Table S3. List of included comparative accuracy studies.

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Comparative accuracy studies evaluate the relative performance of two or more diagnostic tests. As any other form of research, such studies should be reported in an informative manner, to allow replication and to be useful for decision‐making. In this study we aimed to assess whether and how components of test comparisons were reported in comparative accuracy studies. We evaluated 100 comparative accuracy studies, published in 2015, 2016 or 2017, randomly sampled from 238 comparative accuracy systematic reviews. We extracted information on 20 reporting items, pertaining to the identification of the test comparison, its validity, and the actual results of the comparison. About a third of the studies ( n  = 36) did not report the comparison as a study objective or hypothesis. Although most studies ( n  = 86) reported how participants had been allocated to index tests, we could often not evaluate whether test interpreters had been blinded to the results of other index tests ( n  = 40; among 59 applicable studies), nor could we identify the sequence of index tests ( n  = 52; among 90 applicable studies) or the methods for comparing measures of accuracy ( n  = 59). Two‐by‐four table data (revealing the agreement between index tests) were only reported by 9 of 90 paired comparative studies. More than half of the studies ( n  = 64) did not provide measures of statistical imprecision for comparative accuracy. Our findings suggest that components of test comparisons are frequently missing or incompletely described in comparative accuracy studies included in systematic reviews. Explicit guidance for reporting comparative accuracy studies may facilitate the production of full and informative study reports.

What is already known?

• Transparent reporting of studies is essential to allow readers to appreciate study findings and limitations.

What is new?

• Comparative accuracy studies (a specific type of diagnostic accuracy studies that evaluate and compare the accuracy of two or more tests) frequently fail to report, or incompletely report information about comparisons of index tests that helps study identification, validity assessment, and interpretation of results.

Potential impact for RSM readers?

• Incomplete reporting of comparative accuracy studies will complicate their appraisal and synthesis in diagnostic test accuracy systematic reviews. There is a clear need for more informative study reports, which could be facilitated through the development of explicit reporting guidelines specifically for comparative accuracy studies.

1. INTRODUCTION

Diagnostic accuracy studies provide information on the performance of a test in accurately distinguishing individuals with and without a target condition. Such studies can be focused on a single index test, but they can also evaluate two or more index tests for detecting the same target condition and compare their accuracy. 1 , 2 Well‐designed comparative accuracy studies can provide valuable evidence to clinicians and policy‐makers, helping them to select the optimal test for patients among competing tests.

As other clinical studies, comparative accuracy studies should be reported in an informative and reproducible way to allow the reader to evaluate the validity of the study, to appreciate the study findings, and to consider their applicability to other patient groups and settings. 3 , 4 , 5 , 6 Deficiencies in reporting not only can lead to incorrect conclusions and make decision‐making difficult, it is also a source of avoidable research waste and, as such, a threat to evidence‐based medicine. 7 , 8

While all diagnostic accuracy studies need to be transparently reported, comparative accuracy studies face an added reporting challenge. Authors of comparative accuracy studies should not only report details of each index test under investigation, but also describe how the index tests were compared to each other. They have to specify the design and methodology of their comparison in a transparent and reproducible manner and report the comparative accuracy results in such a way that statistical inference regarding the relative performance of the tests is possible.

Existing guidance for reporting diagnostic studies has no specific instructions for comparative accuracy studies. 3 Previous evaluations of the informativeness of reports of diagnostic accuracy studies largely focused on single test evaluations, without targeting comparative accuracy studies. 9 , 10 We evaluated published reports of recent comparative accuracy studies to evaluate whether and how components of test comparisons were described.

2. OBJECTIVES

We aimed to examine the reporting characteristics of comparative accuracy studies and assess whether information on identifying the comparison, aspects of validity, and results of the comparison were adequately reported.

3.1. Study design

This study is a literature survey of comparative accuracy studies. The study protocol was made available through the Open Science Framework ( https://osf.io/72xpy ).

3.2. Data sources

For the purpose of this study, comparative accuracy studies were sampled from studies included in systematic reviews that had compared the accuracy of two or more tests. We selected these systematic reviews from an existing overview of 238 comparative accuracy systematic reviews published between 2017 and 2018. 11 Briefly, the overview included all systematic reviews including a comparison between the accuracy of index tests indexed in MEDLINE in 2017. The search strategy for this overview is provided in Table S1 in Data S1 .

3.3. Eligibility criteria and study selection

Eligible were all comparative accuracy studies on humans. We defined a comparative accuracy study as a study that (1) evaluates the accuracy of two or more index tests and (2) for which the published study report contains at least one statement in which the accuracy of these index tests is being compared. For assessing whether such a statement is present, we looked for comparative language, that is, terms as “comparison”, “comparative”, “higher/lower”, “superior/inferior”, “better/best/worst/worse”, and “more/most”.

One reviewer retrieved all references to primary studies included in the 238 systematic reviews. We then applied a filter based on year of publication. We restricted inclusion to comparative accuracy studies published in 2015, 2016, or 2017, to evaluate recent practice. From the primary studies of these 3 years, we randomly selected published study reports and evaluated eligibility. Each study was assigned a random study number, using a random number generator on Google Sheets software (Google, Mountain View, California, U.S.). The studies were evaluated for eligibility starting with the lowest study numbers, until 100 primary comparative accuracy studies were included. As there is no widely accepted sample size calculation for our type of methodological review, the authors agreed to sample 100 studies for feasibility reasons. Evaluation of eligibility was done in two steps: first based on the title and abstract, and then the full‐text article. Each study report was assessed by two independent assessors for eligibility. Disagreements were resolved by consensus, or by consulting a senior author. We excluded non‐English language studies.

3.4. Data extraction

For each comparative accuracy study, two independent assessors looked for key reporting items regarding comparisons in the body of the full‐text report. Study abstracts were not assessed. In the absence of a specific reporting guidance for comparative accuracy studies, we developed a set of items which we deemed specifically relevant to comparative accuracy studies (Table S2 in Data S1 ). An initial list of potentially relevant items was developed by two authors (Y.V. and B.Y.) through brainstorming and consultation of existing reporting guidelines. This list was subsequently reviewed and revised iteratively by the co‐authors. The items in the final list were largely adapted from the items in STARD 2015 (reporting guidelines for diagnostic accuracy studies), 3 many of which were directly applicable to comparative accuracy studies (See Table S2 in Data S1 for the source of each item). In addition, the QUADAS‐C Delphi study for developing a risk of bias tool for comparative accuracy studies produced a number of items related to potential bias. 12 Lastly, items from CONSORT 2010 (reporting guideline for parallel group randomized trials) were adapted for comparative accuracy studies that use random allocation. 13

A comparative accuracy study may contain more than one test comparison. Some studies evaluate large numbers of index tests, with the possibility of presenting numerous comparisons. We therefore made an additional restriction, by focusing exclusively on the first comparison reported in the article. For example, if a study had evaluated five index tests and reported 10 pairwise comparisons, one by one, the first pair reported would be considered to be the first comparison.

3.5. Data analysis

We used descriptive statistics to summarize the results. We presented the number of studies that reported a particular item, accompanied by examples how these items were reported (if applicable).

4.1. Search results

From the 238 systematic reviews of comparative accuracy, we retrieved 5789 references to primary accuracy studies, of which 946 were published in 2015, 2016, or 2017. We assigned a random number to each of the 946 primary studies and selected an arbitrary number of 321 studies with the lowest assigned numbers for title and abstract screening. We then excluded 176 of 321 studies during this phase. For the remaining 145 studies, we assessed full text articles in random order until we included 100 comparative accuracy studies. Eventually, we assessed the first 113 full text articles in order to include 100 studies (Figure ​ (Figure1 1 ).

Flow diagram of included studies [Colour figure can be viewed at wileyonlinelibrary.com ]

4.2. Characteristics of included studies

The characteristics of included comparative accuracy studies are described in Table ​ Table1. 1 . Fifty‐nine studies were published in 2015, 36 studies in 2016, and 5 studies in 2017. A wide range of target conditions were evaluated, the most frequent being neoplasms ( n = 54), digestive system disorders ( n = 14), and infectious diseases ( n = 10). In almost half of the studies, the index tests in the comparisons were biochemical tests ( n = 50) and imaging modalities ( n = 47). In 49 studies the first comparison reported in the article was between two index tests, while in the other 47 studies the first comparisons consisted of three or more index tests. This was unclear in four studies.

Characteristics of 100 included comparative accuracy studies

4.3. Reporting of components of a test comparison

4.3.1. identifying the comparison.

We summarized reporting items that could help to identify comparative accuracy studies in Table ​ Table2. 2 . The comparative nature of the study could not be identified in the titles of most of the studies ( n = 73). Of those that identified their study as a comparative accuracy study in the title ( n = 27), 6 used a study label indicating a comparison (such as “comparative study” or “prospective randomized study”) and 21 studies indicated the comparison otherwise (e.g., using comparative language such as “superior” or “outperformed”). More than half of the studies reported the test comparison as part of their objectives ( n = 60) or hypothesis ( n = 3) or both ( n = 1), while about a third ( n = 36) did not mention test comparison as a specific objective or hypothesis.

Frequency of studies that reported a particular item for identifying the comparison

The majority ( n = 66) stated which index tests were being compared, before reporting the performance of the tests in the results section of the article. In the remaining 34 studies, the index tests within the comparison were found after the methods section: in the results, tables, figures, or discussion section.

4.3.2. Information relevant to the validity of the comparison

Table ​ Table3 3 summarizes the results for items related to the validity of the comparison. In 16 studies including composite index tests, only 5 reported the criterion for test positivity. The majority of studies did not report whether participants were either consecutively or randomly sampled ( n = 62). Only 34 studies reported this explicitly by using the words “consecutive” or “random”; four studies reported this by describing the sampling process.

Frequency of studies that reported a particular item relevant to the validity of the comparison

In 14 studies, it was not clear how the participants were allocated to each index test. Of the 86 studies that described the allocation method only 34 did so explicitly (e.g., by stating “participants were screened using all three cognitive measures” 14 ), while for 52 studies this was inferable from a description provided in the methods, or from figures or tables.

We found 90 studies with a paired design, in which a participant received two or more index tests (for nine of these studies the allocation was not clearly reported, but it became clear that at least some participants received multiple index tests based on other descriptions in the methods or results section). In such paired studies, each index test should ideally be interpreted blinded from the other index test results, if the index test involves subjective interpretation. We judged that one or more index tests in the comparison may involve subjective interpretation in 59 paired studies. However, only 19 of 59 studies reported whether blinding was implemented, by using the word “blinding” or similar wording ( n = 7), for example by declaring that interpreters were not aware of other test results, 15 or by describing the study process ( n = 12). An example of the latter was a comparison of endoscopy techniques, each interpreted by a single endoscopist. 16 Information on blinding was missing from 40 of 59 study reports.

Only 38 of the 90 paired studies reported the sequence of index tests performed on each participant. Most often there was a fixed order for all participants. For example, by reporting the exact sequence of three cognitive tests performed on each participant. 14 One study reported a fixed test order for one subgroup and a reversed order for a second subgroup. 17

The time interval between the index tests was not specified in approximately half of the study reports ( n = 47). Of 53 studies that reported this item, 15 described it explicitly, for example, by reporting the median number of days. For 38 studies this was inferable from other information in the study report. An example of the latter was a study in which all biomarkers were tested in the same blood sample. 18

The majority of studies failed to report the time interval between the index tests and the reference standard ( n = 69). One study reported this only for one of the two index tests, while the other 30 studies stated this for each of the index tests in the comparison.

Ideally, a single, preferred (best available) reference standard should be used to verify all index test results. In 18 studies, two or more reference standards were used. We examined in these 18 studies how the reference standard was chosen for each participant. Six studies reported that the choice depended on index test results ( n = 2), on a third test not in the comparison ( n = 3), or on clinical indication ( n = 1). In 12 studies, the choice for the reference standard was not explained.

We examined whether methods for comparing diagnostic accuracy estimates, statistical or other, were described in the article (e.g., McNemar's test statistic for paired data, or tests for differences in the area under the receiver operating characteristic [ROC] curve). Such methods were reported only by a minority of studies ( n = 41), and absent in the 59 other study reports.

Of all 100 evaluated studies, only 16 provided a participant flow diagram. Of these, seven studies did not include any of the index tests in the flow diagram.

When studies were not fully paired, that is, partially paired or unpaired ( n = 6), we examined whether studies reported baseline characteristics of participants (at least age and gender). Two studies reported baseline characteristics for each index test group, thereby allowing the reader to assess the comparability of the index test groups. However, the remaining studies either reported baseline characteristics for the entire group of participants ( n = 2) or did not report them at all ( n = 2).

4.3.3. Results of the comparison

Table ​ Table4 4 shows the number of comparative accuracy studies that reported information on the results of the actual comparison. Not all studies reported sufficient data for construction of two‐by‐two tables (index test against reference standard). Such data were reported by only 35 studies; 33 studies provided data for each index test in the comparison, while two studies provided data for only some of the index tests.

Frequency of studies that reported a particular item relevant to the results of the comparison and limitations

We also examined whether data for the construction of two‐by‐four tables 19 (tables that cross‐classify the results of two index tests being compared within diseased and non‐diseased participants; Table ​ Table5) 5 ) were reported by paired accuracy studies. This was the case in only 9 of 90 paired studies.

Joint classification of index tests and reference standard results in a paired design (also called two‐by‐four table), adapted from Reference 19

Only four studies reported their findings using a measure of comparative accuracy. Reported measures were difference in sensitivity and difference in specificity ( n = 2), odds ratio of sensitivity and odds ratio of specificity ( n = 1) or difference in the area under the ROC curves ( n = 1). The others ( n = 96) did not report measures of comparative accuracy. Expressions of statistical uncertainty for the comparisons were reported in 36 studies, either as p‐values ( n = 33), confidence intervals ( n = 1) or both ( n = 2). The 64 other studies reported the comparison without statistical uncertainty, while 10 of those studies explicitly reported that they used a statistical method for comparing diagnostic accuracy estimates.

Potential limitations regarding the test comparison were mentioned in a minority of study reports ( n = 18). For example, one study that compared the accuracy of 22‐gauge versus 25‐gauge needles for diagnosing pancreatic tumors reported that the endoscopists were not blinded to needle type. 20 In another study that compared the accuracy of two MRI‐based scoring systems for prostate cancer diagnosis, authors admitted that MRI readers assigned both scores in a single session, thereby introducing the possibility that scores for one scheme could have influenced the other. 21 Although we expected from even an almost perfect study to discuss at least one potential or true limitation, 82 studies did not mention any such limitations.

5. DISCUSSION

5.1. summary of findings.

Well‐conducted comparative accuracy studies, while scarce, have the potential to yield high‐certainty evidence for informing clinical decision making regarding tests. 1 Considering their importance, they should be reported in sufficient detail, to allow readers to appreciate their findings.

Our findings suggest that the reporting of items to identify the comparison, to assess the validity, and to interpret the results of the comparison is suboptimal in comparative accuracy studies. Many of the items were missing or incompletely reported. Even when information on a particular item could be identified, such as allocation method, this could often only be inferred indirectly, from more general descriptions of study processes, or tables and figures.

Measures of comparative accuracy were rarely used, and expressions of statistical uncertainty were not available in the majority of study reports. Thus, it was often not clear how the studies had come to the given conclusion about which test was more/most accurate and we suspect that the conclusions were often simply based on the separate accuracy estimates for the respective tests.

5.2. Strengths and limitations

Previous evaluations have highlighted incomplete and ambiguous reporting of diagnostic accuracy studies, 9 but few studies have focused on comparisons of tests. A recently published methodological survey showed poor reporting of systematic reviews that had evaluated and compared multiple tests. 22 To our knowledge, no previous study assessed the reporting of information on test comparisons in comparative accuracy studies.

Our review has a number of limitations. First, there is currently no agreed definition of comparative accuracy studies. The definition used in this survey is a “multiple test accuracy study with a comparative statement”, which others may disagree with. Second, given our limited sample size of 100 studies and sampling method (we only included comparative accuracy studies included in systematic reviews), results from an independent sample of comparative accuracy studies may differ. However, since systematic review authors may have selected studies for inclusion when reporting was sufficient, the number of studies in our sample that reported a particular item (e.g., identification as a comparative study in the title) may even be overestimated rather than underestimated. Third, there may be inherent subjectivity in the assessment of specific reporting items. For instance, a seemingly straightforward question as “was the allocation method reported” was difficult to answer when the authors did not explicitly describe the allocation process. This required a judgment as to whether the different pieces of information in the study report were sufficient to ascertain the allocation method. To minimize subjectivity, we used assessment in duplicate and had internal discussions. Lastly, we assessed the reporting in relation to only the first comparison of each study. Although unlikely, we cannot exclude the possibility that subsequent comparisons were more completely reported.

5.3. Interpretation of findings

The potential implications of poor reporting are wide and serious. Those that search for comparative accuracy studies may spend excessive time and effort due to the lack of appropriate identifiers. The absence of an explicit description of the study design may impede attempts at evaluating the risk of bias of such studies. Incomplete reporting of the results of test comparisons may complicate the incorporation of study data in evidence syntheses and may lead to misinterpretation when translating study findings to clinical practice.

Although we can merely speculate on the causes of poor reporting, one possibility is a low overall awareness of the importance of test comparisons in diagnostic accuracy studies. Investigators may believe that diagnostic accuracy studies (even when multiple index tests are included) should mostly aim to provide reliable estimates of the accuracy of individual tests. Many authors did not state the comparison as a study objective or hypothesis; this was the case in 36 studies in our sample. Despite this, authors still compared the accuracy of index tests and made a comparative statement. We believe that even these casual comparisons should be based on solid evidence. Making comparative statements without a proper description of the study features on which the comparison is based can increase the risk of “spin” or over‐interpretation of the results. Over‐interpretation and misreporting of results are frequent in diagnostic accuracy studies 23 , 24 and using exaggerated language in comparative accuracy studies may not only result in research waste but also cause harm to patients by misleading clinicians to select inappropriate tests.

Another possibility is that investigators are less aware of potential sources of bias in test accuracy comparisons, and therefore could not know which items were essential to report. This would be unsurprising, since methodological research on bias in test accuracy comparisons is limited, compared to numerous publications and instruments to identify sources of bias in estimates of accuracy of a single test. 25 , 26 , 27

Authors should aim to improve the quality of reporting in their studies, by becoming more familiar with different aspects of comparative study designs and appropriately interpreting their findings. Yet they are not the only ones bearing responsibility. To ensure accurate interpretation and dissemination of research, editors and peer reviewers should also be held accountable for identifying any comparative statements and for critically examining whether such claims can be supported by the study design and results.

Providing authors with guidance may help them to improve the reporting of their studies. The STARD statement, which was developed in 2003 and updated in 2015, 3 , 28 is the established reporting guideline for all types of diagnostic accuracy studies. While STARD includes reporting items pertaining to test comparisons, it also lacks some key items, such as participant allocation method. STARD could be extended to address these additional comparative items, and a revised explanation document could highlight the importance of explicit and rigorous test comparisons.

6. CONCLUSIONS

Information about comparisons of index tests that helps study identification, validity assessment, and interpretation of results is missing or incompletely reported in comparative accuracy studies included in systematic reviews. This illustrates a clear need for improvement in the standards of reporting for comparative accuracy studies. Better reporting could be facilitated through the development of explicit reporting guidelines specifically for comparative accuracy studies.

CONFLICT OF INTEREST

All authors declare that there are no conflicts of interest.

AUTHOR CONTRIBUTIONS

Y.V.: Conceptualization, Project administration, Methodology, Investigation, Formal analysis, Writing – Original Draft, Writing – Review & Editing. B.Y.: Conceptualization, Methodology, Investigation, Formal analysis, Writing – Original Draft, Writing – Review & Editing. M.O.: Methodology, Investigation, Writing – Review & Editing. M.M.G.L.: Methodology, Writing – Review & Editing, Supervision. P.M.M.B.: Methodology, Writing – Review & Editing, Supervision.

Supporting information

Data S1. Supporting information.

ACKNOWLEDGEMENTS

This work was supported by the LITMUS (Liver Investigation: Testing Marker Utility in Steatohepatitis) study. The LITMUS study is funded by the Innovative Medicines Initiative (IMI2) Program of the European Union (Grant Agreement 777377).

Vali Y, Yang B, Olsen M, Leeflang MMG, Bossuyt PMM. Reporting of test comparisons in diagnostic accuracy studies: A literature review . Res Syn Meth . 2021; 12 :357–367. 10.1002/jrsm.1469 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

DATA AVAILABILITY STATEMENT

Reviews of Peer-Reviewed Journals in the Humanities and Social Sciences

We give you the scuttlebutt on academic journals—aiding you in selecting the right journal for publication—in reviews that are sometimes snarky, sometimes lengthy, always helpful. Written by Princeton University graduate students and Wendy Laura Belcher.

• Journal Article Prize Competitions
• Journal Disciplines
• Journals Reviewed
• Journals Ranking

Comparative Literature Studies

For those interested in publishing articles of a comparative nature on literary history, the history of ideas, critical theory, studies between authors, and literary relations within and beyond the Western tradition.

As their web description states, “Comparative Literature Studies publishes comparative critical articles that range across the rich traditions of Africa, Asia, Europe, and North and South America, and that examine the literary relations between East and West, North and South. Articles may also explore movements, themes, forms, the history of ideas, relations between authors, the foundations of criticism and theory, and issues of language and translation.”

Compared to some other journals that strive to be comprehensive and yet fall short of achieving it, the CLS is successful in covering diverse regions and topics that gives an exciting wide scope view of the world of Comparative Literature.  Their selection of articles covers a wide range of topics pertaining to literature, language, and literary theory; from Shakespeare to Internet Literature in China.  Also, the regions covered in the journal range from Africa, Asia, and to the Americas– which is against the typical Euro-centricism often seen in the field of Comparative Literature.

The length of an article varies from the minimum 15 pages to over 45, perhaps for this reason, the number of articles appearing in each issue varies greatly.  The journal publishes a number of co-authored articles, which may or may not be due to the comparative nature of articles.   The journal also features papers written by graduate students that had previously won the A. Owen Aldridge prize that is given by the ACLA. In an issue of the third quarter of 2016, for example, the journal featured such a paper that won the prize back in 2014.

Share this:

• Click to share on Facebook (Opens in new window)
• Click to share on Twitter (Opens in new window)
• Click to share on Tumblr (Opens in new window)
• Click to email a link to a friend (Opens in new window)
• Click to print (Opens in new window)
• Click to share on Reddit (Opens in new window)

About Wendy Belcher

Information

Recent posts.

• L’Esprit Createur
• Latino Studies
• Modern & Contemporary France
• Ramus: Critical Studies in Greek and Roman Literature
• The Black Scholar

Informal street vending: a comparative literature review

International Journal of Sociology and Social Policy

ISSN : 0144-333X

Article publication date: 29 September 2020

Issue publication date: 23 June 2021

Informal street vending is traditionally widespread and studied concerning developing countries. Nevertheless, recently, interest in the study of this practice has also increased regarding specific developed countries. The aim of the article is to contribute to overcoming the tendency to investigate this informal economy sector with different analytical lenses between the global South and global North and to highlight the usefulness of analyzing the phenomenon from a comparative perspective.

Design/methodology/approach

Therefore, the article represents a comparative review of the existing literature on informal street vending considering both the global South and global North.

The analysis revealed similarities and differences in the characteristics the phenomenon assumes in the two areas of the world while at the same time, showing how there are aspects mainly explored in the literature of southern countries and little explored in the literature of northern countries and vice-versa.

Research limitations/implications

This analytical attempt allows us to highlight any gaps present in the literature, which may represent the basis for future comparative research on the topic. Comparative research will improve both theoretical and empirical knowledge of the phenomenon.

Originality/value

On the one hand, the article represents an innovative literature review attempt, as it explicitly compares the street vending between developing and developed countries. On the other hand, it represents the first academic contribution to review street vending in the global North.

• Informal economy
• Street vendors
• Global South and global North
• Urban policies
• Strategies of resistance

Recchi, S. (2021), "Informal street vending: a comparative literature review", International Journal of Sociology and Social Policy , Vol. 41 No. 7/8, pp. 805-825. https://doi.org/10.1108/IJSSP-07-2020-0285

Emerald Publishing Limited

Copyright © 2021, Sara Recchi

Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

Introduction

Street vending represents one of the most visible manifestations of the informal economy and has been studied for over forty years by various disciplines, especially anthropology, economics and sociology. Street vending is traditionally rooted in the social and economic fabric of many southern countries and, therefore, has mostly been studied concerning Africa (e.g. Steel et al. , 2014 ; Brown et al. , 2010 ), Asia (e.g. Milgram, 2011 ; Schindler, 2014 ) and Latin America (e.g. Crossa, 2009 ; Donovan, 2008 ). Nonetheless, in recent decades, interest in the study of the phenomenon regarding European and North American countries has also increased (e.g. Devlin, 2011 ; Boels, 2014 ). In fact, street vending is no longer considered as a residual activity typical of global South and destined to disappear, but rather as a constantly growing phenomenon, affecting both developed and developing countries ( Graaff and Ha, 2015 ). Despite the knowledge concerning street vending activities and more in general the informal economy has increased, to date there is no a univocal theory in defining the causes and characteristics of the informal economy. The theoretical contradictions are even more evident when one compares this phenomenon between developing and developed countries ( Gerxhani, 2004 ; Chen, 2012 ). Moreover, while the literature on street vending in developing countries is characterized by a great deal of research, still few studies in industrialized countries exist. Besides, while some literature review articles strive to analyze this informal activity in the global South exist (e.g. Forkour et al. , 2017 ; Mitullah, 2004 ), there are no similar contributions concerning developed countries. Furthermore, except for some review articles that adopt a global perspective to study street vending (e.g. Bromley, 2000 ; Cross, 2000 ; Wongtada, 2014 ), there are no review attempts that explicitly compare the phenomenon in the two areas of the world. Therefore, this article represents a comparative literature review on street vending, considering both the southern and northern countries. It also constitutes the first attempt to review street vending in developed countries. The aim is to contribute in overcoming the tendency to investigate this informal economy sector with different analytical lenses between the global South and global North. Moreover, the article also aims to highlight how the comparison across the two institutionally distinct types of countries is fundamental to understand the informal street vending sector dynamics. Although the main focus of the article is on informal actors, it groups different manifestations of street vending. This choice stems from the difficulty of clearly distinguishing between the formal and informal economy as well as the awareness of the varying degrees of informal practices and, in turn, of different violations of trade regulations.

In the first section, the article will highlight the criteria for choosing and analyzing the studies included in the review. Subsequently, the article will present the main research methods and techniques adopted to study the phenomenon. The subsequent sections will discuss the aspects dealt with by the literature, highlighting the similarities and differences between the global South and global North. Therefore, the second passage will focus on street vendors' daily strategies and working conditions and, subsequently, it will highlight the street vendors' profile. The following section will present the informal street vending causes as well as the individual motives that drive to work informally. Subsequently, the article will underline the dimensions through which the literature investigates the link between the formal economy and the informal street trade sector. Then, the urban governance theme will be analyzed, by describing the perceptions behind street vending and the policies and practices adopted to regulate it. Finally, the paper will emphasize the street vendors' strategies to confront exclusionary policies and to negotiate for space. In the last paragraph, a discussion of the main similarities and differences between developing and developed countries will be proposed, supporting theoretical explanations. The article will conclude by presenting the implications for future comparative research on the topic.

Literature review: selection criteria and method of analysis

To select and identify the most relevant studies on the topic, the Google Scholar online database was used. The selection process was guided by specific keywords, which have been reproduced in the following search phrase: informal street vending OR street vendors OR street vendors' practices and working conditions OR street vending urban polices . The search resulted in about 26,300 studies. Then the search was limited to studies published from 2000 to 2019, to present an updated picture of the phenomenon. Following this criterion, the search was reduced to approximately 20,100 studies. Moreover, non-scientific journals were excluded from the analysis. Therefore, the article considered scientific peer review journal articles, book chapters, research reports and working papers. Finally, only empirical studies were included in the analysis. Considering these selection criteria, the unqualified studies were rejected, through the implementation of a screening process. Initially, a large number of studies were eliminated after reading the title, the abstract and the keywords. Subsequently, other studies were excluded after a more detailed full-text screening. After the selection process, only 59 studies were considered: 42 concerning developing countries and 17 regarding developed countries. In the analysis process, each study was read twice, identifying: the main topics and objectives, the research method and techniques and the geographical context. The analysis led to the construction of two tables: one for studies conducted in developing countries ( Table 1 ) and one for those carried out in developed countries ( Table 2 ). The tables summarize the main topics addressed in the literature, which have been included through a codification scheme, based on categories and subcategories of analysis. For instance, in relation to urban policies, the category “ Urban policies and regulations ” has been created, which corresponds to the subcategories: neoliberal and other exclusionary policies; evictions; harassment; displacement policies and relocation; tolerance policies and others regulations . Moreover, for each considered study has also been outlined: the name of the author, the year of publication, the city/country in which the study was conducted, the name of the journal or publishing house and the research techniques adopted.

Review of methodology: doing research on street vending

In both developing and developed countries’ literature, qualitative investigation techniques are predominantly employed to study street selling. Nonetheless, in the literature on developing countries, 21 out of a total of 42 studies adopt qualitative investigation techniques ( Table 1 ). In the literature on developed countries almost all studies, 14 out of 17, resort to participant observation, qualitative interviews or focus groups techniques ( Table 2 ). This methodological tendency can be explained in light of the complexity of studying the informal economy through quantitative techniques, due to the lack of official statistical data. Moreover, almost all the studies considered represent case study research, studying street vending concerning a specific city or neighborhood. This aspect responds to the tendency to implement qualitative techniques, which inevitably affect the empirical context dimensions. Nevertheless, ten cross-national and continental studies, conducted in different developing countries, were also selected. These studies, which consider either several cities in the same country or various cities across different continents, employ a comparative approach to study the phenomenon ( Table 1 ). Despite the tendency to adopt qualitative techniques, a portion of the studies conducted in developing countries implement quantitative investigation techniques, six of them use mixed method techniques and eight resort to survey techniques ( Table 1 ). In the developed countries' literature, this methodological aspect hardly emerges. Indeed, only 3 out of 17 studies adopt quantitative techniques ( Table 2 ). Finally, as regards the geographical composition of the selected studies, 13 of them were conducted in Asia, 12 in Latin America and 7 in Africa. As regards the developed countries, 11 studies were conducted in Europe and 6 in North America.

The street vendors' daily strategies and working conditions

Informal street vending is defined as the production and selling of legal goods and services in urban public spaces, which is not officially regulated by the law and is carried out in non-permanent built structures ( Cross, 2000 ). Although street vending represents an important informal sector, the exact global number of street vendors is unknown. Despite that, official regional statistics and research, which highlight the vast extent of the sector in specific developing counties, exist. For instance, in sub-Saharan Africa, street vendors account from 12 to 14% of the total urban informal employment, in India 14% and in Lima and Peru, street vendors represent 9% of the total informal urban workers ( Roever, 2014 ). Moreover, while in the global South a large segment of street vendors work informally ( Roever, 2016 ), in many developed countries, informal sales activities still represent a minority in relation to the entire sector ( Boels, 2014 ; Blanchard, 2011 ). Traditionally, street vending is defined using the dichotomous distinction between formal and informal, or regular and irregular economic activities. Nevertheless, to highlight the heterogeneity of the phenomenon, in both developing and developed countries literature, the tendency to distinguish the sector according to street vendors' working strategies, prevails. In this regard, three main street vendors' categories can be defined: mobile or itinerant vendors , who undertake their activities by moving to different places throughout the working day; fixed-stall or stationary vendors , who work in a stable place during the working day and semi-fixed street vendors, who momentarily dispose their goods on improvised structures along the street ( Coletto, 2019 ; Boels, 2014 ; Mitullah, 2004 ; Brown et al. , 2010 ; Boonjubun, 2017 ; Cuvi, 2016 ). Each category of street vendors corresponds to a different degree of legal regulation violation, but also to differences in terms of working conditions. Commonly, the street vendors' working conditions are characterized by low incomes, the absence of social security or state benefits, long working hours and unsafe workplace environments ( Eltzon, 2015 ; Saha, 2009 ). Furthermore, several studies highlight how street vendors' working conditions also depend, for instance, on street vendors' sex ( Turner and Schoenberger, 2012 ; Milgram, 2011 ; Munoz, 2016 ) and ethnicity, thus their context of origin ( Martin, 2014 ; DeLuca, 2012 ). Finally, the working conditions of street vendors can also vary regarding the type of goods sold ( Cuvi, 2016 ). Street vendors' daily working conditions and strategies are aspects highly investigated in the recent literature on the phenomenon, emerging in almost all the studies considered, both in developing and developed countries ( Tables 1 and 2 ).

The street vendors' profile

Almost all the selected studies highlight the socio-demographic and economic characteristics of street vendors ( Tables 1 and 2 ). In many developing countries, street vending represents an important source of income mainly for marginalized and poor people, especially for internal rural migrants ( Roever, 2014 ; Onodugo et al. , 2016 ; Turner and Schoenberger, 2012 ; Swider, 2015 ). Moreover, in many developing countries, petty trade is performed mainly by less educated people ( Steel et al. , 2014 ; Turner and Schoenberger, 2012 ). Street vending requires low professional skills and human capital resources, is chosen especially by those who have fewer resources to invest ( Roever, 2014 ). In many developing countries, primarily in Asia and Africa, gender also influences street selling practices. In fact, in many southern countries, women represent a majority of street vendors ( Bhowmik, 2001 ; Milgram, 2011 ; Williams and Gurtoo, 2012 ). Finally, as some studies emphasize, the sector is also comprised by workers who were previously employed in the formal sector and who, in the absence of alternatives, were forced to enter the informal economic circuit ( Milgram, 2011 ; Schindler, 2014 ). In developed countries, instead of internal rural migrants, informal street vending activities are carried out by immigrants. In some North American cities, Latin Americans represent the main categories of migrants who carry out informal street vending activities ( Bhimji, 2010 ; Martin, 2014 ; Munoz, 2016 ). Moving the analysis to the European context, the majority of informal street vendors are from North-African and South-East Asian countries ( L'Hote and Gasta, 2007 ; Blanchard, 2011 ; Harvey, 2004 ; DeLuca, 2012 ). The close relationship between informal street vending and immigrant status highlights how in many developed countries, migrants, mostly irregular, suffer from socio-economic discrimination, which drives them to find work in unskilled segments of the labor market, often in the informal economy ( Raijman, 2001 ; Boels, 2014 ; DeLuca, 2012 ).

Street vending causes and individual motives

Another amount of studies, almost half of those conducted in the northern countries and a half in the southern ones ( Tables 1 and 2 ), investigates the informal street vending causes and the individual reasons that drive to work informally. In many developing countries, the low level of industrialization, the surplus of labor and the process of urbanization, combined with an economic system based on the use of little technology and low-paid unskilled workers, represent the main factors affecting the high presence of the informal economy ( Gerxhani, 2004 ; Roever, 2014 ). Moreover, the ambiguous role of the law and high level of institutional corruption also affect the high impact of informal street vending in many developing countries ( Milgram, 2011 ; Schindler, 2014 ; Mitullha, 2004 ). In developed countries, instead, the growth of the informal economy is linked to the recent economic crisis, the growing levels of unemployment and the decrease in welfare spending. Moreover, globalization has accelerated non-standard work activity and a type of low-capital and labor-intensive tertiary sector, which is often reproduced in the informal economy, thanks to the implementation of unskilled migrant labor force ( Coletto, 2019 ; Devlin, 2019 ; Boels, 2014 ).

Specific individual motivations also influence the choice to undertake informal economic activities. In both areas of the world, street vending represents, first and foremost, an economic survival activity. In fact, in the absence of alternatives, street vending represents a choice driven by the necessity to obtain daily income ( Boels, 2014 ; Raijman, 2001 ; Crossa, 2009 ; Bromley and Mackie, 2009 ). Moreover, in many developing countries, street selling is also described as a voluntary and rational choice, which allows individuals to reach a certain degree of flexibility and work independence ( Williams and Gurtoo, 2012 ). Furthermore, while in many southern countries, street vending represents a traditional and stable activity, performed for generations ( Crossa, 2009 ; Williams and Gurtoo, 2012 ), in industrialized countries it often represents a temporary occupation ( DeLuca, 2012 ; L'hote and Gasta, 2007 ). In addition, while in many developing countries street vending often represents a cultural choice, in which social capital and family network play an important role ( Turner and Schoenberger, 2012 ; Crossa, 2016 ; Williams and Gurtoo, 2012 ), in many industrialized countries, the diffusion of informal vending activities can be explained by the fact that migrants often reproduce in their host countries economic activity that are traditionally widespread in their countries of origin ( Boels, 2014 ; Blanchar, 2011 ).

The relation between formal and informal economy

The link between the formal economy and informal street vending represents another aspect investigated in the literature, even if by relatively few studies ( Tables 1 and 2 ). First of all, the analysis of this piece of literature denotes the blurriness of the boundary between the informal and formal economy, because street vending activities are often characterized by an overlap of regular and irregular, legal and illegal conditions ( Milgram, 2011 ; Schindler, 2014 ; Coletto, 2019 ). Considering the literature on developing countries, the relationship between the formal economy and street vending is mainly addressed concerning the system to obtain permits to sell on the streets and regarding the connection between informal workers and the global value chains system ( Schindler, 2014 ; Cuvj, 2016 ; Martinez et al. , 2018 ). In fact, some studies highlight how the supply chain process of the products sold by street vendors take place, in almost all cases, in the formal economy ( Schindler, 2014 ; Sekhani et al. , 2019 ). Moreover, despite the various efforts to formalize street vendors, other studies highlight how workers often remain in a condition of permanent informality, due to often ambiguous commercial and urban regulations as well as the element of discretion with which, in many cases, state or local authorities grant permits to sell in the urban public space ( Milgram, 2011 ; Schindler, 2014 ; Cuvi, 2016 ). Furthermore, in some cases, the total number of licenses available is lower than the number of overall requests ( Anjaria, 2006 ). In addition, many street vendors highlight the difficulty of earning enough money to pay regular fees ( Roever, 2014 ). Focusing instead on developed countries, the link between informal street vending and the formal economy mostly concerns the dynamics of conflict between regular and informal street vendors. In this regard, several studies show how informal selling activities often take place within open-air markets, where informal actors share the location with regular vendors ( Coletto, 2019 ; Boels, 2014 ; Blanchard, 2011 ; Devlin, 2019 ). The coexistence, often in the same urban spaces, of the two types of vendors encourages dynamics of conflict and competition ( Coletto, 2019 ; Tchoukaleyska, 2015 ). Some other studies also highlight the regulation system and the procedures for obtaining licenses as factors that affect the informal street vendors' condition. In fact, migrants often sell without regular permits, since licenses are inaccessible in terms of costs and unobtainable due to the incompatible status of irregular migrants, which is common to many street vendors in developed countries ( Martin, 2014 ; Tchoukaleyska, 2015 ).

Urban policies and street vending regulations

The aspects mostly addressed in the recent literature concern the policies adopted to regulate the street vending sector and the informal actors' reactions when faced with such regulations. In this section, the article will investigate the policies and practices adopted by local authorities, while in the next section it will highlight the informal actors' resistance strategies. The literature on the global South gives particular attention to urban policies and practices adopted by local and state authorities to regulate street vending, with 34 out of 42 studies addressing this theme ( Table 1 ). In contrast, the literature on developed countries includes a less substantial number of studies focused on local regulations and enforcement agents practices ( Table 2 ).

The perceptions behind street vending

The analysis of this piece of the literature allows presenting a picture of how public opinion and local authorities perceive and describe the street vending activity. Although different studies, conducted both in developing and developed countries, describe street vending as a fundamental economic activity, which acts as a social safety net for marginal and poor populations ( Rogerson, 2016 ; Bhowmik, 2001 ; Boels, 2014 ; DeLuca, 2012 ), the tendency to perceive street vendors as a problem prevails. Primarily, street vendors are often accused of illegally using public spaces and of damaging the image of the city ( Lindell, 2019 ). This emerges mostly in the literature on developing countries, where informal street vending undermines the success of modernization projects aimed at making Southern cities of the world competitive with the Northern global cities ( Donovan, 2008 ; Rogerson, 2016 , Anjaria, 2006 ). Moreover, street vendors are also accused of promoting forms of degradation and chaos and, therefore, of undermining the social order ( Saha, 2009 ; Boonjubun, 2017 ; Turner and Schoenberg, 2012 ; Devlin, 2019 ). Finally, informal street vending competes with the formal trade market, generating negative economic effects for the commercial sector ( Steel, 2012 ; Forkuor et al. , 2017 ; Mitullah, 2004 ).

Urban governance model: between exclusionary policies and more tolerant regulations

During the 1970s, a tolerant policy against street vending prevailed in almost all developing countries. Nevertheless, between the 80s and 90s, the tendency to intervene to reduce the street vendors' presence spread, in line with the neoliberal urban governance model promoted in the United States and later introduced to many developing countries ( Lindell, 2019 ; Swansom, 2007 ; Donovan, 2008 : Crossa, 2016 ). Therefore, the recent public discourse on security have stimulated street trade criminalization and the adoption of exclusionary policies in many developing countries ( Reyes, 2013 ; Rogerson, 2016 ; Eltzold, 2015 ; Hanser, 2016 ; Morange, 2015 ). First of all, in various developing countries, the adoption of eviction campaigns against street vendors, to make cities more attractive for foreign investment and international tourism is a widespread trend ( Falla and Valencia, 2019 ; Reyes, 2013 ; Rogerson, 2016 ; Setsabi and Leduka, 2008 ; Eltzon, 2015 ). Other studies highlight the implementation of displacement policies and relocation projects, aimed at moving street vendors to decentralized areas of the city, assigning them regular working places ( Reyes, 2013 ; Swanson, 2007 ; Donovan, 2008 ; Weng and Kim, 2016 ). Nonetheless, the objective to make street vendors invisible often persists behind most of the relocation projects, in line with the gentrification plans widespread in many developing countries, especially in Latin American cities ( Hunt, 2009 ; Bromley and Mackie, 2009 ; Crossa, 2009 ). Finally, another set of studies shows how policemen and local authorities often adopt forms of harassment and abuse against street vendors, such as monetary extortion or arbitrary confiscation of merchandise for personal consumption ( Schindler, 2014 ; Rogerson, 2016 ; Lyons and Snoxell, 2005 ; Etzold, 2015 ; Milgram, 2011 ; Brown et al. , 2010 ). Despite that, a small number of other studies highlight how local authorities appear to oscillate between the adoption of exclusionary policies and more tolerant regulations ( Table 1 ). Indeed, the awareness that street vending represents an important source of income for marginalized people also persists. These ambivalent positions often generate contradictory regulatory models ( Rogerson, 2016 ; Bell and Loukaitou-Sideris, 2014 ; Huang et al. , 2014 ).

An analysis of the urban policies adopted in the global North reveals, instead, the contrast between the more heterogeneous and tolerant model that prevails in the European context and the repressive and no-tolerance regulations dominant in the US ( Table 2 ). Recent studies conducted in different North American cities highlight how various local governments are characterized by forms of uncertainty and legislative ambiguity in the regulation of street trade ( Devlin, 2019 ; Bhimji, 2010 ). In this regard, contradictory laws favor city authorities' discretionary acts against street vendors, who are in some cases, subjected to intimidation mechanisms and physical expulsion measures ( Martin, 2014 ; Devlin, 2019 ). Moving the analysis to the European context, the literature reveals how in some cities, street vending activities are repressed because the control of street vendors is often linked to the aim of controlling irregular immigration ( Moffette, 2018 ; L'hote and Gasta, 2007 ). Nevertheless, the local authorities' tendency to give little priority to informal vending activities emerges ( Nelken, 2006 ; Boels, 2014 ). In fact, some studies highlight how greater attention is often paid to intercepting the mechanisms behind the sale of counterfeit goods, rather than to massively controlling those who sell the goods on the streets ( Boels, 2014 ; Schmoll and Semi, 2013 ; Milliot and Tonnelat, 2013 ).

Street vendors' strategies of resistance and negotiation

For decades, street vendors have been considered as marginal individuals who passively practice their activities ( Lindell, 2019 ; Falla and Valencia, 2019 ). More recently, several studies, mainly those conducted in global South, have emphasized the role of the street vendors' agency, highlighting the strategies of resistance and negotiation they employ to confront exclusionary policies and guarantee their right to work in the public space ( Crossa, 2016 ; Schindler, 2014 ; Hunt, 2009 ; Boonjubun, 2017 ). In this regard, almost all studies conducted in both developing and developed countries ( Tables 1 and 2 ), highlight the informal actors' ability to modify their daily strategies and resiliently resist political limitations. Through the analysis of the literature, it is possible to identify both individual and collective strategies of resistance.

Individual strategies

At a more individual level, street vendors, in both developing and developed countries, adopt similar strategies of resistance. First of all, street vendors often act in an itinerant manner, adjusting their practices of vending. Bringing fewer goods and constantly moving around the streets not only allows to escape faster but also to be more invisible and attract less attention ( Milgram, 2011 ; Crossa, 2009 ; Boels, 2014 ; DeLuca, 2012 ). Another strategy, in this sense, is to move the businesses to less controlled places, to avoid forms of harassment and confiscation by city authorities ( Stell, 2012 ; Milgram, 2011 ; Boels, 2014 ). Moreover, street vendors also tend to anticipate the local authorities' actions as a way of avoiding forms of control. Indeed, they identify the police working hours and adjust their activities accordingly, to coincide with the lower presence of the police ( Bhimji, 2010 ; L'hote and Gasta, 2007 ; Coletto, 2019 ; Recio and Gomez, 2013 ; Turner and Schoenberger, 2012 ). Finally, another daily strategy involves bribing city authorities or street gangs, to guarantee the possibility to work on the street. This practice makes it possible for street vendors to improve their conditions and survive in the urban public space ( Eltzon, 2015 ; Milgram, 2011 ; Munoz, 2016 ; Anjaria, 2006 ; Mitullah, 2004 ).

Collective strategies

The literature also highlights how social capital and street vendor networks play an important role in deploying daily strategies of collective resistance ( Lyon and Snoxell, 2005 ). Some studies conducted in both developing and developed countries, show how vendors tend to support each other, for example by sending messages or adopting communications signals to prevent eviction from city authorities ( Cuvi, 2016 ; Crossa, 2009 ; Steel, 2012 ; Lyons and Snoxell, 2005 ; L'Hote and Gasta, 2007 ; Boels, 2014 ). Other studies, conducted in some developing countries, show how, in specific situations, street vendors resort to public demonstrations to protect their right to work on public spaces ( Crossa, 2009 ; Recio and Gomez, 2013 ; Roever, 2016 ; Millgram, 2011 ). Moreover, in many developing countries, the presence of street vendors' associations and organizations is often crucial for negotiating with local authorities and improving bargaining power. In fact, local street vendors' associations, which are spreading across many cities of the global South, give voice to the street vendors' interests ( Donovan, 2008 ; Weng and Kim, 2016 ; Roever, 2016 ; Saha, 2009 ; Crossa, 2016 ). Other studies highlight how, in many less developed countries, street vendors can also use their vote as bargaining power ( Donovan, 2008 ; Milgram, 2011 ). The leaders of local associations, who control the street vendors' political consensus, grant the local authorities electoral and political favors in exchange for privileged treatment and the possibility to work ( Crossa, 2009 ; Eltozon, 2015 ). Finally, in many less developed countries, non-governmental organizations also play an important role. For example, Street Net International and WIEGO negotiate with governments and local authorities to defend informal workers' rights and encourage the regularization of their conditions ( Roever, 2014 ; Bhowmik, 2001 ; Cuvi, 2016 ).

The following table summarizes the main differences and similarities that emerged to study street vending in both developing and developed countries, aimed to explain these comparative findings ( Table 3 ). The table represents an attempt at synthesis, which necessarily extremes some comparative elements that are more subtle and complex and, therefore, need to be deepened and investigated through future research.

The first distinction emerges if we look at the informal street vending sector size. In the global South, many street vendors work informally, while in developed countries informal vending activities represent a minority compared to the entire sector. Another difference concerns the street vendors' profile and the sector role. In global South, street vending is mainly performed by internal rural migrants and by poorer and less educated people, while in developed countries it is carried out by immigrants. Moreover, in southern countries street vending represents a stable working activity, performed for generations, however in industrialized countries it is mainly perceived as a temporary occupation, carried out as immigrants' first activity in host countries. The different economic system and the diversified effects that economic globalization has generated in the two types of countries may partially explain these differences ( Temkin and Veizaga, 2010 ; Gerxhani, 2004 ). In the literature several are the theories that look at economic factors to explain the different degrees of the informal economy between countries. For instance, the world-system theory applied to the study of informality allows us to highlight how due to the developing countries' economic system, based on a low level of industrialization, on the use of little technology and low-paid unskilled workers, these countries have experienced more rapid and widespread development of the informal economy than developed countries. Furthermore, the rapid urbanization process of the 1980s, which affected many southern cities, has increased the surplus of labor, encouraging the development of informality. On the contrary, the industrialized countries tend to specialize in capital-intensive and high-skill service sectors ( Roberts, 2013 ; Bhowmik, 2012 ). Therefore, in these contexts, only people without the skills to compete for high-tech formal employment, especially immigrants, are forced to work informally, mostly in the labor-intensive tertiary sector ( Sassen, 2007 ). Other theoretical approaches look rather at institutional factors to explain the size and characteristics of the informal economy of a given country. Borrowing the institutionalist approach , the high weight of the informal street vending sector in many developing countries can be explained by a widespread distrust in institutions and by the asymmetry between formal institutions and informal norms. This asymmetry promotes a distance between the state morality and that of the individual, who therefore may be more inclined to enter the informal economy ( Lyon, 2007 ; Williams et al. , 2015 ). Continuing to analyze the main differences, in the global South literature the relation between formal economy and informal street vending is mostly studied concerning the system to obtain permits to sell in public space and regarding the link between the informal workers and the global value chains system. Meanwhile, in the developed countries' literature, this link has mostly been studied regarding the conflict dynamics between regular and irregular street vendors. Furthermore, the studies on Southern countries focused more on the policies implemented to regulating street vending and the street vendors' resistance strategies, while the literature on industrialized countries analyzed more the relationship between possessing the discriminating status of immigrant and entering into the informal economy. Finally, while a substantial number of studies on both developing and developed countries highlight the street vendors' strategies of resistance, only several studies on developing countries emphasize the individual agency's role and the street vendors' collective bargaining power. This can be explained by the fact that are mostly global South' researchers who adopt an actor-oriented approach to study manifestations of urban informality ( Devlin, 2019 ; Lindell, 2019 ). Using the conceptual framework proposed by Bayat to study informality in global South, street vending is not simply an activity to cope with socio-economic injustices but also an intentional resistance practice that transgresses spatial and legal norms ( Bayat, 2004 ).

Continuing to analyze the literature review findings, common aspects are also highlighted. The first similarity is represented by the tendency to use qualitative investigation techniques and case study research approach to study street vending. Furthermore, similar trajectories emerge if we look at the urban governance model adopted to regulate informal trade. Recent studies conducted in both developing and North American countries reveal the tendency to embrace neoliberal and exclusionary policies to regulate informal street vending. It can be explained by the fact that global economic trends affect the state and local authorities' approach to manage and conceive marginality and the urban informality in the global cities' public spaces. Many countries tend to adopt exclusionary policies to eliminate the presence of street vendors, as their marginal activity does not meet the standards imposed by economic globalization ( Graaff and Ha, 2015 ; Lindell, 2019 ; Roever, 2014 ). Furthermore, the theoretical debate on the informal economy highlights how concerning most informal sectors the use of survival criterion in defining the reasons to work informally is valid for the southern countries and not for industrialized countries ( Gerxhani, 2004 ). On the contrary, the analysis on street vending sector showed how, although in some cases in global South it represents a voluntary or rational choice, in both two types of countries it constitutes, first and foremost, an economic survival activity. This aspect can be explained using the conceptual framework proposed by the structuralist theory to study the informal economy. This approach considers the informal work functional and subordinated to the formal economy. Indeed, the market liberalization and the economic globalization increase dynamics of socio-economic polarization, with the result that unskilled workers and marginal people are forced to work informally, as they cannot find a job in the formal economy ( Portes et al. , 1989 ; Sassen, 2007 ).

Conclusions

The article aimed to contribute in overcoming the tendency to employ different analytical lenses to study street vending between the global South and global North and to highlight the usefulness of analyzing the phenomenon in a comparative perspective. To do this, a comparative literature review on the topic has been conducted. It represents an innovative review attempt, as it explicitly compares this manifestation of informality between the two institutionally different types of countries. Moreover, it constitutes also the first contribution to review street vending in the global North.

The findings discussed in the last section suggest both similarities and differences regarding the street vending characteristics and concerning the aspects deal with by the literature to study it between global South and global North. Therefore, the analysis highlights some gaps present in the literature to study this informal economy sector, which can represent the basis for future comparative research on the topic. On the one hand, comparative research can stimulate the development of a theory that promotes a univocal interpretation and explanation of the informal street vending sector, considering simultaneously a set of factors that affect the informal economy of a given country (institutional, economic and cultural factors). Comparative case study research can also encourage the building of a theory aims at explain this manifestation of informality considering the specificities of both developing and developed countries. Moreover, despite informal street selling occurs both in developing and developed countries, in the literature comparative case studies aimed at investigating this informal sector between so different contexts from both an institutional and economic point of view are missing. Therefore, on the other hand, future comparative research can improve also empirical knowledge of the phenomenon. Comparative research will allow us to better understand to what extent individual motives, institutional and economic contextual factors and economic globalization affect the street trade sector in a given context. Furthermore, to address the transformations, but also the opportunities generated by economic globalization, a change in economic and social policies regarding urban informality is also necessary. In a climate of global growing income inequality, informal practices like street vending play an important economic and social role. Indeed, informal street vending guarantees daily income for an increasing poor and marginal side of the world population. Furthermore, while in developing countries street vendors provide goods and services that satisfy the demand of a growing low-income population, in industrialized countries, as this activity often represents the first immigrants' livelihood occupation, street vending also prevents entry into illicit or criminal circuits. Therefore, the state and local institutions should recognize the positive implications that would be generated if the street vendors' working conditions and rights were improved. Hence institutions should investigate the extend of the informal street vendors population and increase the number of licenses, promoting the transition into formal entrepreneurship. Increasing specific regulations to support street vending livelihoods would represent the basis for a more supportive urban economy, which aims to guarantee rights and protections to actors excluded from the formal economy circuit.

Studies on developing countries

Studies on developed countries

Differences and similarities between developing and developed countries

Anjaria , J.S. ( 2006 ), “ Street hawkers and public space in Mumbai ”, Economic and Political Weekly , Vol. 27 No. 3 , pp. 2140 - 46 .

Bayat , A. ( 2004 ), “ Globalization and the politics of the informal in the Global South ”, in Roy , A. and AlSayyad , N. (Eds), Urban Informality: Transnational Perspectives from the Middle East, Latin America, and South Asia , Lexington Books , Boulder, CO , pp. 7 - 30 .

Bell , J.S. and Loukaitou-Sideris , A. ( 2014 ), “ Sidewalk informality: an examination of street vending regulation in China ”, International Planning Studies , Vol. 19 Nos 3-4 , pp. 221 - 243 .

Bhimji , F. ( 2010 ), “ Struggles, urban citizenship, and belonging: the experience of undocumented street vendors and food truck owners in Los Angeles ”, Urban Anthropology and Studies of Cultural Systems and World Economic Development , Vol. 39 No. 4 , pp. 455 - 492 .

Bhowmik , S.K. ( 2001 ), Hawkers and the Urban Informal Sector: A Study of Street Vending in Seven Cities , Unpublished report prepared for National Association of Street Vendors in India (NASVI) , available at: http://wiego.org/sites/wiego.org/files/publications/files/Bhowmik-Hawkers-URBAN-INFORMAL-SECTOR.pdf ( accessed 25 February 2014 ).

Bhowmik , S.K. ( 2012 ), Street Vendors in the Global Urban Economy , Taylor & Francis , London .

Blanchard , M. ( 2011 ), “ Fare mercato a Torino: carriere professionali e pratiche quotidiane degli ambulanti stranieri nei mercati rionali ”, Mondi Migranti , Vol. 2 , pp. 75 - 99 .

Boels , D. ( 2014 ), “ It's better than stealing: informal street selling in Brussels ”, International Journal of Sociology and Social Policy , Vol. 34 Nos 9-10 , pp. 670 - 693 .

Boonjubun , C. ( 2017 ), “ Conflicts over streets: the eviction of Bangkok street vendors ”, Cities , Vol. 70 , pp. 22 - 31 .

Bromley , R. and Mackie , P. ( 2009 ), “ Displacement and the new spaces for informal trade in Latin American city centre ”, Urban Studies , Vol. 46 No. 7 , pp. 1485 - 1506 .

Bromley , R. ( 2000 ), “ Street vending and public policies: a global review ”, International Journal of Sociology and Social Policy , Vol. 20 Nos 1-2 , pp. 1 - 28 .

Brown , A. , Lyons , M. and Dankoco , I. ( 2010 ), “ Street traders and the emerging spaces for urban voice and citizenship in African cities ”, Urban Studies , Vol. 47 No. 3 , pp. 666 - 683 .

Chen , M.A. ( 2012 ), “ The informal economy: definitions, theories and policies ”, Working Paper, WIEGO , Vol. 1 No. 26 , pp. 90141 - 90144 .

Coletto , D. ( 2010 ), “ Ambulantes and Camelôs (The street vendors) ”, in Coletto , D. (Ed.), The Informal Economy and Employment in Brazil , Palgrave Macmillan , New York, NY , pp. 97 - 150 ..

Coletto , D. ( 2019 ), “ L'economia informale e le sue rappresentazioni sociali: il caso dei mercati all'aperto ”, in Andreotti , A. (Ed.), Governare Milano Nel Nuovo Millennio , Il Mulino , Bologna , pp. 239 - 261 ..

Cross , J. ( 2000 ), “ Street vendors, and postmodernity: conflict and compromise in the global economy ”, International Journal of Sociology and Social Policy , Vol. 20 Nos 1-2 , pp. 29 - 51 .

Crossa , V. ( 2009 ), “ Resisting the entrepreneurial city: street vendors' struggle in Mexico City's historic center ”, International Journal of Urban and Regional Research , Vol. 33 No. 1 , pp. 43 - 63 .

Crossa , V. ( 2016 ), “ Reading for difference on the street: De-homogenising street vending in Mexico City ”, Urban Studies , Vol. 53 No. 2 , pp. 287 - 301 .

Cuvi , J. ( 2016 ), “ The politics of field destruction and the survival of Sao Paolo's street vendors ”, Social Problems , Vol. 63 No. 3 , pp. 395 - 412 .

DeLuca , J.S. ( 2012 ), “ Street vendors in the global city: exploring Genoa's informal economy ”, Journal for Undergraduate Ethnography , Vol. 2 No. 1 , pp. 41 - 51 .

Devlin , R.T. ( 2011 ), “ An area that governs itself': informality, uncertainty and the management of street vending in New York City ”, Planning Theory , Vol. 10 No. 1 , pp. 53 - 65 .

Devlin , R.T. ( 2019 ), “ A focus on needs: toward a more nuanced understanding of inequality and urban informality in the global North ”, Journal of Cultural Geography , Vol. 36 No. 2 , pp. 121 - 143 .

Donovan , M.G. ( 2008 ), “ Informal cities and the contestation of public space: the case of Bogotá's street vendors 1988—2003 ”, Urban Studies , Vol. 45 No. 1 , pp. 29 - 51 .

Etzold , B. ( 2015 ), “ Selling in insecurity – living with violence: eviction drives against street food vendors in Dhaka and the informal politics of exploitation ", in Graaf , K. and Ha , N. (Eds), Street Vending in the Neoliberal City: A Global Perspective on the Practices and Policies of a Marginalized Economy , Berghahn Books , New York, NY , pp. 164 - 190 .

Falla , A.M.V. and Valencia , S.C. ( 2019 ), “ Beyond state regulation of informality: understanding access to public space by street vendors in Bogotà ”, International Development Planning Review , Vol. 41 No. 1 , pp. 85 - 105 .

Forkuor , J.B. , Akuoko , K.O. and Yeboah , E.H. ( 2017 ), “ Negotiation and management strategies of street vendors in developing countries: a narrative review ”, SAGE Open , Vol. 7 No. 1 , pp. 1 - 13 .

Gerxhani , K. ( 2004 ), “ The informal sector in developed and less developed countries: a literature survey ”, Public Choice , Vol. 120 Nos 3-4 , pp. 267 - 300 .

Graaff , K. and Ha , N. ( 2015 ), Street Vending in the Neoliberal City: A Global Perspective on the Practices and Policies of a Marginalized Economy , Berghahn Books , New York, NY .

Hanser , A. ( 2016 ), “ Street politics: street vendors and urban governance in China ”, The China Quarterly , Vol. 226 , pp. 363 - 382 .

Harney , N.D. ( 2004 ), “ Migrant productivities: street vendors and the informal knowledge work in Naples ”, International Journal of Economic Development , Vol. 6 No. 3 , pp. 306 - 331 .

Huang , G. , Xue , D. and Li , Z. ( 2014 ), “ From revanchism to ambivalence: the changing politics of street vending in Guangzhou ”, Antipode , Vol. 46 No. 1 , pp. 170 - 189 .

Hunt , S. ( 2009 ), “ Citizenship's place: the state's creation of public space and street vendors' culture of informality in Bogota, Colombia ”, Environment and Planning D: Society and Space , Vol. 27 No. 2 , pp. 331 - 351 .

Ilahiane , H. and Sherry , J. ( 2008 ), “ Joutia: street vendor entrepreneurship and the informal economy of information and communication technologies in Morocco ”, Journal of North African Studies , Vol. 13 No. 2 , pp. 243 - 255 .

L'hote , L. and Gasta , C. ( 2007 ), “ Immigration and street entrepreneurship in Alicante, Spain ”, International Journal of Iberian Studies , Vol. 20 No. 1 , pp. 3 - 22 .

Linares , L.A. ( 2018 ), “ The paradoxes of informalizing street trade in the Latin American city ”, International Journal of Sociology and Social Policy , Vol. 38 Nos 7-8 , pp. 651 - 672 .

Lindell , I. ( 2019 ), “ Introduction: re-spatialising urban informality: reconsidering the spatial politics of street work in the global South ”, International Development Planning Review , Vol. 41 , pp. 3 - 21 .

Lyon , F. ( 2007 ), “ Institutional perspectives on understanding street retailer behaviour and networks ”, in Cross , J. and Morales , A. (Eds), Street Entrepreneurs: People, Place and Politics in Local and Global Perspective , Routledge , London , pp. 164 - 179 ..

Lyons , M. and Snoxell , S. ( 2005 ), “ Creating urban social capital: some evidence from informal traders in Nairobi ”, Urban Studies , Vol. 42 No. 7 , pp. 1077 - 1097 .

Martin , N. ( 2014 ), “ Food fight! Immigrant street vendors, Gourmet food trucks and the differential valuation of creative producers in Chicago ”, International Journal of Urban and Regional Research , Vol. 38 No. 5 , pp. 1867 - 1883 .

Martínez , L. , Short , J.R. and Estrada , D. ( 2018 ), “ The diversity of the street vending: a case study of street vending in Cali ”, Cities , Vol. 79 , pp. 18 - 25 .

Milgram , B.L. ( 2011 ), “ Reconfiguring space, mobilizing livelihood street vending, legality, and work in the Philippines ”, Journal of Developing Societies , Vol. 27 Nos 3-4 , pp. 261 - 293 .

Milliot , V. and Tonnelat , S. ( 2013 ), “ Contentious policing in Paris: the street as a space for emotional public solidariety ”, in Lippert , R.K. and Walby , K. (Eds), Policing Cities: Urban Securitization and Regulation in a 21st Century World , Routledge , London , pp. 209 - 222 .

Mitullah , W. ( 2004 ), A Review of Street Trade in Africa , Unpublished Manuscript , WIEGO , Cambridge, MA , p. 34 .

Moffette , D. ( 2018 ), “ The jurisdictional games of immigration policing: Barcelona's fight against unauthorized street vending ”, Theoretical Criminology , Vol. 24 No. 2 , pp. 1 - 18 .

Morange , M. ( 2015 ), “ Street trade, neoliberalisation and the control of space: Nairobi's Central Buisiness District in the era of entrepreneurial urbanism ”, Journal of Eastern African Studies , Vol. 9 No. 2 , pp. 247 - 269 .

Munoz , L. ( 2016 ), “ Agency, choice and restrictions in producing Latina/o street‐vending landscapes in Los Angeles ”, Area , Vol. 48 No. 3 , pp. 339 - 345 .

Nelken , D. ( 2006 ), “ Immigrant beach selling along the Italian Adriatic coast: De-constructing a social problem ”, Crime, Law and Social Change , Vol. 45 Nos 4-5 , pp. 297 - 313 .

Onodugo , V.A. , Ezeadichie , N.H. , Onwuneme , C.A. and Anosike , A.E. ( 2016 ), “ the dilemma of managing the challenges of street vending in public spaces: the case of Enugu City, Nigeria ”, Cities , Vol. 59 , pp. 95 - 101 .

Portes , A. , Castells , M. and Benton , L.A. ( 1989 ), The Informal Economy: Studies in Advanced and Less Developed Countries , Johns Hopkins University Press , Baltimore .

Raijman , R. ( 2001 ), “ Mexican immigrants and informal self-employment in Chicago ”, Human Organization , Vol. 60 No. 1 , pp. 47 - 55 .

Recio , R.B. and Gomez , G.H. ( 2013 ), “ Street vendors, their contested spaces, and the policy environment: a view from Caloócan, Metro Manila ”, Environment and Urbanization Asia , Vol. 4 No. 1 , pp. 173 - 190 .

Reid , D.M. , Fram , E.H. and Guotai , C. ( 2010 ), “ A study of Chinese street vendors: how they operate ”, Journal of Asia-Pacific Business , Vol. 11 No. 4 , pp. 244 - 257 .

Reyes , R.M. ( 2013 ), “ Crime, street vendors and the historical Downtown in post-Giuliani Mexico City ”, International Journal of Criminology and Sociology , Vol. 2 , pp. 186 - 198 .

Roberts , A. ( 2013 ), “ Peripheral accumulation in the world economy: a cross-national analysis of the informal economy ”, International Journal of Comparative Sociology , Vol. 54 Nos 5-6 , pp. 420 - 444 .

Roever , S. ( 2010 ), “ Street trade in Latin America: demographic trends, legal issues, and vending organizations in six cities ”, in Bhowmik , S. (Ed.), Street Vendors in the Global Urban Economy , Taylor & Francis , London , pp. 208 - 240 ..

Roever , S. ( 2014 ), Informal Economy Monitoring Study Sector Report: Street Vendors , WIEGO , Cambridge, MA , pp. 1 - 72 .

Roever , S. ( 2016 ), “ Informal trade meets informal governance: street vendors and legal reform in India, South Africa, and Peru ”, Cityscape , Vol. 18 No. 1 , pp. 27 - 46 .

Rogerson , C.M. ( 2016 ), “ Progressive rhetoric, ambiguous policy pathways: street trading in inner-city Johannesburg, South Africa ”, Local Economy , Vol. 31 Nos 1-2 , pp. 204 - 218 .

Saha , D. ( 2009 ), “ Decent work for the street vendors in Mumbai, India—a distant vision! ”, Journal of Workplace Rights , Vol. 14 No. 2 , pp. 229 - 250 .

Sassen , S. ( 2007 ), A Sociology of Globalization , W.W. Norton and Company , New York, NY .

Schindler , S. ( 2014 ), “ Producing and contesting the formal/informal divide: regulating street hawking in Delhi, India ”, Urban Studies , Vol. 51 No. 12 , pp. 2596 - 2612 .

Schmoll , C. and Semi , G. ( 2013 ), “ Shadow circuits: urban spaces and mobilities across the Mediterranean ”, Identities: Global Studies in Culture and Power , Vol. 20 No. 4 , pp. 377 - 392 .

Sekhani , R. , Mohan , D. and Medipally , S. ( 2019 ), “ Street vending in urban ‘informal’ markets: reflections from case-studies of street vendors in Delhi (India) and Phnom Penh City (Cambodia) ”, Cities , Vol. 89 , pp. 120 - 129 .

Setšabi , S. and Leduka , R.C. ( 2008 ), “ The politics of street trading in Maseru, Lesotho ”, Urban Forum , Springer, Netherlands , Vol. 19 No. 3 , pp. 221 - 241 .

Steel , W.F. , Ujoranyi , T.D. and Owusu , G. ( 2014 ), “ Why eviction do not deter street traders: case study in Accra, Ghana ”, Ghana Social Science Journal , Vol. 11 No. 2 , pp. 52 - 76 .

Steel , G. ( 2012 ), “ Whose paradise? Itinerant street vendors' individual and collective practices of political agency in the tourist streets of Cusco, Peru ”, International Journal of Urban and Regional Research , Vol. 36 No. 5 , pp. 1007 - 1021 .

Swanson , K. ( 2007 ), “ Revanchist urbanism heads south: the regulation of indigenous beggars and street vendors in Ecuador ”, Antipode , Vol. 39 No. 4 , pp. 708 - 728 .

Swider , S. ( 2015 ), “ Reshaping China's urban citizenship: street vendors, Chengguan and struggles over the right to the city ”, Critical Sociology , Vol. 41 Nos 4-5 , pp. 701 - 716 .

Tchoukaleyska , R. ( 2015 ), “ Illicit mint and unregulated vendors: constructing illegality in French public spaces ”, Area , Vol. 47 No. 1 , pp. 81 - 87 .

Temkin , B. and Veizaga , J. ( 2010 ), “ The impact of economic globalization on labor informality ”, New Global Studies , Vol. 4 No. 1 , pp. 1 - 26 .

Turner , S. and Schoenberger , L. ( 2012 ), “ Street vendor livelihoods and everyday politics in Hanoi, Vietnam: the seeds of a diverse economy? ”, Urban Studies , Vol. 49 No. 5 , pp. 1027 - 1044 .

Vargas , A.M. and Urinboyev , R. ( 2015 ), “ Everyday forms of resistance to the law: an ethnographic study of street vendors in Bogotá ”, Droit et Société , Vol. 3 , pp. 623 - 638 .

Weng , C.Y. and Kim , A.M. ( 2016 ), “ The critical role of street vendor organizations in relocating street vendors into public markets: the case of Hsinchu city, Taiwan ”, Cityscape , Vol. 18 No. 1 , pp. 47 - 70 .

Williams , C.C. and Gurtoo , A. ( 2012 ), “ Evaluating competing theories of street entrepreneurship: some lessons from a study of street vendors in Bangalore, India ”, The International Entrepreneurship and Management Journal , Vol. 8 No. 4 , pp. 391 - 409 .

Williams , C.C. , Horodnic , I.A. and Windebank , J. ( 2015 ), “ Explaining participation in the informal economy: an institutional incongruence perspective ”, International Sociology , Vol. 30 No. 3 , pp. 294 - 313 .

Wongtada , N. ( 2014 ), “ Street vending phenomena: a literature review and research agenda ”, Thunderbird International Business Review , Vol. 56 No. 1 , pp. 55 - 75 .

Acknowledgements

I thank, first of all, the anonymous referees for their comments and suggestions. I also thank Giovanna Fullin and Diego Coletto for supervising my research activities. The following article did not provide specific research funding. This paper is part of a broader Ph.D. research project, within the Analysis of Social and Economic Process doctoral course, funded by the Department of Sociology and Social Research, of the University of Milan-Bicocca, Italy.

Corresponding author

Related articles, we’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

Literature Review: A Comparative Study of Software Defect Prediction Techniques

• Conference paper
• First Online: 15 April 2023
• Cite this conference paper

• Tarunim Sharma 8 ,
• Aman Jatain 8 ,
• Shalini Bhaskar 8 &
• Kavita Pabreja 9  

Part of the book series: Algorithms for Intelligent Systems ((AIS))

259 Accesses

At the latest, a lot of studies have been conducted to forecast software flaws and determine if modules are defective or not, in order to build high-quality software at a low cost before executing the testing stage. To solve the limitations of prediction, researchers have used a variety of machine learning techniques to build models. Though the time and resources required utilizing these procedures are calculated to be less because study concentrates on only those modules that are projected to be problematic. But a disproportionately big portion of the studies demonstrated low prediction capability, and their accuracy for varied datasets was also not persistent. In this literature survey paper, all the techniques have been studied and analyzed for predicting the bugs in the software. All the approaches being adopted by the researchers for handling the dimension reduction, noise filtering and class imbalancing have been studied in this review. There is still a need for handling feature selection, noise filtering, and class imbalancing, which can further affect the performance and accuracy of the model. To further comprehend these flaws, a full review has been undertaken from 2016 to 2021 period in order to gain a better knowledge of how the various models proposed by a number of authors are still lacking in terms of performing better on the available datasets and accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Dam HK, Pham T, Ng SW, Tran T, Grundy J, Ghose A, Kim T, Kim CJ (2019) Lessons learned from using a deep tree-based model for software defect prediction in practice. In: Proceedings of 16th international conference on mining software repositories (MSR). IEEE, Montreal, QC, Canada, pp 46–57

Google Scholar  

Jin C (2021) Software defect prediction model based on distance metric learning. Soft Comput:447–461. Springer

Shenvi AA (2021) Defect prevention with orthogonal defect classification. In: Proceedings of the 2nd India software engineering conference. pp 83–88

Yu L, Liu H (2004) Efficient feature selection via analysis of relevance and redundancy. J Mach Learn Res:1205–1224

Gao K, Khoshgoftaar TM, Wang H, Seliya N (2020) Choosing software metrics for defect prediction: an investigation on feature selection techniques. Softw: Pract Exp 41(5):579–606. Wiley Online Library

Rodriguez D, Ruiz R, Cuadrado-Gallego J, Aguilar-Ruiz J, Garre M (2007) Attribute selection in software engineering datasets for detecting fault modules. In: 33rd EUROMICRO Conference on software engineering and advanced applications, EUROMICRO 2007. IEEE, pp 418–423

Peng Y, Wu Z, Jiang J (2010) A novel feature selection approach for biomedical data classification. J Biomed Inform 43(1):15–23. Elsevier

Sharma D, Chandra P (2018) Software fault prediction using machine-learning techniques. In: Smart computing and informatics. Springer, Singapore, pp 541–549

Meiliana SK, Warnars HLHS, Gaol FL, Abdurachman E, Soewito B (2017) Software metrics for fault prediction using machine learning approaches: A literature review with PROMISE repository dataset, In: 2017 IEEE international conference on cybernetics and computational intelligence (CyberneticsCom). IEEE, pp 19–23

Ran L, Zhou L, Zhang S, Liu H, Huang X, Sun Z (2019) Software defect prediction based on ensemble learning. In: Proceedings of the 2019 2nd International conference on data science and information technology. pp 1–6

Matloob F, Aftab S, Iqbal A (2019). A framework for software defect prediction using feature selection and ensemble learning techniques. Int J Mod Educ & Comput Sci 11 (12)

Khan MZ (2020) Hybrid ensemble learning technique for software defect prediction. Int J Mod Educ & Comput Sci 12(1)

Matloob F, Ghazal TM, Taleb N, Aftab S, Ahmad M, Khan MA (2021) Software defect prediction using ensemble learning: a systematic literature review. IEEE Access:98754–98771

Agrawal P, Abutarboush HF, Ganesh T, Mohamed AW (2021) Metaheuristic algorithms on feature selection: a survey of one decade of research (2009-2019). IEEE Access 9:26766-26791

Oluwagbemiga BA, Shuib B, AbdulKadir SJ, Sobri AH (2019) A hybrid multi-filter wrapper feature selection method for software defect predictors

Raheem M, Ameen A, Ayinla F, Ayeyemi B (2020) Software defect prediction using metaheuristic algorithms and classification techniques. Ilorin J Comput Sci Inf Technol 3(1):23-39. Bennin KE, Keung J, Phannachitta P, Monden A, Mensah S (2017) Mahakil: Diversity based oversampling approach to alleviate the class imbalance issue in software defect prediction. IEEE Trans Softw Eng 44(6):534-550

Alweshah M, Alkhalaileh S, Albashish D, Mafarja M, Bsoul Q, Dorgham O (2021) A hybrid mine blast algorithm for feature selection problems. Soft Comput 25(1):517–534

Article   Google Scholar  

Suresh Kumar P, Behera HS, Nayak J, Naik B (2021) Bootstrap aggregation ensemble learning-based reliable approach for software defect prediction by using characterized code feature. Innov Syst Softw Eng:1–25

Goyal S, Bhatia PK (2020) Heterogeneous stacked ensemble classifier for software defect prediction. In: 2020 Sixth International conference on parallel, distributed and grid computing (PDGC). pp 126–130. IEEE

Alsawalqah H, Hijazi NM, Eshtay M, Faris H, Radaideh AA, Aljarah I, Alshamaila Y (2020) Software defect prediction using heterogeneous ensemble classification based on segmented patterns. Appl.Sci 10(5):1745

Jiang K, Iwahori Y, Wang A, Wu H, Zhang Y (2020) Heterogeneous defect prediction based on transfer learning to handle extreme imbalance. Appl Sci 10(1):396

Chakraborty T, Chakraborty AK (2020) Hellinger net: A hybrid imbalance learning model to improve software defect prediction. IEEE Trans Reliab, IEEE 70(2):481-494

Bejjanki KK, Gyani J, Gugulothu N (2020) Class imbalance reduction (CIR): a novel approach to software defect prediction in the presence of class imbalance. MDPI, Symmetry 12(3):407

Wei H, Hu C, Chen S, Xue Y, Zhang Q (2009) Establishing a software defect prediction model via effective dimension reduction. Inf Sci 477:399-409

Khuat TH, Le MH (2019) Ensemble learning for software fault prediction problem with imbalanced data. Int J Electr Comput Eng 9(4):3241

Cai X, Niu Y, Geng S, Zhang J, Cui Z, Li J, Chen J (2019) An under-sampled software defect prediction method based on hybrid multi-objective cuckoo search. Concurr Comput: Pract Exp 32(5):e5478

NezhadShokouhi MM, Majidi MA, Rasoolzadegan A (2020) Software defect prediction using over-sampling and feature extraction based on Mahalanobis distance. J Supercomput 76(1):602-635

Ghosh S, Rana A, Kansal V (2018) A nonlinear manifold detection-based model for software defect prediction. Procedia Comput Sci 132:581–594. Elsevier

Miholca DL (2018) An improved approach to software defect prediction using a hybrid machine learning model. In: 2018 20th International symposium on symbolic and numeric algorithms for scientific computing (SYNASC), IEEE. pp 443–448

Nawaz A, Rehman AU, Abbas M (2020) A novel multiple ensemble learning models based on different datasets for software defect prediction. arXiv preprint arXiv: 2008.13114. Cornell University

Tomar D, Agarwal S (2016) Prediction of defective software modules using class imbalance learning, Applied Computational Intelligence and Soft Computing Volume 2016. Article ID 7658207:6

Bahaweres RB, Agustian F, Hermadi I, Suroso AI (2020) Software defect prediction using neural network based SMOTE. In: 7th international conference on electrical engineering, computer sciences and informatics (EECSI)

Download references

Author information

Authors and affiliations.

Amity School of Engineering and Technology, Gurgaon, Haryana, India

Tarunim Sharma, Aman Jatain & Shalini Bhaskar

Maharaja Surajmal Institute, Janakpuri, New Delhi, India

Kavita Pabreja

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Tarunim Sharma .

Editor information

Editors and affiliations.

Poornima College of Engineering, Jaipur, Rajasthan, India

Garima Mathur

Mahesh Bundele

Department of Computer Science and Engineering, Malaviya National Institute of Technology Jaipur, Jaipur, India

Ashish Tripathi

Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

Marcin Paprzycki

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Cite this paper.

Sharma, T., Jatain, A., Bhaskar, S., Pabreja, K. (2023). Literature Review: A Comparative Study of Software Defect Prediction Techniques. In: Mathur, G., Bundele, M., Tripathi, A., Paprzycki, M. (eds) Proceedings of 3rd International Conference on Artificial Intelligence: Advances and Applications. Algorithms for Intelligent Systems. Springer, Singapore. https://doi.org/10.1007/978-981-19-7041-2_2

Download citation

DOI : https://doi.org/10.1007/978-981-19-7041-2_2

Published : 15 April 2023

Publisher Name : Springer, Singapore

Print ISBN : 978-981-19-7040-5

Online ISBN : 978-981-19-7041-2

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Literature Review : A Comparative Study of Real Time Streaming Technologies and Apache Kafka

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.