quantitative research about medicine and its contribution

Advertisement

• Previous Issue
• Previous Article
• Next Article

Clarifying the Research Purpose

Methodology, measurement, data analysis and interpretation, tools for evaluating the quality of medical education research, research support, competing interests, quantitative research methods in medical education.

Submitted for publication January 8, 2018. Accepted for publication November 29, 2018.

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Open the PDF for in another window
• Cite Icon Cite
• Get Permissions
• Search Site

John T. Ratelle , Adam P. Sawatsky , Thomas J. Beckman; Quantitative Research Methods in Medical Education. Anesthesiology 2019; 131:23–35 doi: https://doi.org/10.1097/ALN.0000000000002727

Download citation file:

• Ris (Zotero)
• Reference Manager

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

Image: J. P. Rathmell and Terri Navarette.

There has been an explosion of research in the field of medical education. A search of PubMed demonstrates that more than 40,000 articles have been indexed under the medical subject heading “Medical Education” since 2010, which is more than the total number of articles indexed under this heading in the 1980s and 1990s combined. Keeping up to date requires that practicing clinicians have the skills to interpret and appraise the quality of research articles, especially when serving as editors, reviewers, and consumers of the literature.

While medical education shares many characteristics with other biomedical fields, substantial particularities exist. We recognize that practicing clinicians may not be familiar with the nuances of education research and how to assess its quality. Therefore, our purpose is to provide a review of quantitative research methodologies in medical education. Specifically, we describe a structure that can be used when conducting or evaluating medical education research articles.

Clarifying the research purpose is an essential first step when reading or conducting scholarship in medical education. 1   Medical education research can serve a variety of purposes, from advancing the science of learning to improving the outcomes of medical trainees and the patients they care for. However, a well-designed study has limited value if it addresses vague, redundant, or unimportant medical education research questions.

What is the research topic and why is it important? What is unknown about the research topic? Why is further research necessary?

What is the conceptual framework being used to approach the study?

What is the statement of study intent?

What are the research methodology and study design? Are they appropriate for the study objective(s)?

Which threats to internal validity are most relevant for the study?

What is the outcome and how was it measured?

Can the results be trusted? What is the validity and reliability of the measurements?

How were research subjects selected? Is the research sample representative of the target population?

Was the data analysis appropriate for the study design and type of data?

What is the effect size? Do the results have educational significance?

Fortunately, there are steps to ensure that the purpose of a research study is clear and logical. Table 1   2–5   outlines these steps, which will be described in detail in the following sections. We describe these elements not as a simple “checklist,” but as an advanced organizer that can be used to understand a medical education research study. These steps can also be used by clinician educators who are new to the field of education research and who wish to conduct scholarship in medical education.

Steps in Clarifying the Purpose of a Research Study in Medical Education

Literature Review and Problem Statement

A literature review is the first step in clarifying the purpose of a medical education research article. 2 , 5 , 6   When conducting scholarship in medical education, a literature review helps researchers develop an understanding of their topic of interest. This understanding includes both existing knowledge about the topic as well as key gaps in the literature, which aids the researcher in refining their study question. Additionally, a literature review helps researchers identify conceptual frameworks that have been used to approach the research topic. 2  

When reading scholarship in medical education, a successful literature review provides background information so that even someone unfamiliar with the research topic can understand the rationale for the study. Located in the introduction of the manuscript, the literature review guides the reader through what is already known in a manner that highlights the importance of the research topic. The literature review should also identify key gaps in the literature so the reader can understand the need for further research. This gap description includes an explicit problem statement that summarizes the important issues and provides a reason for the study. 2 , 4   The following is one example of a problem statement:

“Identifying gaps in the competency of anesthesia residents in time for intervention is critical to patient safety and an effective learning system… [However], few available instruments relate to complex behavioral performance or provide descriptors…that could inform subsequent feedback, individualized teaching, remediation, and curriculum revision.” 7  

This problem statement articulates the research topic (identifying resident performance gaps), why it is important (to intervene for the sake of learning and patient safety), and current gaps in the literature (few tools are available to assess resident performance). The researchers have now underscored why further research is needed and have helped readers anticipate the overarching goals of their study (to develop an instrument to measure anesthesiology resident performance). 4  

The Conceptual Framework

Following the literature review and articulation of the problem statement, the next step in clarifying the research purpose is to select a conceptual framework that can be applied to the research topic. Conceptual frameworks are “ways of thinking about a problem or a study, or ways of representing how complex things work.” 3   Just as clinical trials are informed by basic science research in the laboratory, conceptual frameworks often serve as the “basic science” that informs scholarship in medical education. At a fundamental level, conceptual frameworks provide a structured approach to solving the problem identified in the problem statement.

Conceptual frameworks may take the form of theories, principles, or models that help to explain the research problem by identifying its essential variables or elements. Alternatively, conceptual frameworks may represent evidence-based best practices that researchers can apply to an issue identified in the problem statement. 3   Importantly, there is no single best conceptual framework for a particular research topic, although the choice of a conceptual framework is often informed by the literature review and knowing which conceptual frameworks have been used in similar research. 8   For further information on selecting a conceptual framework for research in medical education, we direct readers to the work of Bordage 3   and Irby et al. 9  

To illustrate how different conceptual frameworks can be applied to a research problem, suppose you encounter a study to reduce the frequency of communication errors among anesthesiology residents during day-to-night handoff. Table 2 10 , 11   identifies two different conceptual frameworks researchers might use to approach the task. The first framework, cognitive load theory, has been proposed as a conceptual framework to identify potential variables that may lead to handoff errors. 12   Specifically, cognitive load theory identifies the three factors that affect short-term memory and thus may lead to communication errors:

Conceptual Frameworks to Address the Issue of Handoff Errors in the Intensive Care Unit

Intrinsic load: Inherent complexity or difficulty of the information the resident is trying to learn ( e.g. , complex patients).

Extraneous load: Distractions or demands on short-term memory that are not related to the information the resident is trying to learn ( e.g. , background noise, interruptions).

Germane load: Effort or mental strategies used by the resident to organize and understand the information he/she is trying to learn ( e.g. , teach back, note taking).

Using cognitive load theory as a conceptual framework, researchers may design an intervention to reduce extraneous load and help the resident remember the overnight to-do’s. An example might be dedicated, pager-free handoff times where distractions are minimized.

The second framework identified in table 2 , the I-PASS (Illness severity, Patient summary, Action list, Situational awareness and contingency planning, and Synthesis by receiver) handoff mnemonic, 11   is an evidence-based best practice that, when incorporated as part of a handoff bundle, has been shown to reduce handoff errors on pediatric wards. 13   Researchers choosing this conceptual framework may adapt some or all of the I-PASS elements for resident handoffs in the intensive care unit.

Note that both of the conceptual frameworks outlined above provide researchers with a structured approach to addressing the issue of handoff errors; one is not necessarily better than the other. Indeed, it is possible for researchers to use both frameworks when designing their study. Ultimately, we provide this example to demonstrate the necessity of selecting conceptual frameworks to clarify the research purpose. 3 , 8   Readers should look for conceptual frameworks in the introduction section and should be wary of their omission, as commonly seen in less well-developed medical education research articles. 14  

Statement of Study Intent

After reviewing the literature, articulating the problem statement, and selecting a conceptual framework to address the research topic, the final step in clarifying the research purpose is the statement of study intent. The statement of study intent is arguably the most important element of framing the study because it makes the research purpose explicit. 2   Consider the following example:

This study aimed to test the hypothesis that the introduction of the BASIC Examination was associated with an accelerated knowledge acquisition during residency training, as measured by increments in annual ITE scores. 15  

This statement of study intent succinctly identifies several key study elements including the population (anesthesiology residents), the intervention/independent variable (introduction of the BASIC Examination), the outcome/dependent variable (knowledge acquisition, as measure by in In-training Examination [ITE] scores), and the hypothesized relationship between the independent and dependent variable (the authors hypothesize a positive correlation between the BASIC examination and the speed of knowledge acquisition). 6 , 14  

The statement of study intent will sometimes manifest as a research objective, rather than hypothesis or question. In such instances there may not be explicit independent and dependent variables, but the study population and research aim should be clearly identified. The following is an example:

“In this report, we present the results of 3 [years] of course data with respect to the practice improvements proposed by participating anesthesiologists and their success in implementing those plans. Specifically, our primary aim is to assess the frequency and type of improvements that were completed and any factors that influence completion.” 16  

The statement of study intent is the logical culmination of the literature review, problem statement, and conceptual framework, and is a transition point between the Introduction and Methods sections of a medical education research report. Nonetheless, a systematic review of experimental research in medical education demonstrated that statements of study intent are absent in the majority of articles. 14   When reading a medical education research article where the statement of study intent is absent, it may be necessary to infer the research aim by gathering information from the Introduction and Methods sections. In these cases, it can be useful to identify the following key elements 6 , 14 , 17   :

Population of interest/type of learner ( e.g. , pain medicine fellow or anesthesiology residents)

Independent/predictor variable ( e.g. , educational intervention or characteristic of the learners)

Dependent/outcome variable ( e.g. , intubation skills or knowledge of anesthetic agents)

Relationship between the variables ( e.g. , “improve” or “mitigate”)

Occasionally, it may be difficult to differentiate the independent study variable from the dependent study variable. 17   For example, consider a study aiming to measure the relationship between burnout and personal debt among anesthesiology residents. Do the researchers believe burnout might lead to high personal debt, or that high personal debt may lead to burnout? This “chicken or egg” conundrum reinforces the importance of the conceptual framework which, if present, should serve as an explanation or rationale for the predicted relationship between study variables.

Research methodology is the “…design or plan that shapes the methods to be used in a study.” 1   Essentially, methodology is the general strategy for answering a research question, whereas methods are the specific steps and techniques that are used to collect data and implement the strategy. Our objective here is to provide an overview of quantitative methodologies ( i.e. , approaches) in medical education research.

The choice of research methodology is made by balancing the approach that best answers the research question against the feasibility of completing the study. There is no perfect methodology because each has its own potential caveats, flaws and/or sources of bias. Before delving into an overview of the methodologies, it is important to highlight common sources of bias in education research. We use the term internal validity to describe the degree to which the findings of a research study represent “the truth,” as opposed to some alternative hypothesis or variables. 18   Table 3   18–20   provides a list of common threats to internal validity in medical education research, along with tactics to mitigate these threats.

Threats to Internal Validity and Strategies to Mitigate Their Effects

Experimental Research

The fundamental tenet of experimental research is the manipulation of an independent or experimental variable to measure its effect on a dependent or outcome variable.

True Experiment

True experimental study designs minimize threats to internal validity by randomizing study subjects to experimental and control groups. Through ensuring that differences between groups are—beyond the intervention/variable of interest—purely due to chance, researchers reduce the internal validity threats related to subject characteristics, time-related maturation, and regression to the mean. 18 , 19  

Quasi-experiment

There are many instances in medical education where randomization may not be feasible or ethical. For instance, researchers wanting to test the effect of a new curriculum among medical students may not be able to randomize learners due to competing curricular obligations and schedules. In these cases, researchers may be forced to assign subjects to experimental and control groups based upon some other criterion beyond randomization, such as different classrooms or different sections of the same course. This process, called quasi-randomization, does not inherently lead to internal validity threats, as long as research investigators are mindful of measuring and controlling for extraneous variables between study groups. 19  

Single-group Methodologies

All experimental study designs compare two or more groups: experimental and control. A common experimental study design in medical education research is the single-group pretest–posttest design, which compares a group of learners before and after the implementation of an intervention. 21   In essence, a single-group pre–post design compares an experimental group ( i.e. , postintervention) to a “no-intervention” control group ( i.e. , preintervention). 19   This study design is problematic for several reasons. Consider the following hypothetical example: A research article reports the effects of a year-long intubation curriculum for first-year anesthesiology residents. All residents participate in monthly, half-day workshops over the course of an academic year. The article reports a positive effect on residents’ skills as demonstrated by a significant improvement in intubation success rates at the end of the year when compared to the beginning.

This study does little to advance the science of learning among anesthesiology residents. While this hypothetical report demonstrates an improvement in residents’ intubation success before versus after the intervention, it does not tell why the workshop worked, how it compares to other educational interventions, or how it fits in to the broader picture of anesthesia training.

Single-group pre–post study designs open themselves to a myriad of threats to internal validity. 20   In our hypothetical example, the improvement in residents’ intubation skills may have been due to other educational experience(s) ( i.e. , implementation threat) and/or improvement in manual dexterity that occurred naturally with time ( i.e. , maturation threat), rather than the airway curriculum. Consequently, single-group pre–post studies should be interpreted with caution. 18  

Repeated testing, before and after the intervention, is one strategy that can be used to reduce the some of the inherent limitations of the single-group study design. Repeated pretesting can mitigate the effect of regression toward the mean, a statistical phenomenon whereby low pretest scores tend to move closer to the mean on subsequent testing (regardless of intervention). 20   Likewise, repeated posttesting at multiple time intervals can provide potentially useful information about the short- and long-term effects of an intervention ( e.g. , the “durability” of the gain in knowledge, skill, or attitude).

Observational Research

Unlike experimental studies, observational research does not involve manipulation of any variables. These studies often involve measuring associations, developing psychometric instruments, or conducting surveys.

Association Research

Association research seeks to identify relationships between two or more variables within a group or groups (correlational research), or similarities/differences between two or more existing groups (causal–comparative research). For example, correlational research might seek to measure the relationship between burnout and educational debt among anesthesiology residents, while causal–comparative research may seek to measure differences in educational debt and/or burnout between anesthesiology and surgery residents. Notably, association research may identify relationships between variables, but does not necessarily support a causal relationship between them.

Psychometric and Survey Research

Psychometric instruments measure a psychologic or cognitive construct such as knowledge, satisfaction, beliefs, and symptoms. Surveys are one type of psychometric instrument, but many other types exist, such as evaluations of direct observation, written examinations, or screening tools. 22   Psychometric instruments are ubiquitous in medical education research and can be used to describe a trait within a study population ( e.g. , rates of depression among medical students) or to measure associations between study variables ( e.g. , association between depression and board scores among medical students).

Psychometric and survey research studies are prone to the internal validity threats listed in table 3 , particularly those relating to mortality, location, and instrumentation. 18   Additionally, readers must ensure that the instrument scores can be trusted to truly represent the construct being measured. For example, suppose you encounter a research article demonstrating a positive association between attending physician teaching effectiveness as measured by a survey of medical students, and the frequency with which the attending physician provides coffee and doughnuts on rounds. Can we be confident that this survey administered to medical students is truly measuring teaching effectiveness? Or is it simply measuring the attending physician’s “likability”? Issues related to measurement and the trustworthiness of data are described in detail in the following section on measurement and the related issues of validity and reliability.

Measurement refers to “the assigning of numbers to individuals in a systematic way as a means of representing properties of the individuals.” 23   Research data can only be trusted insofar as we trust the measurement used to obtain the data. Measurement is of particular importance in medical education research because many of the constructs being measured ( e.g. , knowledge, skill, attitudes) are abstract and subject to measurement error. 24   This section highlights two specific issues related to the trustworthiness of data: the validity and reliability of measurements.

Validity regarding the scores of a measurement instrument “refers to the degree to which evidence and theory support the interpretations of the [instrument’s results] for the proposed use of the [instrument].” 25   In essence, do we believe the results obtained from a measurement really represent what we were trying to measure? Note that validity evidence for the scores of a measurement instrument is separate from the internal validity of a research study. Several frameworks for validity evidence exist. Table 4 2 , 22 , 26   represents the most commonly used framework, developed by Messick, 27   which identifies sources of validity evidence—to support the target construct—from five main categories: content, response process, internal structure, relations to other variables, and consequences.

Sources of Validity Evidence for Measurement Instruments

Reliability

Reliability refers to the consistency of scores for a measurement instrument. 22 , 25 , 28   For an instrument to be reliable, we would anticipate that two individuals rating the same object of measurement in a specific context would provide the same scores. 25   Further, if the scores for an instrument are reliable between raters of the same object of measurement, then we can extrapolate that any difference in scores between two objects represents a true difference across the sample, and is not due to random variation in measurement. 29   Reliability can be demonstrated through a variety of methods such as internal consistency ( e.g. , Cronbach’s alpha), temporal stability ( e.g. , test–retest reliability), interrater agreement ( e.g. , intraclass correlation coefficient), and generalizability theory (generalizability coefficient). 22 , 29  

Example of a Validity and Reliability Argument

This section provides an illustration of validity and reliability in medical education. We use the signaling questions outlined in table 4 to make a validity and reliability argument for the Harvard Assessment of Anesthesia Resident Performance (HARP) instrument. 7   The HARP was developed by Blum et al. to measure the performance of anesthesia trainees that is required to provide safe anesthetic care to patients. According to the authors, the HARP is designed to be used “…as part of a multiscenario, simulation-based assessment” of resident performance. 7  

Content Validity: Does the Instrument’s Content Represent the Construct Being Measured?

To demonstrate content validity, instrument developers should describe the construct being measured and how the instrument was developed, and justify their approach. 25   The HARP is intended to measure resident performance in the critical domains required to provide safe anesthetic care. As such, investigators note that the HARP items were created through a two-step process. First, the instrument’s developers interviewed anesthesiologists with experience in resident education to identify the key traits needed for successful completion of anesthesia residency training. Second, the authors used a modified Delphi process to synthesize the responses into five key behaviors: (1) formulate a clear anesthetic plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize one’s limits. 7 , 30  

Response Process Validity: Are Raters Interpreting the Instrument Items as Intended?

In the case of the HARP, the developers included a scoring rubric with behavioral anchors to ensure that faculty raters could clearly identify how resident performance in each domain should be scored. 7  

Internal Structure Validity: Do Instrument Items Measuring Similar Constructs Yield Homogenous Results? Do Instrument Items Measuring Different Constructs Yield Heterogeneous Results?

Item-correlation for the HARP demonstrated a high degree of correlation between some items ( e.g. , formulating a plan and modifying the plan under changing conditions) and a lower degree of correlation between other items ( e.g. , formulating a plan and identifying performance improvement opportunities). 30   This finding is expected since the items within the HARP are designed to assess separate performance domains, and we would expect residents’ functioning to vary across domains.

Relationship to Other Variables’ Validity: Do Instrument Scores Correlate with Other Measures of Similar or Different Constructs as Expected?

As it applies to the HARP, one would expect that the performance of anesthesia residents will improve over the course of training. Indeed, HARP scores were found to be generally higher among third-year residents compared to first-year residents. 30  

Consequence Validity: Are Instrument Results Being Used as Intended? Are There Unintended or Negative Uses of the Instrument Results?

While investigators did not intentionally seek out consequence validity evidence for the HARP, unanticipated consequences of HARP scores were identified by the authors as follows:

“Data indicated that CA-3s had a lower percentage of worrisome scores (rating 2 or lower) than CA-1s… However, it is concerning that any CA-3s had any worrisome scores…low performance of some CA-3 residents, albeit in the simulated environment, suggests opportunities for training improvement.” 30  

That is, using the HARP to measure the performance of CA-3 anesthesia residents had the unintended consequence of identifying the need for improvement in resident training.

Reliability: Are the Instrument’s Scores Reproducible and Consistent between Raters?

The HARP was applied by two raters for every resident in the study across seven different simulation scenarios. The investigators conducted a generalizability study of HARP scores to estimate the variance in assessment scores that was due to the resident, the rater, and the scenario. They found little variance was due to the rater ( i.e. , scores were consistent between raters), indicating a high level of reliability. 7  

Sampling refers to the selection of research subjects ( i.e. , the sample) from a larger group of eligible individuals ( i.e. , the population). 31   Effective sampling leads to the inclusion of research subjects who represent the larger population of interest. Alternatively, ineffective sampling may lead to the selection of research subjects who are significantly different from the target population. Imagine that researchers want to explore the relationship between burnout and educational debt among pain medicine specialists. The researchers distribute a survey to 1,000 pain medicine specialists (the population), but only 300 individuals complete the survey (the sample). This result is problematic because the characteristics of those individuals who completed the survey and the entire population of pain medicine specialists may be fundamentally different. It is possible that the 300 study subjects may be experiencing more burnout and/or debt, and thus, were more motivated to complete the survey. Alternatively, the 700 nonresponders might have been too busy to respond and even more burned out than the 300 responders, which would suggest that the study findings were even more amplified than actually observed.

When evaluating a medical education research article, it is important to identify the sampling technique the researchers employed, how it might have influenced the results, and whether the results apply to the target population. 24  

Sampling Techniques

Sampling techniques generally fall into two categories: probability- or nonprobability-based. Probability-based sampling ensures that each individual within the target population has an equal opportunity of being selected as a research subject. Most commonly, this is done through random sampling, which should lead to a sample of research subjects that is similar to the target population. If significant differences between sample and population exist, those differences should be due to random chance, rather than systematic bias. The difference between data from a random sample and that from the population is referred to as sampling error. 24  

Nonprobability-based sampling involves selecting research participants such that inclusion of some individuals may be more likely than the inclusion of others. 31   Convenience sampling is one such example and involves selection of research subjects based upon ease or opportuneness. Convenience sampling is common in medical education research, but, as outlined in the example at the beginning of this section, it can lead to sampling bias. 24   When evaluating an article that uses nonprobability-based sampling, it is important to look for participation/response rate. In general, a participation rate of less than 75% should be viewed with skepticism. 21   Additionally, it is important to determine whether characteristics of participants and nonparticipants were reported and if significant differences between the two groups exist.

Interpreting medical education research requires a basic understanding of common ways in which quantitative data are analyzed and displayed. In this section, we highlight two broad topics that are of particular importance when evaluating research articles.

The Nature of the Measurement Variable

Measurement variables in quantitative research generally fall into three categories: nominal, ordinal, or interval. 24   Nominal variables (sometimes called categorical variables) involve data that can be placed into discrete categories without a specific order or structure. Examples include sex (male or female) and professional degree (M.D., D.O., M.B.B.S., etc .) where there is no clear hierarchical order to the categories. Ordinal variables can be ranked according to some criterion, but the spacing between categories may not be equal. Examples of ordinal variables may include measurements of satisfaction (satisfied vs . unsatisfied), agreement (disagree vs . agree), and educational experience (medical student, resident, fellow). As it applies to educational experience, it is noteworthy that even though education can be quantified in years, the spacing between years ( i.e. , educational “growth”) remains unequal. For instance, the difference in performance between second- and third-year medical students is dramatically different than third- and fourth-year medical students. Interval variables can also be ranked according to some criteria, but, unlike ordinal variables, the spacing between variable categories is equal. Examples of interval variables include test scores and salary. However, the conceptual boundaries between these measurement variables are not always clear, as in the case where ordinal scales can be assumed to have the properties of an interval scale, so long as the data’s distribution is not substantially skewed. 32  

Understanding the nature of the measurement variable is important when evaluating how the data are analyzed and reported. Medical education research commonly uses measurement instruments with items that are rated on Likert-type scales, whereby the respondent is asked to assess their level of agreement with a given statement. The response is often translated into a corresponding number ( e.g. , 1 = strongly disagree, 3 = neutral, 5 = strongly agree). It is remarkable that scores from Likert-type scales are sometimes not normally distributed ( i.e. , are skewed toward one end of the scale), indicating that the spacing between scores is unequal and the variable is ordinal in nature. In these cases, it is recommended to report results as frequencies or medians, rather than means and SDs. 33  

Consider an article evaluating medical students’ satisfaction with a new curriculum. Researchers measure satisfaction using a Likert-type scale (1 = very unsatisfied, 2 = unsatisfied, 3 = neutral, 4 = satisfied, 5 = very satisfied). A total of 20 medical students evaluate the curriculum, 10 of whom rate their satisfaction as “satisfied,” and 10 of whom rate it as “very satisfied.” In this case, it does not make much sense to report an average score of 4.5; it makes more sense to report results in terms of frequency ( e.g. , half of the students were “very satisfied” with the curriculum, and half were not).

Effect Size and CIs

In medical education, as in other research disciplines, it is common to report statistically significant results ( i.e. , small P values) in order to increase the likelihood of publication. 34 , 35   However, a significant P value in itself does necessarily represent the educational impact of the study results. A statement like “Intervention x was associated with a significant improvement in learners’ intubation skill compared to education intervention y ( P < 0.05)” tells us that there was a less than 5% chance that the difference in improvement between interventions x and y was due to chance. Yet that does not mean that the study intervention necessarily caused the nonchance results, or indicate whether the between-group difference is educationally significant. Therefore, readers should consider looking beyond the P value to effect size and/or CI when interpreting the study results. 36 , 37  

Effect size is “the magnitude of the difference between two groups,” which helps to quantify the educational significance of the research results. 37   Common measures of effect size include Cohen’s d (standardized difference between two means), risk ratio (compares binary outcomes between two groups), and Pearson’s r correlation (linear relationship between two continuous variables). 37   CIs represent “a range of values around a sample mean or proportion” and are a measure of precision. 31   While effect size and CI give more useful information than simple statistical significance, they are commonly omitted from medical education research articles. 35   In such instances, readers should be wary of overinterpreting a P value in isolation. For further information effect size and CI, we direct readers the work of Sullivan and Feinn 37   and Hulley et al. 31  

In this final section, we identify instruments that can be used to evaluate the quality of quantitative medical education research articles. To this point, we have focused on framing the study and research methodologies and identifying potential pitfalls to consider when appraising a specific article. This is important because how a study is framed and the choice of methodology require some subjective interpretation. Fortunately, there are several instruments available for evaluating medical education research methods and providing a structured approach to the evaluation process.

The Medical Education Research Study Quality Instrument (MERSQI) 21   and the Newcastle Ottawa Scale-Education (NOS-E) 38   are two commonly used instruments, both of which have an extensive body of validity evidence to support the interpretation of their scores. Table 5 21 , 39   provides more detail regarding the MERSQI, which includes evaluation of study design, sampling, data type, validity, data analysis, and outcomes. We have found that applying the MERSQI to manuscripts, articles, and protocols has intrinsic educational value, because this practice of application familiarizes MERSQI users with fundamental principles of medical education research. One aspect of the MERSQI that deserves special mention is the section on evaluating outcomes based on Kirkpatrick’s widely recognized hierarchy of reaction, learning, behavior, and results ( table 5 ; fig .). 40   Validity evidence for the scores of the MERSQI include its operational definitions to improve response process, excellent reliability, and internal consistency, as well as high correlation with other measures of study quality, likelihood of publication, citation rate, and an association between MERSQI score and the likelihood of study funding. 21 , 41   Additionally, consequence validity for the MERSQI scores has been demonstrated by its utility for identifying and disseminating high-quality research in medical education. 42  

Kirkpatrick’s hierarchy of outcomes as applied to education research. Reaction = Level 1, Learning = Level 2, Behavior = Level 3, Results = Level 4. Outcomes become more meaningful, yet more difficult to achieve, when progressing from Level 1 through Level 4. Adapted with permission from Beckman and Cook, 2007. 2  

The Medical Education Research Study Quality Instrument for Evaluating the Quality of Medical Education Research

The NOS-E is a newer tool to evaluate the quality of medication education research. It was developed as a modification of the Newcastle-Ottawa Scale 43   for appraising the quality of nonrandomized studies. The NOS-E includes items focusing on the representativeness of the experimental group, selection and compatibility of the control group, missing data/study retention, and blinding of outcome assessors. 38 , 39   Additional validity evidence for NOS-E scores includes operational definitions to improve response process, excellent reliability and internal consistency, and its correlation with other measures of study quality. 39   Notably, the complete NOS-E, along with its scoring rubric, can found in the article by Cook and Reed. 39  

A recent comparison of the MERSQI and NOS-E found acceptable interrater reliability and good correlation between the two instruments 39   However, noted differences exist between the MERSQI and NOS-E. Specifically, the MERSQI may be applied to a broad range of study designs, including experimental and cross-sectional research. Additionally, the MERSQI addresses issues related to measurement validity and data analysis, and places emphasis on educational outcomes. On the other hand, the NOS-E focuses specifically on experimental study designs, and on issues related to sampling techniques and outcome assessment. 39   Ultimately, the MERSQI and NOS-E are complementary tools that may be used together when evaluating the quality of medical education research.

Conclusions

This article provides an overview of quantitative research in medical education, underscores the main components of education research, and provides a general framework for evaluating research quality. We highlighted the importance of framing a study with respect to purpose, conceptual framework, and statement of study intent. We reviewed the most common research methodologies, along with threats to the validity of a study and its measurement instruments. Finally, we identified two complementary instruments, the MERSQI and NOS-E, for evaluating the quality of a medical education research study.

Bordage G: Conceptual frameworks to illuminate and magnify. Medical education. 2009; 43(4):312–9.

Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: Theory and application. The American journal of medicine. 2006; 119(2):166. e7–166. e116.

Franenkel JR, Wallen NE, Hyun HH: How to Design and Evaluate Research in Education. 9th edition. New York, McGraw-Hill Education, 2015.

Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB: Designing clinical research. 4th edition. Philadelphia, Lippincott Williams & Wilkins, 2011.

Irby BJ, Brown G, Lara-Alecio R, Jackson S: The Handbook of Educational Theories. Charlotte, NC, Information Age Publishing, Inc., 2015

Standards for Educational and Psychological Testing (American Educational Research Association & American Psychological Association, 2014)

Swanwick T: Understanding medical education: Evidence, theory and practice, 2nd edition. Wiley-Blackwell, 2013.

Sullivan GM, Artino Jr AR: Analyzing and interpreting data from Likert-type scales. Journal of graduate medical education. 2013; 5(4):541–2.

Sullivan GM, Feinn R: Using effect size—or why the P value is not enough. Journal of graduate medical education. 2012; 4(3):279–82.

Tavakol M, Sandars J: Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical teacher. 2014; 36(10):838–48.

Support was provided solely from institutional and/or departmental sources.

The authors declare no competing interests.

Citing articles via

Most viewed, email alerts, related articles, social media, affiliations.

• ASA Practice Parameters
• Online First
• Author Resource Center
• About the Journal
• Editorial Board
• Rights & Permissions
• Online ISSN 1528-1175
• Print ISSN 0003-3022
• Anesthesiology
• ASA Monitor

• Terms & Conditions Privacy Policy
• Manage Cookie Preferences
• © Copyright 2024 American Society of Anesthesiologists

This Feature Is Available To Subscribers Only

Sign In or Create an Account

• Research article
• Open access
• Published: 06 January 2021

Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study

• Aaron J. Harries   ORCID: orcid.org/0000-0001-7107-0995 1 ,
• Carmen Lee 1 ,
• Lee Jones 2 ,
• Robert M. Rodriguez 1 ,
• John A. Davis 2 ,
• Megan Boysen-Osborn 3 ,
• Kathleen J. Kashima 4 ,
• N. Kevin Krane 5 ,
• Guenevere Rae 6 ,
• Nicholas Kman 7 ,
• Jodi M. Langsfeld 8 &
• Marianne Juarez 1  

BMC Medical Education volume  21 , Article number:  14 ( 2021 ) Cite this article

139k Accesses

166 Citations

38 Altmetric

Metrics details

The COVID-19 pandemic disrupted the United States (US) medical education system with the necessary, yet unprecedented Association of American Medical Colleges (AAMC) national recommendation to pause all student clinical rotations with in-person patient care. This study is a quantitative analysis investigating the educational and psychological effects of the pandemic on US medical students and their reactions to the AAMC recommendation in order to inform medical education policy.

The authors sent a cross-sectional survey via email to medical students in their clinical training years at six medical schools during the initial peak phase of the COVID-19 pandemic. Survey questions aimed to evaluate students’ perceptions of COVID-19’s impact on medical education; ethical obligations during a pandemic; infection risk; anxiety and burnout; willingness and needed preparations to return to clinical rotations.

Seven hundred forty-one (29.5%) students responded. Nearly all students (93.7%) were not involved in clinical rotations with in-person patient contact at the time the study was conducted. Reactions to being removed were mixed, with 75.8% feeling this was appropriate, 34.7% guilty, 33.5% disappointed, and 27.0% relieved.

Most students (74.7%) agreed the pandemic had significantly disrupted their medical education, and believed they should continue with normal clinical rotations during this pandemic (61.3%). When asked if they would accept the risk of infection with COVID-19 if they returned to the clinical setting, 83.4% agreed.

Students reported the pandemic had moderate effects on their stress and anxiety levels with 84.1% of respondents feeling at least somewhat anxious. Adequate personal protective equipment (PPE) (53.5%) was the most important factor to feel safe returning to clinical rotations, followed by adequate testing for infection (19.3%) and antibody testing (16.2%).

Conclusions

The COVID-19 pandemic disrupted the education of US medical students in their clinical training years. The majority of students wanted to return to clinical rotations and were willing to accept the risk of COVID-19 infection. Students were most concerned with having enough PPE if allowed to return to clinical activities.

Peer Review reports

The COVID-19 pandemic has tested the limits of healthcare systems and challenged conventional practices in medical education. The rapid evolution of the pandemic dictated that critical decisions regarding the training of medical students in the United States (US) be made expeditiously, without significant input or guidance from the students themselves. On March 17, 2020, for the first time in modern US history, the Association of American Medical Colleges (AAMC), the largest national governing body of US medical schools, released guidance recommending that medical students immediately pause all clinical rotations to allow time to obtain additional information about the risks of COVID-19 and prepare for safe participation in the future. This decisive action would also conserve scarce resources such as personal protective equipment (PPE) and testing kits; minimize exposure of healthcare workers (HCWs) and the general population; and protect students’ education and wellbeing [ 1 ].

A similar precedent was set outside of the US during the SARS-CoV1 epidemic in 2003, where an initial cluster of infection in medical students in Hong Kong resulted in students being removed from hospital systems where SARS surfaced, including Hong Kong, Singapore and Toronto [ 2 , 3 ]. Later, studies demonstrated that the exclusion of Canadian students from those clinical environments resulted in frustration at lost learning opportunities and students’ inability to help [ 3 ]. International evidence also suggests that medical students perceive an ethical obligation to participate in pandemic response, and are willing to participate in scenarios similar to the current COVID-19 crisis, even when they believe the risk of infection to themselves to be high [ 4 , 5 , 6 ].

The sudden removal of some US medical students from educational settings has occurred previously in the wake of local disasters, with significant academic and personal impacts. In 2005, it was estimated that one-third of medical students experienced some degree of depression or post-traumatic stress disorder (PTSD) after Hurricane Katrina resulted in the closure of Tulane University School of Medicine [ 7 ].

Prior to the current COVID-19 pandemic, we found no studies investigating the effects of pandemics on the US medical education system or its students. The limited pool of evidence on medical student perceptions comes from two earlier global coronavirus surges, SARS and MERS, and studies of student anxiety related to pandemics are also limited to non-US populations [ 3 , 8 , 9 ]. Given the unprecedented nature of the current COVID-19 pandemic, there is concern that students may be missing out on meaningful educational experiences and months of clinical training with unknown effects on their current well-being or professional trajectory [ 10 ].

Our study, conducted during the initial peak phase of the COVID-19 pandemic, reports students’ perceptions of COVID-19’s impact on: medical student education; ethical obligations during a pandemic; perceptions of infection risk; anxiety and burnout; willingness to return to clinical rotations; and needed preparations to return safely. This data may help inform policies regarding the roles of medical students in clinical training during the current pandemic and prepare for the possibility of future pandemics.

We conducted a cross-sectional survey during the initial peak phase of the COVID-19 pandemic in the United States, from 4/20/20 to 5/25/20, via email sent to all clinically rotating medical students at six US medical schools: University of California San Francisco School of Medicine (San Francisco, CA), University of California Irvine School of Medicine (Irvine, CA), Tulane University School of Medicine (New Orleans, LA), University of Illinois College of Medicine (Chicago, Peoria, Rockford, and Urbana, IL), Ohio State University College of Medicine (Columbus, OH), and Zucker School of Medicine at Hofstra/Northwell (Hempstead, NY). Traditional undergraduate medical education in the US comprises 4 years of medical school with 2 years of primarily pre-clinical classroom learning followed by 2 years of clinical training involving direct patient care. Study participants were defined as medical students involved in their clinical training years at whom the AAMC guidance statement was directed. Depending on the curricular schedule of each medical school, this included intended graduation class years of 2020 (graduating 4th year student), 2021 (rising 4th year student), and 2022 (rising 3rd year student), exclusive of planned time off. Participating schools were specifically chosen to represent a broad spectrum of students from different regions of the country (West, South, Midwest, East) with variable COVID-19 prevalence. We excluded medical students not yet involved in clinical rotations. This study was deemed exempt by the respective Institutional Review Boards.

We developed a survey instrument modeled after a survey used in a previously published peer reviewed study evaluating the effects of the COVID-19 pandemic on Emergency Physicians, which incorporated items from validated stress scales [ 11 ]. The survey was modified for use in medical students to assess perceptions of the following domains: perceived impact on medical student education; ethical beliefs surrounding obligations to participate clinically during the pandemic; perceptions of personal infection risk; anxiety and burnout related to the pandemic; willingness to return to clinical rotations; and preparation needed for students to feel safe in the clinical environment. Once created, the survey underwent an iterative process of input and review from our team of authors with experience in survey methodology and psychometric measures to allow for optimization of content and validity. We tested a pilot of our preliminary instrument on five medical students to ensure question clarity, and confirm completion of the survey in approximately 10 min. The final survey consisted of 29 Likert, yes/no, multiple choice, and free response questions. Both medical school deans and student class representatives distributed the survey via email, with three follow-up emails to increase response rates. Data was collected anonymously.

For example, to assess the impact on students’ anxiety, participants were asked, “How much has the COVID-19 pandemic affected your stress or anxiety levels?” using a unipolar 7-point scale (1 = not at all, 4 = somewhat, 7 = extremely). To assess willingness to return to clinical rotations, participants were asked to rate on a bipolar scale (1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = neither disagree nor agree, 5 = somewhat agree, 6 = agree, and 7 = strongly agree) their agreement with the statement: “to the extent possible, medical students should continue with normal clinical rotations during this pandemic.” (Survey Instrument, Supplemental Table  1 ).

Survey data was managed using Qualtrics hosted by the University of California, San Francisco. For data analysis we used STATA v15.1 (Stata Corp, College Station, TX). We summarized respondent characteristics and key responses as raw counts, frequency percent, medians and interquartile ranges (IQR). For responses to bipolar questions, we combined positive responses (somewhat agree, agree, or strongly agree) into an agreement percentage. To compare differences in medians we used a signed rank test with p value < 0.05 to show statistical difference. In a secondary analysis we stratified data to compare questions within key domains amongst the following sub-groups: female versus male, graduation year, local community COVID-19 prevalence (high, medium, low), and students on clinical rotations with in-person patient care. This secondary analysis used a chi square test with p value < 0.05 to show statistical difference between sub-group agreement percentages.

Of 2511 students contacted, we received 741 responses (29.5% response rate). Of these, 63.9% of respondents were female and 35.1% were male, with 1.0% reporting a different gender identity; 27.7% of responses came from the class of 2020, 53.5% from the class of 2021, and 18.7% from the class of 2022. (Demographics, Table 1 ).

Most student respondents (74.9%) had a clinical rotation that was cut short or canceled due to COVID-19 and 93.7% reported not being involved in clinical rotations with in-person patient contact at the time of the study. Regarding students’ perceptions of cancelled rotations (allowing for multiple reactions), 75.8% felt this was appropriate, 34.7% felt guilty for not being able to help patients and colleagues, 33.5% felt disappointed, and 27.0% felt relieved.

Most students (74.7%) agreed that their medical education had been significantly disrupted by the pandemic. Students also felt they were able to find meaningful learning experiences during the pandemic (72.1%). Free response examples included: taking a novel COVID-19 pandemic elective course, telehealth patient care, clinical rotations transitioned to virtual online courses, research or education electives, clinical and non-clinical COVID-19-related volunteering, and self-guided independent study electives. Students felt their medical schools were doing everything they could to help students adjust (72.7%). Overall, respondents felt the pandemic had interfered with their ability to develop skills needed to prepare for residency (61.4%), though fewer (45.7%) felt it had interfered with their ability to apply to residency. (Educational Impact, Fig.  1 ).

Perceived educational impacts of the COVID-19 pandemic on medical students

A majority of medical students agreed they should be allowed to continue with normal clinical rotations during this pandemic (61.3%). Most students agreed (83.4%) that they accepted the risk of being infected with COVID-19, if they returned. When asked if students should be allowed to volunteer in clinical settings even if there is not a healthcare worker (HCW) shortage, 63.5% agreed; however, in the case of a HCW shortage only 19.5% believed students should be required to volunteer clinically. (Willingness to Participate Clinically, Fig.  2 ).

Willingness to participate clinically during the COVID-19 pandemic

When asked if they perceived a moral, ethical, or professional obligation for medical students to help, 37.8% agreed that medical students have such an obligation during the current pandemic. This is in contrast to their perceptions of physicians: 87.1% of students agreed with a physician obligation to help during the COVID-19 pandemic. For both groups, students were asked if this obligation persisted without adequate PPE: only 10.9% of students believed medical students had this obligation, while 34.0% agreed physicians had this obligation. (Ethical Obligation, Fig.  3 ).

Ethical obligation to volunteer during the COVID-19 pandemic

Given the assumption that there will not be a COVID-19 vaccine until 2021, students felt the single most important factor in a safe return to clinical rotations was having access to adequate PPE (53.3%), followed by adequate testing for infection (19.3%) and antibody testing for possible immunity (16.2%). Few students (5%) stated that nothing would make them feel comfortable until a vaccine is available. On a 1–7 scale (1 = not at all, 4 = somewhat, 7 = extremely), students felt somewhat prepared to use PPE during this pandemic in the clinical setting, median = 4 (IQR 4,6), and somewhat confident identifying symptoms most concerning for COVID-19, median = 4 (IQR 4,5). Students preferred to learn about PPE via video demonstration (76.7%), online modules (47.7%), and in-person or Zoom style conferences (44.7%).

Students believed they were likely to contract COVID-19 in general (75.6%), independent of a return to the clinical environment. Most respondents believed that missing some school or work would be a likely outcome (90.5%), and only a minority of students believed that hospitalization (22.1%) or death (4.3%) was slightly, moderately, or extremely likely.

On a 1–7 scale (1 = not at all, 4 = somewhat, and 7 = extremely), the median (IQR) reported effect of the COVID-19 pandemic on students’ stress or anxiety level was 5 (4, 6) with 84.1% of respondents feeling at least somewhat anxious due to the pandemic. Students’ perceived emotional exhaustion and burnout before the pandemic was a median = 2 (IQR 2,4) and since the pandemic started a median = 4 (IQR 2,5) with a median difference Δ = 2, p value < 0.001.

Secondary analysis of key questions revealed statistical differences between sub-groups. Women were significantly more likely than men to agree that the pandemic had affected their anxiety. Several significant differences existed for the class of 2020 when compared to the classes of 2021 and 2022: they were less likely to report disruptions to their education, to prefer to return to rotations, and to report an effect on anxiety. There were no significant differences with students who were still involved with in-person patient care compared with those who were not. In comparing areas with high COVID-19 prevalence at the time of the survey (New York and Louisiana) with medium (Illinois and Ohio) and low prevalence (California), students were less likely to report that the pandemic had disrupted their education. Students in low prevalence areas were most likely to agree that medical students should return to rotations. There were no differences between prevalence groups in accepting the risk of infection to return, or subjective anxiety effects. (Stratification, Table  2 ).

The COVID-19 pandemic has fundamentally transformed education at all levels - from preschool to postgraduate. Although changes to K-12 and college education have been well documented [ 12 , 13 ], there have been very few studies to date investigating the effects of COVID-19 on undergraduate medical education [ 14 ]. To maintain the delicate balance between student safety and wellbeing, and the time-sensitive need to train future physicians, student input must guide decisions regarding their roles in the clinical arena. Student concerns related to the pandemic, paired with their desire to return to rotations despite the risks, suggest that medical students may take on emotional burdens as members of the patient care team even when not present in the clinical environment. This study offers insight into how best to support medical students as they return to clinical rotations, how to prepare them for successful careers ahead, and how to plan for their potential roles in future pandemics.

Previous international studies of medical student attitudes towards hypothetical influenza-like pandemics demonstrated a willingness (80%) [ 4 ] and a perceived ethical obligation to volunteer (77 and 70%), despite 40% of Canadian students in one study perceiving a high likelihood of becoming infected [ 5 , 6 ]. Amidst the current COVID-19 pandemic, our participants reported less agreement with a medical student ethical obligation to volunteer in the clinical setting at 37.8%, but believed in a higher likelihood of becoming infected at 75.6%. Their willingness to be allowed to volunteer freely (63.5%) may suggest that the stresses of an ongoing pandemic alter students’ perceptions of the ethical requirement more than their willingness to help. Students overwhelmingly agreed that physicians had an ethical obligation to provide care during the COVID-19 pandemic (87.1%), possibly reflecting how they view the ethical transition from student to physician, or differences between paid professionals and paying for an education.

At the time our study was conducted, there were widespread concerns for possible HCW shortages. It was unclear whether medical students would be called to volunteer when residents became ill, or even graduate early to start residency training immediately (as occurred at half of schools surveyed). This timing allowed us to capture a truly unique perspective amongst medical students, a majority of whom reported increased anxiety and burnout due to the pandemic. At the same time, students felt that their medical schools were doing everything possible to support them, perhaps driven by virtual town halls and daily communication updates.

Trends in secondary analysis show important differences in the impacts of the pandemic. Women were more likely to report increased anxiety as compared to men, which may reflect broader gender differences in medical student anxiety [ 15 ] but requires more study to rule out different pandemic stresses by gender. Graduating medical students (class of 2020) overall described less impact on medical education and anxiety, a decreased desire to return to rotations, but equal acceptance of the risk of infection in clinical settings, possibly reflecting a focus on their upcoming intern year rather than the remaining months of undergraduate medical education. Since this class’s responses decreased overall agreement on these questions, educational impacts and anxiety effects may have been even greater had they been assessed further from graduation. Interestingly, students from areas with high local COVID-19 prevalence (New York and Louisiana) reported a less significant effect of the pandemic on their education, a paradoxical result that may indicate that medical student tolerance for the disruptions was greater in high-prevalence areas, as these students were removed at the same, if not higher, rates as their peers. Our results suggest that in future waves of the current pandemic or other disasters, students may be more patient with educational impacts when they have more immediate awareness of strains on the healthcare system.

A limitation of our study was the survey response rate, which was anticipated given the challenges students were facing. Some may not have been living near campus; others may have stopped reading emails due to early graduation or limited access to email; and some would likely be dealing with additional personal challenges related to the pandemic. We attempted to increase response rates by having the study sent directly from medical school deans and leadership, as well as respective class representatives, and by sending reminders for completion. The survey was not incentivized, and a higher response rate in the class of 2021 across all schools may indicate that students who felt their education was most affected were most likely to respond. We addressed this potential source of bias in the secondary analysis, which showed no differences between 2021 and 2022 respondents. Another limitation was the inherent issue with survey data collection of missing responses for some questions that occurred in a small number of surveys. This resulted in slight variability in the total responses received for certain questions, which were not statistically significant. To be transparent about this limitation, we presented our data by stating each total response and denominator in the Tables.

This initial study lays the groundwork for future investigations and next steps. With 72.1% of students agreeing that they were able to find meaningful learning in spite of the pandemic, future research should investigate novel learning modalities that were successful during this time. Educators should consider additional training on PPE use, given only moderate levels of student comfort in this area, which may be best received via video. It is also important to study the long-term effects of missing several months of essential clinical training and identifying competencies that may not have been achieved, since students perceived a significant disruption to their ability to prepare skills for residency. Next steps could be to study curriculum interventions, such as capstone boot camps and targeted didactic skills training, to help students feel more comfortable as they transition into residency. Educators must also acknowledge that some students may not feel comfortable returning to the clinical environment until a vaccine becomes available (5%) and ensure they are equally supported. Lastly, it is vital to further investigate the mental health effects of the pandemic on medical students, identifying subgroups with additional stressors, needs related to anxiety or possible PTSD, and ways to minimize these negative effects.

In this cross-sectional survey, conducted during the initial peak phase of the COVID-19 pandemic, we capture a snapshot of the effects of the pandemic on US medical students and gain insight into their reactions to the unprecedented AAMC national recommendation for removal from clinical rotations. Student respondents from across the US similarly recognized a significant disruption to their medical education, shared a desire to continue with in-person rotations, and were willing to accept the risk of infection with COVID-19. Our novel results provide a solid foundation to help shape medical student roles in the clinical environment during this pandemic and future outbreaks.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Association of American Medical Colleges. Interim Guidance on Medical Students’ Participation in Direct Patient Contact Activities: Principles and Guidelines. https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak . Published March 17, 2020. Accessed April 1, 2020.

Clark J. Fear of SARS thwarts medical education in Toronto. BMJ. 2003;326(7393):784. https://doi.org/10.1136/bmj.326.7393.784/c .

Article   Google Scholar  

Loh LC, Ali AM, Ang TH, Chelliah A. Impact of a spreading epidemic on medical students. Malays J Med Sci. 2006;13(2):30–6.

Google Scholar  

Mortelmans LJ, Bouman SJ, Gaakeer MI, Dieltiens G, Anseeuw K, Sabbe MB. Dutch senior medical students and disaster medicine: a national survey. Int J Emerg Med. 2015;8(1):77. https://doi.org/10.1186/s12245-015-0077-0 .

Huapaya JA, Maquera-Afaray J, García PJ, Cárcamo C, Cieza JA. Conocimientos, prácticas y actitudes hacia el voluntariado ante una influenza pandémica: estudio transversal con estudiantes de medicina en Perú [Knowledge, practices and attitudes toward volunteer work in an influenza pandemic: cross-sectional study with Peruvian medical students]. Medwave. 2015;15(4):e6136Published 2015 May 8. https://doi.org/10.5867/medwave.2015.04.6136 .

Herman B, Rosychuk RJ, Bailey T, Lake R, Yonge O, Marrie TJ. Medical students and pandemic influenza. Emerg Infect Dis. 2007;13(11):1781–3. https://doi.org/10.3201/eid1311.070279 .

Kahn MJ, Markert RJ, Johnson JE, Owens D, Krane NK. Psychiatric issues and answers following hurricane Katrina. Acad Psychiatry. 2007;31(3):200–4. https://doi.org/10.1176/appi.ap.31.3.200 .

Al-Rabiaah A, Temsah MH, Al-Eyadhy AA, et al. Middle East respiratory syndrome-Corona virus (MERS-CoV) associated stress among medical students at a university teaching hospital in Saudi Arabia. J Infect Public Health. 2020;13(5):687–91. https://doi.org/10.1016/j.jiph.2020.01.005 .

Wong JG, Cheung EP, Cheung V, et al. Psychological responses to the SARS outbreak in healthcare students in Hong Kong. Med Teach. 2004;26(7):657–9. https://doi.org/10.1080/01421590400006572 .

Stokes DC. Senior medical students in the COVID-19 response: an opportunity to be proactive. Acad Emerg Med. 2020;27(4):343–5. https://doi.org/10.1111/acem.13972 .

Rodriguez RM, Medak AJ, Baumann BM, et al. Academic emergency medicine physicians’ anxiety levels, stressors, and potential mitigation measures during the acceleration phase of the COVID-19 pandemic. Acad Emerg Med. 2020;27(8):700–7. https://doi.org/10.1111/acem.14065 .

Sahu P. Closure of universities due to coronavirus disease 2019 (COVID-19): impact on education and mental health of students and academic staff. Cureus. 2020;12(4):e7541Published 2020 Apr 4. https://doi.org/10.7759/cureus.7541 .

Reimers FM, Schleicher A. A framework to guide an education response to the COVID-19 pandemic of 2020: OECD. https://www.hm.ee/sites/default/files/framework_guide_v1_002_harward.pdf .

Choi B, Jegatheeswaran L, Minocha A, Alhilani M, Nakhoul M, Mutengesa E. The impact of the COVID-19 pandemic on final year medical students in the United Kingdom: a national survey. BMC Med Educ. 2020;20:206–16. https://doi.org/10.1186/s12909-020-02117-1 .

Dyrbye LN, Thomas MR, Shanafelt TD. Systematic review of depression, anxiety, and other indicators of psychological distress among U.S. and Canadian medical students. Acad Med. 2006;81(4):354–73. https://doi.org/10.1097/00001888-200604000-00009 .

Download references

Acknowledgments

The authors wish to thank Newton Addo, UCSF Statistician.

Author information

Authors and affiliations.

Department of Emergency Medicine, University of California San Francisco School of Medicine, San Francisco General Hospital, 1001 Potrero Avenue, Building 5, Room #6A4, San Francisco, California, 94110, USA

Aaron J. Harries, Carmen Lee, Robert M. Rodriguez & Marianne Juarez

University of California San Francisco School of Medicine, San Francisco, California, USA

Lee Jones & John A. Davis

Clinical Emergency Medicine, University of California Irvine School of Medicine, Irvine, CA, USA

Megan Boysen-Osborn

University of Illinois College of Medicine, Chicago, IL, USA

Kathleen J. Kashima

Deming Department of Medicine, Tulane University School of Medicine, New Orleans, Louisiana, USA

N. Kevin Krane

Basic Science Education, Tulane University School of Medicine, New Orleans, Louisiana, USA

Guenevere Rae

Emergency Medicine, Ohio State College of Medicine, Columbus, OH, USA

Nicholas Kman

Department of Science Education, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, USA

Jodi M. Langsfeld

You can also search for this author in PubMed   Google Scholar

Contributions

All authors made substantial contributions to the study and met the specific conditions listed in the BMC Medical Education editorial policy for authorship. All authors have read and approved the manuscript. AH as principal investigator contributed to study design, survey instrument creation, IRB submission for his respective medical school, acquisition of data and recruitment of other participating medical schools, data analysis, writing and editing the manuscript. CL contributed to background literature review, study design, survey instrument creation, acquisition of data, data analysis, writing and editing the manuscript. LJ contributed to study design, survey instrument creation, acquisition of data from his respective medical school and recruitment of other participating medical schools, data analysis, and editing the manuscript. RR contributed to study design, survey instrument creation, data analysis, writing and editing the manuscript. JD contributed to study design, survey instrument creation, recruitment of other participating medical schools, data analysis, and editing the manuscript. MBO contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. KK contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. NKK contributed as individual site co-principal investigator obtaining IRB exemption acceptance and acquisition of data from his respective medical school along with editing the manuscript. GR contributed as individual site co-principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. NK contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from his respective medical school along with editing the manuscript. JL contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. MJ contributed to study design, survey instrument creation, data analysis, writing and editing the manuscript.

Corresponding authors

Correspondence to Aaron J. Harries or Marianne Juarez .

Ethics declarations

Ethics approval and consent to participate.

This study was reviewed and deemed exempt by each participating medical school’s Institutional Review Board (IRB): University of California San Francisco School of Medicine, IRB# 20–30712, Reference# 280106, Tulane University School of Medicine, Reference # 2020–331, University of Illinois College of Medicine), IRB Protocol # 2012–0783, Ohio State University College of Medicine, Study ID# 2020E0463, Zucker School of Medicine at Hofstra/Northwell, Reference # 20200527-SOM-LAN-1, University of California Irvine School of Medicine, submitted self-exemption IRB form. In accordance with the IRB exemption approval, each survey participant received an email consent describing the study and their optional participation.

Consent for publication

This manuscript does not contain any individualized person’s data, therefore consent for publication was not necessary according to the IRB exemption approval.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

Survey Instrument

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Harries, A.J., Lee, C., Jones, L. et al. Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study. BMC Med Educ 21 , 14 (2021). https://doi.org/10.1186/s12909-020-02462-1

Download citation

Received : 29 July 2020

Accepted : 16 December 2020

Published : 06 January 2021

DOI : https://doi.org/10.1186/s12909-020-02462-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Undergraduate medical education
• COVID-19 pandemic
• Medical student anxiety

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

• Search Menu

Sign in through your institution

• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Numismatics
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Social History
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Legal System - Costs and Funding
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Restitution
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Social Issues in Business and Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Management of Land and Natural Resources (Social Science)
• Natural Disasters (Environment)
• Pollution and Threats to the Environment (Social Science)
• Social Impact of Environmental Issues (Social Science)
• Sustainability
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Politics of Development
• Public Policy
• Public Administration
• Qualitative Political Methodology
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Disability Studies
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

53 Quantitative research methods in medical education

• Published: October 2013
• Cite Icon Cite
• Permissions Icon Permissions

Quantitative research in medical education tends to be predominantly observational research based on survey or correlational studies. As researchers strive towards making inferences about the impact of education interventions, a shift towards experimental research designs may enhance the quality and conclusions made in medical education. The establishment of experiment research designs, where interventions (i.e. curriculum, teaching or assessment interventions) are tested with an experimental group and either a comparison or controlled group of learners, may allow researchers to overcome validity concerns and infer potential cause–effect generalizations. There are a number of internal and external validity concerns that researchers need to be conscious of when designing their own or looking at others’ experimental research studies. The selection of a research design for any study should fit within the parameters of the stated research question or hypothesis. In quantitative research, the findings will reflect the reliability and validity (psychometric characteristics) of the measured outcomes or dependent variables (such as changes in knowledge, skills, or attitudes) used to assess the effectiveness of the medical education intervention (the independent variable of interest). It is important to remember that not all quantitative research involves experimental studies—important results can also be drawn from quantitative observational studies. This chapter outlines commonly used quantitative methods in medical education research. It explains their theoretical underpinnings, the evidence base for their use, and gives practical guidance on their application. It concludes with a section on the role of meta-analyses of quantitative research in medical education.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Skip to main content
• Skip to FDA Search
• Skip to in this section menu
• Skip to footer links

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

U.S. Food and Drug Administration

•   Search
•   Menu
• News & Events for Human Drugs
• CDER Conversations

Streamlining Drug Development and Improving Patient Care: CDER Quantitative Medicine Center of Excellence

CDER recently launched the new CDER Quantitative Medicine (QM) Center of Excellence (CoE). The goal of this CoE is to facilitate and coordinate the continuous evolution and consistent application of QM across CDER. QM involves the development and application of exposure-based, biological, and quantitative modeling and simulation approaches derived from nonclinical, clinical, and real-world sources to inform drug development, regulatory decision-making, and patient care. These approaches contribute to the totality of understanding of a drug's benefits and risks, helping to advance therapeutic medical product development and inform regulatory decision-making.

In this CDER Conversation, Rajanikanth Madabushi, lead for the QM CoE and associate director for Guidance and Scientific Policy in the Office of Clinical Pharmacology in the Office of Translational Science Super Office explains the purpose of the new CoE, provides an overview of current activities and resources, and shares ways the CoE can advance drug development and improve patient care.

What prompted CDER to establish the QM CoE?

For decades, CDER has been at the forefront of advancing QM approaches to inform premarket product review and post-market product assessment. Given the tremendous growth in QM, we see many opportunities to maximize synergies across CDER by centrally coordinating efforts through strategic planning and execution. By establishing this CoE, CDER is putting a stake in the ground to advance initiatives that spur innovation and promote integration of QM approaches in CDER. The CoE provides the organizational framework to foster collaboration and coordination of QM efforts across CDER and facilitate engagements with all stakeholders to advance therapeutic medical product development, inform regulatory decision-making, and promote public health.

What is the goal of the CDER QM CoE and what types of projects will fall under the CoE?

The primary goal of the QM CoE is to facilitate and coordinate the continuous evolution and consistent application of QM across CDER. CDER’s QM CoE is a center-wide effort and will:

• Spearhead QM-related policy development and best practices to facilitate the consistent use of QM approaches during drug development and regulatory assessment.
• Facilitate systematic outreach to scientific societies, patient advocacy groups, and other key stakeholders.
• Coordinate CDER’s efforts around QM education and training.

The CoE will take on key initiatives in the areas of Applied Science Policy, Strategic Planning and Coordination, and Multidisciplinary Education and Exchange.

How do Model-Informed Drug Development (MIDD) and the MIDD Paired Meeting Program fit into the new QM CoE?

As mentioned earlier, QM involves the development and application of exposure-based, biological, and quantitative modeling and simulation approaches derived from nonclinical, clinical, and real-world sources to inform drug development, regulatory decision-making, and patient care. As such, MIDD, an approach that involves developing and applying exposure-based biological and statistical models derived from preclinical and clinical data sources to inform drug development or regulatory decision-making, is a fundamental QM activity. As the use of QM approaches in drug development and regulatory assessment have increased, CDER has created new, dedicated forums for regulatory engagements with drug development stakeholders, including the MIDD Paired Meeting Program , Complex Innovative Trial Design (CID) Program , and the Model-Integrated Evidence (MIE) Pilot Program .

CDER has well-established processes for interactions through these forums. In general, the establishment of the QM CoE is not expected to alter any of the existing processes of these programs. As the central coordinating body, the CoE will facilitate sharing experiences and disseminate lessons learned across these forums to promote the consistent application of QM approaches throughout the life cycle of product development.

We recently noticed that the QM CoE shared an educational series related to MIDD. Can you share more about this resource? Who is the course meant for, and what can they expect to learn from the course?

One of the core focus areas of CDER QM CoE is multidisciplinary education and exchange, including the development, coordination, and dissemination of accessible education and training resources. As part of these efforts, CDER’s QM CoE coordinated the release of the educational series “ Model-Informed Drug Development (MIDD): Methods Advancing Medical Products to Patients ” aimed at increasing community-wide knowledge of these approaches.

The educational content was originally developed for regulatory scientists by experts in the field in partnership with the Critical Path Institute and funded by CDER’s Office of Translational Sciences. The series includes eight modules with 29 presentations and more than 10 hours of content. In this course, participants will learn the applications of MIDD approaches across the development of drugs and biological products. The course is intended as a foundational introduction to the applications of MIDD approaches for those involved in drug development and review.

How do you anticipate the QM CoE will advance drug development and, ultimately, improve patient care?

New drug development is a multifaceted, complex, high-risk endeavor involving a range of scientific, financial, and regulatory challenges. Innovative technologies, tools, and approaches have been proposed to increase drug development efficiency and optimize treatments reaching patients. QM is one of those innovations.

For example, QM approaches play a critical role in organizing diverse data sets, enhancing the mechanistic understanding of disease, exploring alternate study designs, informing dose selection and optimization, identifying subpopulations for therapy, evaluating critical regulatory review questions such as evidence of effectiveness, strategizing lifecycle plans in the post-approval setting, and supporting development of complex generic products. For more examples of how QM approaches have contributed to drug development and review, I encourage readers to visit the web page for the recording and slides of our recently held public workshop, Streamlining Drug Development and Improving Public Health through Quantitative Medicine: An Introduction to the CDER Quantitative Medicine Center of Excellence .

By continuing to foster the integration and broader adoption of QM approaches across CDER, the CoE can help advance drug development and inform regulatory decision-making. As a result, the QM CoE is anticipated to help streamline drug development and accelerate the delivery of safe, effective, therapeutically optimized medicines to the public.

Looking to the future, what are some upcoming projects under the QM CoE?

To advance the science and foster the use of QM methods in drug development, the CoE plans to:

• Identify and prioritize gaps in knowledge and determine areas of further research and development in CDER.
• Develop a strategic plan and create task forces to achieve its goals.
• Create additional opportunities to engage with the QM CoE in public forums.
• Provide additional education and training opportunities for all stakeholders involved in drug development and review.
• Develop a repository of case studies of when and how quantitative approaches helped streamline drug development, for example, by informing clinical trial design, optimizing drug dosages, helping determine patient populations, or improving benefit/safety profiles.

The QM CoE encourages all stakeholders to explore how they can help the QM CoE advance and integrate QM to maximize societal benefit and patient care. Interested parties can visit the CDER QM CoE web page or email [email protected] for more information.

• Research article
• Open access
• Published: 03 May 2019

Qualitative and quantitative research of medication review and drug-related problems in Hungarian community pharmacies: a pilot study

• András Szilvay 1 ,
• Orsolya Somogyi 1 ,
• Attiláné Meskó 1 ,
• Romána Zelkó 1 &
• Balázs Hankó 1  

BMC Health Services Research volume  19 , Article number:  282 ( 2019 ) Cite this article

7983 Accesses

8 Citations

Metrics details

Pharmaceutical care is the pharmacist’s contribution to the care of individuals to optimize medicines use and improve health outcomes. The primary tool of pharmaceutical care is medication review. Defining and classifying Drug-Related Problems (DRPs) is an essential pillar of the medication review. Our objectives were to perform a pilot of medication review in Hungarian community pharmacies, a DRP classification was applied for the first time. Also, our goal was the qualitative and quantitative description of the discovered DRPs, and of the interventions for their solution in order to prove the safety relevance of the service and to map out the competence limits of GPs and community pharmacists to drug therapy.

The project took place in Hungarian community pharmacies. The study was performed with patients taking vitamin K antagonist (VKA) and/or ACE inhibitor and NSAID simultaneously (ACEI-NSAID). 61 pharmacists and 606 patients participated in the project. Pharmacists reviewed the medication for 3 months and the classification of DRPs was performed (category of DRP1 – DRP6). Patient data were statistically analyzed.

Patients consumed on average 7.9 ± 3.1 medications and other products. 571 DRPs were detected in 540 patients, averaging 1.06 DRPs per patient (SD = 1.07). The highest frequency category was DRP5 (non-quantitative safety problem; 51.0%). The most common root cause was an interaction (42.0%) and non-adherence (19.4%.). The most commonly used intervention was education (25.4%) and medication replacement by the pharmacist (20.1%). The changing of the frequency and dosage in any direction were negligible.

Conclusions

Patients are struggling with many DRPs that can be assessed and categorized by this system and which remain unrecognizable without pharmacists. Further projects need to be developed to assist in the development of physician-pharmacist cooperation and the widespread dissemination of pharmaceutical care.

Peer Review reports

According to the definition of Pharmaceutical Care Network Europe “Pharmaceutical Care is the pharmacist’s contribution to the care of individuals to optimize medicines use and improve health outcomes”. [ 1 ] The goal of the pharmacists is to collect the patient’s medications (Rx, OTC) and other products (e.g. dietary supplements) to ensure their necessity, efficacy and safety. [ 2 ]

The main tool of pharmaceutical care is medication review, “a structured, critical examination of a patient’s medicines with the objective of reaching an agreement with the patient about treatment, optimizing the impact of medicines, minimizing the number of medication-related problems and reducing waste”. [ 3 ] The best way to make medication review is in collaboration with the patients and their general practitioners. [ 4 , 5 ] To demonstrate the benefits of medication review, several but controversial articles have published. It reduces the number of cases requiring emergency care [ 6 , 7 ], the number of (re) hospitalizations, but its beneficial effects on quality of life, adverse drug reactions and mortality are non-significant in high-risk groups. [ 7 , 8 ] However according to other articles it reduces the number of (unnecessary) drugs [ 9 , 10 , 11 , 12 ], it helps to detect and solve drug-related problems (DRPs) [ 9 , 13 , 14 , 15 , 16 , 17 ], especially in collaboration with hospital pharmacists [ 18 ], it increases the patients’ trust in the therapy [ 19 ] and the cost-effectiveness of the treatment [ 20 ].

Defining and classifying drug-related problems is an essential pillar of the medication review. The drug-related problems are “situations in which in the process of use of medicines cause or may cause the appearance of a negative outcome associated with the medication.” [ 21 ] There are many reasons for the drug-related problems, which may result that drug therapy is not achieving its purpose or even becoming harmful. There are more than 20 types of DRP classification system in the literature, which differ in, e.g. DRP groups and methodology. [ 22 ] It is also important to involve patients in the process of detecting drug-related problems. [ 23 ]

The aim of our research was to perform a pilot of medication review in Hungarian community pharmacies as part of basic pharmaceutical care, using a drug-related problem classification for the first time to lay the foundation for wider adoption of this service in Hungary. Also, our goal was the qualitative and quantitative description of the discovered drug-related problems, and of the interventions for their solution in order to prove the safety relevance of the service and to map out the competence limits of GPs and community pharmacists to drug therapy. The latter can contribute to the development of a future “target model” of doctor-patient-pharmacist cooperation.

Description of the project

The project took place between December 2015 and August 2016. The data were collected by pharmacists (they have not received monetary compensation) participating in specialist training at Semmelweis University. The participation of pharmacists was obligatory to complete the second year of the training, and the cooperative pharmacies were their own workplaces. All participating pharmacies were accredited at the Semmelweis University. All the participating pharmacists from all around the country went to Budapest and participated in one-day training at Semmelweis University, which included the description and requirements of the project, and the presentation of the drug-related problem classification.

The pharmacists at the beginning of the project visited family practitioners working near the pharmacy, practically those whose patients often go to the pharmacy. They were invited to participate in the project. After that, the patients were involved in the pharmacy. The study was performed with patients taking vitamin K antagonist (VKA) and/or ACE inhibitor and NSAID simultaneously (ACEI-NSAID). The latter is considered as a high-risk group because of the increased chance of renal failure [ 24 ] and the potential inadequacy of the therapy [ 25 ]. A patient could have been in both categories. Patients had to be at least 18 years old, had to buy medicines themselves and had to be the patients of the general practitioner involved. All pharmacists tried to have around 10 patients. The process of the first and further occasions is described in Fig.  1 .

The process of the occasions

Characteristics of participating pharmacies and patients

Data for patients and pharmacies in the study are shown in Table  1 . 61 pharmacists took part in 61 pharmacies. The survey was close to nationwide coverage (16 of 20 counties). Most of the pharmacies were in the capital (35.6%). 606 patients participated in the project (9.9 patients/pharmacy; SD = 3.0), 57.3% were women and 42.7% were males. 497 patients (mean = 8.1; SD = 2.2) took part in every requested meeting with the pharmacist (traced patients), 18.0% of the patients left the project. However, we used data from all the patients involved, not only from traced patients. The average age of patients was 65.0 years (SD = 11.9). 55.6% of the participants took ACE inhibitor and NSAID simultaneously, 39.8% of them took vitamin K antagonists, and 4.6% of them were included in both categories.

• Medication review

Pharmacists have been conducting medication review at each consultation; they looked at medication taken from the point of necessity, effectiveness and safety. The medication review and the classification of drug-related problems were performed according to the Third Consensus of Granada on Drug-Related Problems classification system [ 21 ] and to the Hungarian National Committee of Pharmaceutical Care Metabolic Syndrome Pharmaceutical Care Programme. [ 26 ] In the process of assessing drug-related problems, the pharmacist classified the DRPs into six classes and identified the root cause. (Table  2 ).

The number and cause of the drug-related problem were recorded by the pharmacist. In addition, the intervention was also recorded. The medication review was performed at all pharmacist meetings. Anonymous patient data were statistically analyzed.

Statistical analysis

Statistical calculations were performed using SPSS 20.0. After the descriptive statistical analysis, two sample t test, paired sample t test, and a variance analysis were performed on continuous data to detect differences and correlations. When calculating the Pearson correlation coefficient, the p value for the correlation coefficient was < 0.005. For discrete data, the Kruskal-Wallis test and chi-square test were used. Control of normality was performed with Kolmogorov Smirnov test. The significance level was p = 0.05.

Ethics approval and consent to participate

The project was implemented with the support and cooperation of the National Health Development Institute’s Primary Care Directorate [ 27 ]. The unified professional protocol made available in the course of the co-operation is a document agreed with the Primary Service Directorate. We have not received a waiver of ethics approval since the participation in the questionnaire survey, and the pharmaceutical service was not linked to one Institute (University) and was absolutely free and undoubtedly noninvasive, so IRB deemed unnecessary according to the similar national regulations. In Hungary according to Regulation No 44/2004 MoHSFA and Act XLVII of 1997, pharmacies did not need to be individually ethically licensed, because the service complies with statutory regulations, and pharmacies are legally entitled to perform such activities [ 28 , 29 , 30 , 31 ]. Verbal informed consent was obtained from all participants in the pharmacies; no written consent was required according to the Act CLIV of 1997 on Health (noninvasive pharmaceutical service and questionnaire survey) [ 32 ].

The investigation was a free service of pharmacies with operating licenses. The patients involved voluntarily participated in the process. Patients participating in the project received verbal information in accordance with the national regulations mentioned above. Qualified pharmacists conducted the project. The data were handled by pharmacy and health data management according to Act XLVII of 1997. Data were transmitted without personal information to process the results. The personal and health data of the patients included in the study were not damaged.

Descriptive results

In the assessment of drug-related problems, 540 patients from 606 patients were collected and analyzed. On average, patients consumed 7.9 ± 3.1 medications and other products: 6.3 were prescription drug (SD = 2.8), 1.1 OTC (SD = 1.1) and 0.4 other product, for example dietary supplements (SD = 0.8). (Table 1 ).

During the study, 571 drug-related problems were detected in these 540 patients, averaging 1.06 DRP per patient (SD = 1.07). The highest frequency category was DRP5 (non-quantitative safety problem: 51.0%). Approximately one-fourth of cases (24.0%) belonged to DRP3 (non-quantitative ineffectiveness) and 10% to DRP1 (untreated health problem). DRP2 (Effect of unnecessary medicine), DRP4 (Quantitative ineffectiveness) and DRP6 (Quantitative safety problem) were less frequent (8.2, 4.6, 2.3%). (Fig.  2 -All patients).

The relative proportion of different drug-related problem categories per patient group (All patients: all the participating patients ( n  = 571); ACEI-NSAID: patients taking ACE inhibitor and NSAID simultaneously ( n  = 330); VKA: patients taking vitamin K antagonist ( n  = 212); Both: patients included in both categories ( n  = 29); DRP: drug-related problem (see Table 2 ))

Analyzing the root causes of drug-related problems, the most common was the interaction (42.0%), the second was non-adherence (19.4%). The Quantitative safety problem caused by improper dosage was the rarest (2.3%). (Fig.  3 -All patients ).

The relative proportion of the underlying cause of drug-related problems per patient group (All patients: all the participating patients ( n  = 571); ACEI-NSAID: patients taking ACE inhibitor and NSAID simultaneously ( n  = 330); VKA: patients taking vitamin K antagonist ( n  = 212); Both: patients included in both categories ( n  = 29); DRP: drug-related problem (see Table 2 ))

In the case of ACEI-NSAID patients, the DRP1 category appears to be higher (13.9%) than in the case of VKA patients (4.7%). The ratio was reversed in the case of DRP3 (22.1 and 27.8%) and DRP5 (48.2% or 52.4%), the latter is due to a higher rate of interactions. (Figs.  2 and 3 ) The ratio of interaction was extremely high for those patients who were in both categories (65.5%). (Fig. 3 ) However, these differences are not significant either in the number of drug problems or in the occurrence of the individual categories and causes. There was no “other” problem that cannot be categorized elsewhere.

Results of statistical analysis

There are no differences in the prevalence of drug-related problems between men and women (p = 0.070) and between the patients over and under 65 years. (p = 0.552).

There is a significant difference between the types of settlement in the occurrence of the drug-related problem. In the capital city, the pharmacists have found two DRPs per patient in a significantly higher ratio, while in other settlements it was markedly higher that the pharmacist found no mistake in the medication. There is a correlation between the number of DRPs and the total number of used medications, but the correlation is weak (Pearson correlation coefficient = 0.214 (p < 0.005)). The relationship between the number of prescription drugs and the number of drug-related problems is similar, somewhat lower (Pearson correlation coefficient = 0.152 (p < 0.005)).

Table  3 summarizes the rates of interventions used to eliminate drug-related problems. The most common intervention for the elimination of each underlying cause was indicated with bold number, while underlined number indicates the interventions with an incidence higher than 10%. Overall, the most commonly used intervention was education (25.4%) and medication replacement by the pharmacist (20.1%). More than 10% of the problems the intervention was not necessary (10.9%), or the pharmacist sends the patient to a physician (14.5%) or the pharmacist warned the GP (11.7%). The changing of the frequency and dosage in any direction were negligible.

Due to a large number of patients involved and the low drop-out rate, patients are interested and find the service provided by pharmacists useful. The fact that a large number of patients who had NSAIDs with an ACE inhibitor were included in the study underlined the relevance of this problem. Such a problem frequently does not show up at the doctor but at the pharmacy. The project involved a large number of patients with more than 5 medicines (also known as polypharmacy patients [ 33 ]).

It is noteworthy that patients use an average of 1 OTC drug on a regular basis, and that 4 out of 10 patients also use some other formulations (e.g. dietary supplements).

The use of these two product categories can only be supervised by the pharmacist. The patient’s medication is fully matched at expedition at the pharmacy only (Rx, OTC, other products) so the pharmacy service presented in the project plays an essential role in the assessment and resolution of drug-related problems. It is supported by a large number of DRPs that have been identified and classified in this project based on a drug-related problem classification system that has been used for the first time in Hungary. In addition to the reasons mentioned above, the overload of general practitioner services can also contribute. Among the DRP categories, there is a high amount of non-quantitative safety problems in all patient groups, which are mainly drug-drug or drug-other product interactions. The latter is also influenced by the patient’s involvement in the ACEI-NSAID group. However, we cannot talk about such a factor in the VKA group. This phenomenon is due to the uncontrolled use of the vast amounts of prescription and OTC medicines mentioned earlier and the other medicines are taken by 4 out of 10 patients. The problem may be solved by pharmacists who have resolved the situation in our research with education, medication replacement (especially OTC-OTC drug switching) and by sending the patient to the GP. In the case of interactions, “stop medication” has hardly occurred, and pharmacists seem to be hesitant to take this step, as they think that the physician is the one who competent to make this decision. Another major problem is the Non-quantitative ineffectiveness of the medication of a quarter of patients due primarily to their deliberate or unintended non-adherence.

Non-adherence is a widespread problem with chronic diseases for example in the case of conditions treated with ACE inhibitor and vitamin K antagonist. The pharmacist can help by detecting the problem and education. It is also important to mention that every tenth ACEI-NSAID patient is suffering from an untreated health problem. The research has shown that in many cases the pharmacist has noticed such a health problem, which has been solved by drug recommendation and by sending the patient to the GP. Based on these results, a medication review in the framework of basic pharmaceutical care can be a solution beyond the problems mentioned above in preventing the risks of self-medication.

In the case of medication review, it is also necessary to address the issue of competency conflict between the pharmacist and the general practitioner. By looking at the pharmacists’ interventions to resolve drug-related problems, we can see that 59.7% of the problems have been solved by the pharmacist without the involvement of a physician, primarily through education and the exchange of a patient’s drug with an OTC drug. The pharmacist in his/her own solved only about 5–6% of the cases by recommending a new drug or stopping a therapy, while the changing of frequency and dosage were as rare as possible without consultation with the physician. Pharmacists sent patients more to the physician without indicating the problem being diagnosed to them. Pharmacists preferred to send patients to doctors without consulting the GP, suggesting low levels of co-operation between two professions and pharmacists’ fears of doctors. Analyzing the effect of certain population factors on drug-related problems, it can be stated that among the examined factors, the number of DRPs is only influenced by the geographical location of the pharmacies. This, assuming that patients seek indirectly the pharmacy closest to their home first, refers indirectly to the influence of the type of residential township. Based on the results, it is assumed that a more extensive settlement poses a higher risk for patients, due to the less personal physician-patient and pharmacist-patient relationship, the more likely to be accessed by more accessible medical services. So the development of a regular pharmacist-patient relationship is of the utmost importance in this area.

Based on the results of the 540 patients surveyed in the 61 Hungarian pharmacies we can conclude that patients are struggling with many drug-related problems that can be assessed and categorized by this system and which remain unrecognizable without pharmacists. To achieve this, further projects need to be developed to assist in the development of physician-pharmacist cooperation and the widespread dissemination of pharmaceutical care. Our results provide a reasonable basis for the widespread use of medication review. In the future, it would be worthwhile extending the study to other patient groups, such as elderly patients with polypharmacy.

Abbreviations

Patients taking ACE inhibitor and NSAID simultaneously

• Drug-related problem

Patients taking vitamin K antagonist

Allemann SS, van Mil JW, Botermann L, Berger K, Griese N, Hersberger KE. Pharmaceutical care: the PCNE definition 2013. Int J Clin Pharm. 2014;36(3):544–55.

Article   PubMed   Google Scholar  

Cipolle RJ, Strand LM, Morley PC. Pharmaceutical care practice: the patient-centered approach to medication management. 3rd ed. McGraw Hill Professional; 2012.

Google Scholar  

Clyne W, Blenkinsopp A, Seal R. A guide to medication review 2008. 2008. http://www.cff.org.br/userfiles/52%20-%20CLYNE%20W%20A%20guide%20to%20medication%20review%202008.pdf . Accessed: 15.11. 2018.

Geurts MM, Talsma J, Brouwers JR, de Gier JJ. Medication review and reconciliation with cooperation between pharmacist and general practitioner and the benefit for the patient: a systematic review. Br J Clin Pharmacol. 2012;74(1):16–33.

Article   PubMed   PubMed Central   Google Scholar  

Kwint HF, Bermingham L, Faber A, Gussekloo J, Bouvy ML. The relationship between the extent of collaboration of general practitioners and pharmacists and the implementation of recommendations arising from medication review: a systematic review. Drugs Aging. 2013;30(2):91–102.

Christensen M, Lundh A. Medication review in hospitalised patients to reduce morbidity and mortality. Cochrane Database Syst Rev. 2016;(2).

Roth MT, Ivey JL, Esserman DA, Crisp G, Kurz J, Weinberger M. Individualized medication assessment and planning: optimizing medication use in older adults in the primary care setting. Pharmacotherapy. 2013;33(8):787–97.

Leendertse AJ, de Koning GH, Goudswaard AN, Belitser SV, Verhoef M, de Gier HJ, et al. Preventing hospital admissions by reviewing medication (PHARM) in primary care: an open controlled study in an elderly population. J Clin Pharm Ther. 2013;38(5):379–87.

Article   CAS   PubMed   Google Scholar  

Lenaghan E, Holland R, Brooks A. Home-based medication review in a high risk elderly population in primary care--the POLYMED randomised controlled trial. Age Ageing. 2007;36(3):292–7.

Frank C, Weir E. Deprescribing for older patients. Cmaj. 2014;186(18):1369–76.

Hurmuz MZM, Janus SIM, van Manen JG. Changes in medicine prescription following a medication review in older high-risk patients with polypharmacy. Int J Clin Pharm. 2018;40(2):480–7.

Mudge A, Radnedge K, Kasper K, Mullins R, Adsett J, Rofail S, et al. Effects of a pilot multidisciplinary clinic for frequent attending elderly patients on deprescribing. Aust Health Rev. 2016;40(1):86–91.

Cooper JA, Cadogan CA, Patterson SM, Kerse N, Bradley MC, Ryan C, et al. Interventions to improve the appropriate use of polypharmacy in older people: a Cochrane systematic review. BMJ Open. 2015;5(12).

Vinks TH, Egberts TC, de Lange TM, de Koning FH. Pharmacist-based medication review reduces potential drug-related problems in the elderly: the SMOG controlled trial. Drugs Aging. 2009;26(2):123–33.

Messerli M, Blozik E, Vriends N, Hersberger KE. Impact of a community pharmacist-led medication review on medicines use in patients on polypharmacy - a prospective randomised controlled trial. BMC Health Serv Res. 2016;16(1):145.

Lenander C, Bondesson Å, Viberg N, Beckman A, Midlöv P. Effects of medication reviews on use of potentially inappropriate medications in elderly patients; a cross-sectional study in Swedish primary care. BMC Health Serv Res. 2018;18(1):616.

Chua SS, Kok LC, Yusof FAM, Tang GH, Lee SWH, Efendie B, et al. Pharmaceutical care issues identified by pharmacists in patients with diabetes, hypertension or hyperlipidaemia in primary care settings. BMC Health Serv Res. 2012;12(1):388.

Ensing HT, Koster ES, Dubero DJ, van Dooren AA, Bouvy ML. Collaboration between hospital and community pharmacists to address drug-related problems: the HomeCoMe-program: Res Social Adm Pharm; 2018.

Geurts MM, Stewart RE, Brouwers JR, de Graeff PA, de Gier JJ. Patient beliefs about medicines and quality of life after a clinical medication review and follow-up by a pharmaceutical care plan: a study in elderly polypharmacy patients with a cardiovascular disorder. J Pharm Health Serv Res. 2015;6(4):171–6.

Article   Google Scholar  

Jodar-Sanchez F, Malet-Larrea A, Martin JJ, Garcia-Mochon L, Lopez Del Amo MP, Martinez-Martinez F, et al. Cost-utility analysis of a medication review with follow-up service for older adults with polypharmacy in community pharmacies in Spain: the conSIGUE program. Pharmacoeconomics. 2015;33(6):599–610.

Cd C. Tercer Consenso de Granada sobre problemas relacionados con medicamentos (PRM) y resultados negativos asociados a la medicación (RNM). Ars Pharmaceutica. 2007;48(1):5–7.

Basger BJ, Moles RJ, Chen TF. Application of drug-related problem (DRP) classification systems: a review of the literature. Eur J Clin Pharmacol. 2014;70(7):799–815.

Kwint HF, Faber A, Gussekloo J, Bouvy ML. The contribution of patient interviews to the identification of drug-related problems in home medication review. J Clin Pharm Ther. 2012;37(6):674–80.

Gooch K, Culleton BF, Manns BJ, Zhang J, Alfonso H, Tonelli M, et al. NSAID use and progression of chronic kidney disease. Am J Med. 2007;120(3):1–7.

Page RL 2nd, O'Bryant CL, Cheng D, Dow TJ, Ky B, Stein CM, et al. Drugs that may cause or exacerbate heart failure: a scientific statement from the American Heart Association. Circulation. 2016;134(6):32–69.

Hungarian National Committee of pharmaceutical care. Metabolic syndrome pharmaceutical care Programme. 2009.

National Health Development Institute's primary care directorate. http://www.egeszseg.hu/szakmai_oldalak/oldal/szervezeti-egysegek,alapellatasi-igazgatosag/alapellatasi-igazgatosag-feladata/ . Accessed: 22.11. 2018.

Regulation no 44/2004 MoHSFA on the ordering and issuing of medicinal products for human use. Sect. 22/A-C (2004).

Ministry of Human Capacities. A profession directive on medication review in the framework of basic pharmaceutical care. Egészségügyi Közlöny. 2013. http://hbcs.hu/uploads/jogszabaly/1529/fajlok/gyogyszerbiztonsagi_ellenorzes.pdf . Accessed: 22.11. 2018.

Act XLVII of 1997 on the management and protection of health data and related personal data, XLVII. Sect. 14/A (1997).

Regulation No 41/2007 Ministry for Health on the operation, Service and registration of public, branch, manual and institute pharmacies. Sect. 36/A (2007).

Act CLIV of 1997 on Health, CLIV Sect 15 (1997).

Jiron M, Herrada L, Rojas A, Lueiza A, Vega E, Buckel E, et al. Prevalence of potentially inappropriate medication prescribing among older adults in emergency Department in Chile, vol. 24; 2015. p. 102.

Download references

Acknowledgements

Thanks for all the pharmacist, pharmacy, general practitioner and patient involved in the project.

No funding was obtained for this study.

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to our privacy policy but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

University Pharmacy Department of Pharmacy Administration, Semmelweis University, Hungary; 7-9 Hőgyes Endre street, Budapest, H-1092, Hungary

András Szilvay, Orsolya Somogyi, Attiláné Meskó, Romána Zelkó & Balázs Hankó

You can also search for this author in PubMed   Google Scholar

Contributions

ASz co-ordinated the collection of data, planned the processing of the data and performed their processing. ASz wrote the first manuscript. OS and BH created the research plan, OS managed the conduct of the research and co-ordinated pharmacists involved. AM did the statistical analyzes. RZ co-ordinated the publication process OS, RZ, and BH contributed to the discussion and reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Balázs Hankó .

Ethics declarations

Consent for publication.

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Szilvay, A., Somogyi, O., Meskó, A. et al. Qualitative and quantitative research of medication review and drug-related problems in Hungarian community pharmacies: a pilot study. BMC Health Serv Res 19 , 282 (2019). https://doi.org/10.1186/s12913-019-4114-1

Download citation

Received : 23 November 2018

Accepted : 22 April 2019

Published : 03 May 2019

DOI : https://doi.org/10.1186/s12913-019-4114-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Pharmaceutical care
• Community pharmacy
• Vitamin K antagonist
• ACE inhibitor

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Methodology
• Research Methodology
• Quantitative Research Methods

Quantitative Research Methods in Medical Education

• Anesthesiology Publish Ahead of Print(&NA;):&NA;
• Publish Ahead of Print(&NA;):&NA;

• Mayo Foundation for Medical Education and Research

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Yingying Yu

• Saulesh Aituganova
• Aslan Alimbayev
• Aiymgul Seiputanova
• Simulat Healthc J Soc Med Simulat
• Franziska Pollok

• Michael D. Traynor
• Mariela Rivera
• Dr. Chandra Sekhar Espenti

• Alfredo Rafael Anaya Narváez

• César-Augusto Padilla-Avalos
• Roberto Hernández Sampieri
• Bibaisha Nurdauletova
• Sholpan Saparbaikyzy
• Bekbolat Tolegenuly

• Eugenia Bonelli
• ANESTHESIOLOGY
• Richard H. Blum
• Sharon L Muret-Wagstaff

• Katherine L Zaleski

• Cynthia A. Lien
• David O. Warner

• David A Cook
• S R Cummings
• Warren S. Browner

• D.L. Kirkpatrick
• Darcy A Reed
• Randolph H Steadman

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

Quantitative Analysis of the Effectiveness of Public Health Measures on COVID-19 Transmission

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• For correspondence: [email protected]
• Info/History
• Preview PDF

Although COVID-19 has spread almost all over the world, social isolation is still a controversial public health policy and governments of many countries still doubt its level of effectiveness. This situation can create deadlocks in places where there is a discrepancy among municipal, state and federal policies. The exponential increase of the number of infectious people and deaths in the last days shows that the COVID-19 epidemics is still at its early stage in Brazil and such political disarray can lead to very serious results. In this work, we study the COVID-19 epidemics in Brazilian cities using early-time approximations of the SIR model in networks. Different from other works, the underlying network is constructed by feeding real-world data on local COVID-19 cases reported by Brazilian cities to a regularized vector autoregressive model, which estimates directional COVID-19 transmission channels (links) of every pair of cities (vertices) using spectral network analysis. Our results reveal that social isolation and, especially, the use of masks can effectively reduce the transmission rate of COVID-19 in Brazil. We also build counterfactual scenarios to measure the human impact of these public health measures in terms of reducing the number of COVID-19 cases at the epidemics peak. We find that the efficiency of social isolation and of using of masks differs significantly across cities. For instance, we find that they would potentially decrease the COVID-19 epidemics peak in São Paulo (SP) and Brasília (DF) by 15% and 25%, respectively. We hope our study can support the design of further public health measures.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This work is supported in part by the Sao Paulo State Research Foundation (FAPESP) under grant numbers 2015/50122-0 and 2013/07375-0, the Brazilian Coordination of Superior Level Staff Improvement (CAPES), the Pro-Rectory of Research (PRP) of University of Sao Paulo under grant number 2018.1.1702.59.8, and the Brazilian National Council for Scientific and Technological Development (CNPq) under grant numbers 303199/2019-9, 308171/2019-5, and 408546/2018-2.

Author Declarations

All relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Data Availability

All data used in this work is public available.

View the discussion thread.

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One

Subject Area

• Infectious Diseases (except HIV/AIDS)
• Addiction Medicine (349)
• Allergy and Immunology (671)
• Anesthesia (181)
• Cardiovascular Medicine (2661)
• Dentistry and Oral Medicine (316)
• Dermatology (224)
• Emergency Medicine (402)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (950)
• Epidemiology (12264)
• Forensic Medicine (10)
• Gastroenterology (765)
• Genetic and Genomic Medicine (4122)
• Geriatric Medicine (387)
• Health Economics (681)
• Health Informatics (2670)
• Health Policy (1008)
• Health Systems and Quality Improvement (992)
• Hematology (364)
• HIV/AIDS (855)
• Infectious Diseases (except HIV/AIDS) (13719)
• Intensive Care and Critical Care Medicine (800)
• Medical Education (399)
• Medical Ethics (109)
• Nephrology (441)
• Neurology (3912)
• Nursing (210)
• Nutrition (582)
• Obstetrics and Gynecology (742)
• Occupational and Environmental Health (697)
• Oncology (2048)
• Ophthalmology (587)
• Orthopedics (242)
• Otolaryngology (306)
• Pain Medicine (250)
• Palliative Medicine (75)
• Pathology (473)
• Pediatrics (1116)
• Pharmacology and Therapeutics (466)
• Primary Care Research (455)
• Psychiatry and Clinical Psychology (3452)
• Public and Global Health (6551)
• Radiology and Imaging (1410)
• Rehabilitation Medicine and Physical Therapy (818)
• Respiratory Medicine (872)
• Rheumatology (412)
• Sexual and Reproductive Health (410)
• Sports Medicine (344)
• Surgery (452)
• Toxicology (54)
• Transplantation (186)
• Urology (167)

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Contribution to the quantitative aspects in medical research

• PMID: 1235444

Both the advantages and dangers of fallacies in the quantitative approach are discussed, having Laplace's "...le bon sens réduit au calcul..." for motto. The general rules are briefly presented. Stressed is the prerequisite of prior thorough qualitative study of the topic at stake. The medical and mathematical ways of thinking are contrasted, stressed is the leading role of the former. The rebound effect of the quantitative study on the qualitative aspects is treated, with special regard to medical terminology. The necessity to distinguish physiological values and population data is emphasized. Some further aspects are briefly mentioned. The illustrative examples are from diverse terrains of Medicine. The leading role of the well-trained and disciplined medical cortex is stressed throughout.

PubMed Disclaimer

Similar articles

• Mathematics and the surgeon. Crank J. Crank J. Ann R Coll Surg Engl. 1976 Jul;58(4):300-8. Ann R Coll Surg Engl. 1976. PMID: 942167 Free PMC article.
• [What is the role of statistics in medical science?--a critical essay]. Hayashi C. Hayashi C. Gan To Kagaku Ryoho. 1992 Feb;19(2):143-59. Gan To Kagaku Ryoho. 1992. PMID: 1736827 Japanese.
• [The analysis of physicians' work: announcing the end of attempts at in vitro fertilization]. Santiago-Delefosse M, Cahen F, Coeffin-Driol C. Santiago-Delefosse M, et al. Encephale. 2003 Jul-Aug;29(4 Pt 1):293-305. Encephale. 2003. PMID: 14615699 French.
• The study of client-provider interactions: a review of methodological issues. Simmons R, Elias C. Simmons R, et al. Stud Fam Plann. 1994 Jan-Feb;25(1):1-17. Stud Fam Plann. 1994. PMID: 8209391 Review.
• The concept of normality through history: a didactic review of features related to philosophy, statistics and medicine. Conti AA, Conti A, Gensini GF. Conti AA, et al. Panminerva Med. 2006 Sep;48(3):203-5. Panminerva Med. 2006. PMID: 17122759 Review.
• Search in MeSH
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Korean Med Sci
• v.37(16); 2022 Apr 25

A Practical Guide to Writing Quantitative and Qualitative Research Questions and Hypotheses in Scholarly Articles

Edward barroga.

1 Department of General Education, Graduate School of Nursing Science, St. Luke’s International University, Tokyo, Japan.

Glafera Janet Matanguihan

2 Department of Biological Sciences, Messiah University, Mechanicsburg, PA, USA.

The development of research questions and the subsequent hypotheses are prerequisites to defining the main research purpose and specific objectives of a study. Consequently, these objectives determine the study design and research outcome. The development of research questions is a process based on knowledge of current trends, cutting-edge studies, and technological advances in the research field. Excellent research questions are focused and require a comprehensive literature search and in-depth understanding of the problem being investigated. Initially, research questions may be written as descriptive questions which could be developed into inferential questions. These questions must be specific and concise to provide a clear foundation for developing hypotheses. Hypotheses are more formal predictions about the research outcomes. These specify the possible results that may or may not be expected regarding the relationship between groups. Thus, research questions and hypotheses clarify the main purpose and specific objectives of the study, which in turn dictate the design of the study, its direction, and outcome. Studies developed from good research questions and hypotheses will have trustworthy outcomes with wide-ranging social and health implications.

INTRODUCTION

Scientific research is usually initiated by posing evidenced-based research questions which are then explicitly restated as hypotheses. 1 , 2 The hypotheses provide directions to guide the study, solutions, explanations, and expected results. 3 , 4 Both research questions and hypotheses are essentially formulated based on conventional theories and real-world processes, which allow the inception of novel studies and the ethical testing of ideas. 5 , 6

It is crucial to have knowledge of both quantitative and qualitative research 2 as both types of research involve writing research questions and hypotheses. 7 However, these crucial elements of research are sometimes overlooked; if not overlooked, then framed without the forethought and meticulous attention it needs. Planning and careful consideration are needed when developing quantitative or qualitative research, particularly when conceptualizing research questions and hypotheses. 4

There is a continuing need to support researchers in the creation of innovative research questions and hypotheses, as well as for journal articles that carefully review these elements. 1 When research questions and hypotheses are not carefully thought of, unethical studies and poor outcomes usually ensue. Carefully formulated research questions and hypotheses define well-founded objectives, which in turn determine the appropriate design, course, and outcome of the study. This article then aims to discuss in detail the various aspects of crafting research questions and hypotheses, with the goal of guiding researchers as they develop their own. Examples from the authors and peer-reviewed scientific articles in the healthcare field are provided to illustrate key points.

DEFINITIONS AND RELATIONSHIP OF RESEARCH QUESTIONS AND HYPOTHESES

A research question is what a study aims to answer after data analysis and interpretation. The answer is written in length in the discussion section of the paper. Thus, the research question gives a preview of the different parts and variables of the study meant to address the problem posed in the research question. 1 An excellent research question clarifies the research writing while facilitating understanding of the research topic, objective, scope, and limitations of the study. 5

On the other hand, a research hypothesis is an educated statement of an expected outcome. This statement is based on background research and current knowledge. 8 , 9 The research hypothesis makes a specific prediction about a new phenomenon 10 or a formal statement on the expected relationship between an independent variable and a dependent variable. 3 , 11 It provides a tentative answer to the research question to be tested or explored. 4

Hypotheses employ reasoning to predict a theory-based outcome. 10 These can also be developed from theories by focusing on components of theories that have not yet been observed. 10 The validity of hypotheses is often based on the testability of the prediction made in a reproducible experiment. 8

Conversely, hypotheses can also be rephrased as research questions. Several hypotheses based on existing theories and knowledge may be needed to answer a research question. Developing ethical research questions and hypotheses creates a research design that has logical relationships among variables. These relationships serve as a solid foundation for the conduct of the study. 4 , 11 Haphazardly constructed research questions can result in poorly formulated hypotheses and improper study designs, leading to unreliable results. Thus, the formulations of relevant research questions and verifiable hypotheses are crucial when beginning research. 12

CHARACTERISTICS OF GOOD RESEARCH QUESTIONS AND HYPOTHESES

Excellent research questions are specific and focused. These integrate collective data and observations to confirm or refute the subsequent hypotheses. Well-constructed hypotheses are based on previous reports and verify the research context. These are realistic, in-depth, sufficiently complex, and reproducible. More importantly, these hypotheses can be addressed and tested. 13

There are several characteristics of well-developed hypotheses. Good hypotheses are 1) empirically testable 7 , 10 , 11 , 13 ; 2) backed by preliminary evidence 9 ; 3) testable by ethical research 7 , 9 ; 4) based on original ideas 9 ; 5) have evidenced-based logical reasoning 10 ; and 6) can be predicted. 11 Good hypotheses can infer ethical and positive implications, indicating the presence of a relationship or effect relevant to the research theme. 7 , 11 These are initially developed from a general theory and branch into specific hypotheses by deductive reasoning. In the absence of a theory to base the hypotheses, inductive reasoning based on specific observations or findings form more general hypotheses. 10

TYPES OF RESEARCH QUESTIONS AND HYPOTHESES

Research questions and hypotheses are developed according to the type of research, which can be broadly classified into quantitative and qualitative research. We provide a summary of the types of research questions and hypotheses under quantitative and qualitative research categories in Table 1 .

Research questions in quantitative research

In quantitative research, research questions inquire about the relationships among variables being investigated and are usually framed at the start of the study. These are precise and typically linked to the subject population, dependent and independent variables, and research design. 1 Research questions may also attempt to describe the behavior of a population in relation to one or more variables, or describe the characteristics of variables to be measured ( descriptive research questions ). 1 , 5 , 14 These questions may also aim to discover differences between groups within the context of an outcome variable ( comparative research questions ), 1 , 5 , 14 or elucidate trends and interactions among variables ( relationship research questions ). 1 , 5 We provide examples of descriptive, comparative, and relationship research questions in quantitative research in Table 2 .

Hypotheses in quantitative research

In quantitative research, hypotheses predict the expected relationships among variables. 15 Relationships among variables that can be predicted include 1) between a single dependent variable and a single independent variable ( simple hypothesis ) or 2) between two or more independent and dependent variables ( complex hypothesis ). 4 , 11 Hypotheses may also specify the expected direction to be followed and imply an intellectual commitment to a particular outcome ( directional hypothesis ) 4 . On the other hand, hypotheses may not predict the exact direction and are used in the absence of a theory, or when findings contradict previous studies ( non-directional hypothesis ). 4 In addition, hypotheses can 1) define interdependency between variables ( associative hypothesis ), 4 2) propose an effect on the dependent variable from manipulation of the independent variable ( causal hypothesis ), 4 3) state a negative relationship between two variables ( null hypothesis ), 4 , 11 , 15 4) replace the working hypothesis if rejected ( alternative hypothesis ), 15 explain the relationship of phenomena to possibly generate a theory ( working hypothesis ), 11 5) involve quantifiable variables that can be tested statistically ( statistical hypothesis ), 11 6) or express a relationship whose interlinks can be verified logically ( logical hypothesis ). 11 We provide examples of simple, complex, directional, non-directional, associative, causal, null, alternative, working, statistical, and logical hypotheses in quantitative research, as well as the definition of quantitative hypothesis-testing research in Table 3 .

Research questions in qualitative research

Unlike research questions in quantitative research, research questions in qualitative research are usually continuously reviewed and reformulated. The central question and associated subquestions are stated more than the hypotheses. 15 The central question broadly explores a complex set of factors surrounding the central phenomenon, aiming to present the varied perspectives of participants. 15

There are varied goals for which qualitative research questions are developed. These questions can function in several ways, such as to 1) identify and describe existing conditions ( contextual research question s); 2) describe a phenomenon ( descriptive research questions ); 3) assess the effectiveness of existing methods, protocols, theories, or procedures ( evaluation research questions ); 4) examine a phenomenon or analyze the reasons or relationships between subjects or phenomena ( explanatory research questions ); or 5) focus on unknown aspects of a particular topic ( exploratory research questions ). 5 In addition, some qualitative research questions provide new ideas for the development of theories and actions ( generative research questions ) or advance specific ideologies of a position ( ideological research questions ). 1 Other qualitative research questions may build on a body of existing literature and become working guidelines ( ethnographic research questions ). Research questions may also be broadly stated without specific reference to the existing literature or a typology of questions ( phenomenological research questions ), may be directed towards generating a theory of some process ( grounded theory questions ), or may address a description of the case and the emerging themes ( qualitative case study questions ). 15 We provide examples of contextual, descriptive, evaluation, explanatory, exploratory, generative, ideological, ethnographic, phenomenological, grounded theory, and qualitative case study research questions in qualitative research in Table 4 , and the definition of qualitative hypothesis-generating research in Table 5 .

Qualitative studies usually pose at least one central research question and several subquestions starting with How or What . These research questions use exploratory verbs such as explore or describe . These also focus on one central phenomenon of interest, and may mention the participants and research site. 15

Hypotheses in qualitative research

Hypotheses in qualitative research are stated in the form of a clear statement concerning the problem to be investigated. Unlike in quantitative research where hypotheses are usually developed to be tested, qualitative research can lead to both hypothesis-testing and hypothesis-generating outcomes. 2 When studies require both quantitative and qualitative research questions, this suggests an integrative process between both research methods wherein a single mixed-methods research question can be developed. 1

FRAMEWORKS FOR DEVELOPING RESEARCH QUESTIONS AND HYPOTHESES

Research questions followed by hypotheses should be developed before the start of the study. 1 , 12 , 14 It is crucial to develop feasible research questions on a topic that is interesting to both the researcher and the scientific community. This can be achieved by a meticulous review of previous and current studies to establish a novel topic. Specific areas are subsequently focused on to generate ethical research questions. The relevance of the research questions is evaluated in terms of clarity of the resulting data, specificity of the methodology, objectivity of the outcome, depth of the research, and impact of the study. 1 , 5 These aspects constitute the FINER criteria (i.e., Feasible, Interesting, Novel, Ethical, and Relevant). 1 Clarity and effectiveness are achieved if research questions meet the FINER criteria. In addition to the FINER criteria, Ratan et al. described focus, complexity, novelty, feasibility, and measurability for evaluating the effectiveness of research questions. 14

The PICOT and PEO frameworks are also used when developing research questions. 1 The following elements are addressed in these frameworks, PICOT: P-population/patients/problem, I-intervention or indicator being studied, C-comparison group, O-outcome of interest, and T-timeframe of the study; PEO: P-population being studied, E-exposure to preexisting conditions, and O-outcome of interest. 1 Research questions are also considered good if these meet the “FINERMAPS” framework: Feasible, Interesting, Novel, Ethical, Relevant, Manageable, Appropriate, Potential value/publishable, and Systematic. 14

As we indicated earlier, research questions and hypotheses that are not carefully formulated result in unethical studies or poor outcomes. To illustrate this, we provide some examples of ambiguous research question and hypotheses that result in unclear and weak research objectives in quantitative research ( Table 6 ) 16 and qualitative research ( Table 7 ) 17 , and how to transform these ambiguous research question(s) and hypothesis(es) into clear and good statements.

a These statements were composed for comparison and illustrative purposes only.

b These statements are direct quotes from Higashihara and Horiuchi. 16

a This statement is a direct quote from Shimoda et al. 17

The other statements were composed for comparison and illustrative purposes only.

CONSTRUCTING RESEARCH QUESTIONS AND HYPOTHESES

To construct effective research questions and hypotheses, it is very important to 1) clarify the background and 2) identify the research problem at the outset of the research, within a specific timeframe. 9 Then, 3) review or conduct preliminary research to collect all available knowledge about the possible research questions by studying theories and previous studies. 18 Afterwards, 4) construct research questions to investigate the research problem. Identify variables to be accessed from the research questions 4 and make operational definitions of constructs from the research problem and questions. Thereafter, 5) construct specific deductive or inductive predictions in the form of hypotheses. 4 Finally, 6) state the study aims . This general flow for constructing effective research questions and hypotheses prior to conducting research is shown in Fig. 1 .

Research questions are used more frequently in qualitative research than objectives or hypotheses. 3 These questions seek to discover, understand, explore or describe experiences by asking “What” or “How.” The questions are open-ended to elicit a description rather than to relate variables or compare groups. The questions are continually reviewed, reformulated, and changed during the qualitative study. 3 Research questions are also used more frequently in survey projects than hypotheses in experiments in quantitative research to compare variables and their relationships.

Hypotheses are constructed based on the variables identified and as an if-then statement, following the template, ‘If a specific action is taken, then a certain outcome is expected.’ At this stage, some ideas regarding expectations from the research to be conducted must be drawn. 18 Then, the variables to be manipulated (independent) and influenced (dependent) are defined. 4 Thereafter, the hypothesis is stated and refined, and reproducible data tailored to the hypothesis are identified, collected, and analyzed. 4 The hypotheses must be testable and specific, 18 and should describe the variables and their relationships, the specific group being studied, and the predicted research outcome. 18 Hypotheses construction involves a testable proposition to be deduced from theory, and independent and dependent variables to be separated and measured separately. 3 Therefore, good hypotheses must be based on good research questions constructed at the start of a study or trial. 12

In summary, research questions are constructed after establishing the background of the study. Hypotheses are then developed based on the research questions. Thus, it is crucial to have excellent research questions to generate superior hypotheses. In turn, these would determine the research objectives and the design of the study, and ultimately, the outcome of the research. 12 Algorithms for building research questions and hypotheses are shown in Fig. 2 for quantitative research and in Fig. 3 for qualitative research.

EXAMPLES OF RESEARCH QUESTIONS FROM PUBLISHED ARTICLES

• EXAMPLE 1. Descriptive research question (quantitative research)
• - Presents research variables to be assessed (distinct phenotypes and subphenotypes)
• “BACKGROUND: Since COVID-19 was identified, its clinical and biological heterogeneity has been recognized. Identifying COVID-19 phenotypes might help guide basic, clinical, and translational research efforts.
• RESEARCH QUESTION: Does the clinical spectrum of patients with COVID-19 contain distinct phenotypes and subphenotypes? ” 19
• EXAMPLE 2. Relationship research question (quantitative research)
• - Shows interactions between dependent variable (static postural control) and independent variable (peripheral visual field loss)
• “Background: Integration of visual, vestibular, and proprioceptive sensations contributes to postural control. People with peripheral visual field loss have serious postural instability. However, the directional specificity of postural stability and sensory reweighting caused by gradual peripheral visual field loss remain unclear.
• Research question: What are the effects of peripheral visual field loss on static postural control ?” 20
• EXAMPLE 3. Comparative research question (quantitative research)
• - Clarifies the difference among groups with an outcome variable (patients enrolled in COMPERA with moderate PH or severe PH in COPD) and another group without the outcome variable (patients with idiopathic pulmonary arterial hypertension (IPAH))
• “BACKGROUND: Pulmonary hypertension (PH) in COPD is a poorly investigated clinical condition.
• RESEARCH QUESTION: Which factors determine the outcome of PH in COPD?
• STUDY DESIGN AND METHODS: We analyzed the characteristics and outcome of patients enrolled in the Comparative, Prospective Registry of Newly Initiated Therapies for Pulmonary Hypertension (COMPERA) with moderate or severe PH in COPD as defined during the 6th PH World Symposium who received medical therapy for PH and compared them with patients with idiopathic pulmonary arterial hypertension (IPAH) .” 21
• EXAMPLE 4. Exploratory research question (qualitative research)
• - Explores areas that have not been fully investigated (perspectives of families and children who receive care in clinic-based child obesity treatment) to have a deeper understanding of the research problem
• “Problem: Interventions for children with obesity lead to only modest improvements in BMI and long-term outcomes, and data are limited on the perspectives of families of children with obesity in clinic-based treatment. This scoping review seeks to answer the question: What is known about the perspectives of families and children who receive care in clinic-based child obesity treatment? This review aims to explore the scope of perspectives reported by families of children with obesity who have received individualized outpatient clinic-based obesity treatment.” 22
• EXAMPLE 5. Relationship research question (quantitative research)
• - Defines interactions between dependent variable (use of ankle strategies) and independent variable (changes in muscle tone)
• “Background: To maintain an upright standing posture against external disturbances, the human body mainly employs two types of postural control strategies: “ankle strategy” and “hip strategy.” While it has been reported that the magnitude of the disturbance alters the use of postural control strategies, it has not been elucidated how the level of muscle tone, one of the crucial parameters of bodily function, determines the use of each strategy. We have previously confirmed using forward dynamics simulations of human musculoskeletal models that an increased muscle tone promotes the use of ankle strategies. The objective of the present study was to experimentally evaluate a hypothesis: an increased muscle tone promotes the use of ankle strategies. Research question: Do changes in the muscle tone affect the use of ankle strategies ?” 23

EXAMPLES OF HYPOTHESES IN PUBLISHED ARTICLES

• EXAMPLE 1. Working hypothesis (quantitative research)
• - A hypothesis that is initially accepted for further research to produce a feasible theory
• “As fever may have benefit in shortening the duration of viral illness, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response when taken during the early stages of COVID-19 illness .” 24
• “In conclusion, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response . The difference in perceived safety of these agents in COVID-19 illness could be related to the more potent efficacy to reduce fever with ibuprofen compared to acetaminophen. Compelling data on the benefit of fever warrant further research and review to determine when to treat or withhold ibuprofen for early stage fever for COVID-19 and other related viral illnesses .” 24
• EXAMPLE 2. Exploratory hypothesis (qualitative research)
• - Explores particular areas deeper to clarify subjective experience and develop a formal hypothesis potentially testable in a future quantitative approach
• “We hypothesized that when thinking about a past experience of help-seeking, a self distancing prompt would cause increased help-seeking intentions and more favorable help-seeking outcome expectations .” 25
• “Conclusion
• Although a priori hypotheses were not supported, further research is warranted as results indicate the potential for using self-distancing approaches to increasing help-seeking among some people with depressive symptomatology.” 25
• EXAMPLE 3. Hypothesis-generating research to establish a framework for hypothesis testing (qualitative research)
• “We hypothesize that compassionate care is beneficial for patients (better outcomes), healthcare systems and payers (lower costs), and healthcare providers (lower burnout). ” 26
• Compassionomics is the branch of knowledge and scientific study of the effects of compassionate healthcare. Our main hypotheses are that compassionate healthcare is beneficial for (1) patients, by improving clinical outcomes, (2) healthcare systems and payers, by supporting financial sustainability, and (3) HCPs, by lowering burnout and promoting resilience and well-being. The purpose of this paper is to establish a scientific framework for testing the hypotheses above . If these hypotheses are confirmed through rigorous research, compassionomics will belong in the science of evidence-based medicine, with major implications for all healthcare domains.” 26
• EXAMPLE 4. Statistical hypothesis (quantitative research)
• - An assumption is made about the relationship among several population characteristics ( gender differences in sociodemographic and clinical characteristics of adults with ADHD ). Validity is tested by statistical experiment or analysis ( chi-square test, Students t-test, and logistic regression analysis)
• “Our research investigated gender differences in sociodemographic and clinical characteristics of adults with ADHD in a Japanese clinical sample. Due to unique Japanese cultural ideals and expectations of women's behavior that are in opposition to ADHD symptoms, we hypothesized that women with ADHD experience more difficulties and present more dysfunctions than men . We tested the following hypotheses: first, women with ADHD have more comorbidities than men with ADHD; second, women with ADHD experience more social hardships than men, such as having less full-time employment and being more likely to be divorced.” 27
• “Statistical Analysis
• ( text omitted ) Between-gender comparisons were made using the chi-squared test for categorical variables and Students t-test for continuous variables…( text omitted ). A logistic regression analysis was performed for employment status, marital status, and comorbidity to evaluate the independent effects of gender on these dependent variables.” 27

EXAMPLES OF HYPOTHESIS AS WRITTEN IN PUBLISHED ARTICLES IN RELATION TO OTHER PARTS

• EXAMPLE 1. Background, hypotheses, and aims are provided
• “Pregnant women need skilled care during pregnancy and childbirth, but that skilled care is often delayed in some countries …( text omitted ). The focused antenatal care (FANC) model of WHO recommends that nurses provide information or counseling to all pregnant women …( text omitted ). Job aids are visual support materials that provide the right kind of information using graphics and words in a simple and yet effective manner. When nurses are not highly trained or have many work details to attend to, these job aids can serve as a content reminder for the nurses and can be used for educating their patients (Jennings, Yebadokpo, Affo, & Agbogbe, 2010) ( text omitted ). Importantly, additional evidence is needed to confirm how job aids can further improve the quality of ANC counseling by health workers in maternal care …( text omitted )” 28
• “ This has led us to hypothesize that the quality of ANC counseling would be better if supported by job aids. Consequently, a better quality of ANC counseling is expected to produce higher levels of awareness concerning the danger signs of pregnancy and a more favorable impression of the caring behavior of nurses .” 28
• “This study aimed to examine the differences in the responses of pregnant women to a job aid-supported intervention during ANC visit in terms of 1) their understanding of the danger signs of pregnancy and 2) their impression of the caring behaviors of nurses to pregnant women in rural Tanzania.” 28
• EXAMPLE 2. Background, hypotheses, and aims are provided
• “We conducted a two-arm randomized controlled trial (RCT) to evaluate and compare changes in salivary cortisol and oxytocin levels of first-time pregnant women between experimental and control groups. The women in the experimental group touched and held an infant for 30 min (experimental intervention protocol), whereas those in the control group watched a DVD movie of an infant (control intervention protocol). The primary outcome was salivary cortisol level and the secondary outcome was salivary oxytocin level.” 29
• “ We hypothesize that at 30 min after touching and holding an infant, the salivary cortisol level will significantly decrease and the salivary oxytocin level will increase in the experimental group compared with the control group .” 29
• EXAMPLE 3. Background, aim, and hypothesis are provided
• “In countries where the maternal mortality ratio remains high, antenatal education to increase Birth Preparedness and Complication Readiness (BPCR) is considered one of the top priorities [1]. BPCR includes birth plans during the antenatal period, such as the birthplace, birth attendant, transportation, health facility for complications, expenses, and birth materials, as well as family coordination to achieve such birth plans. In Tanzania, although increasing, only about half of all pregnant women attend an antenatal clinic more than four times [4]. Moreover, the information provided during antenatal care (ANC) is insufficient. In the resource-poor settings, antenatal group education is a potential approach because of the limited time for individual counseling at antenatal clinics.” 30
• “This study aimed to evaluate an antenatal group education program among pregnant women and their families with respect to birth-preparedness and maternal and infant outcomes in rural villages of Tanzania.” 30
• “ The study hypothesis was if Tanzanian pregnant women and their families received a family-oriented antenatal group education, they would (1) have a higher level of BPCR, (2) attend antenatal clinic four or more times, (3) give birth in a health facility, (4) have less complications of women at birth, and (5) have less complications and deaths of infants than those who did not receive the education .” 30

Research questions and hypotheses are crucial components to any type of research, whether quantitative or qualitative. These questions should be developed at the very beginning of the study. Excellent research questions lead to superior hypotheses, which, like a compass, set the direction of research, and can often determine the successful conduct of the study. Many research studies have floundered because the development of research questions and subsequent hypotheses was not given the thought and meticulous attention needed. The development of research questions and hypotheses is an iterative process based on extensive knowledge of the literature and insightful grasp of the knowledge gap. Focused, concise, and specific research questions provide a strong foundation for constructing hypotheses which serve as formal predictions about the research outcomes. Research questions and hypotheses are crucial elements of research that should not be overlooked. They should be carefully thought of and constructed when planning research. This avoids unethical studies and poor outcomes by defining well-founded objectives that determine the design, course, and outcome of the study.

Disclosure: The authors have no potential conflicts of interest to disclose.

Author Contributions:

• Conceptualization: Barroga E, Matanguihan GJ.
• Methodology: Barroga E, Matanguihan GJ.
• Writing - original draft: Barroga E, Matanguihan GJ.
• Writing - review & editing: Barroga E, Matanguihan GJ.

• Open access
• Published: 11 September 2024

Value of CDR1-AS as a predictive and prognostic biomarker for patients with breast cancer receiving neoadjuvant chemotherapy in a prospective Chinese cohort

• Chenwei Yuan 1   na1 ,
• Yaqian Xu 2   na1 ,
• Liheng Zhou   nAff1 ,
• Jing Peng 1 ,
• Rui Sha 1 ,
• Yanping Lin 1 ,
• Shuguang Xu 1 ,
• Yumei Ye 1 ,
• Fan Yang 1 ,
• Tingting Yan 1 ,
• Xinrui Dong 1 ,
• Yaohui Wang   nAff1 ,
• Wenjin Yin 1 &
• Jinsong Lu   nAff1  

European Journal of Medical Research volume  29 , Article number:  454 ( 2024 ) Cite this article

Metrics details

Neoadjuvant chemotherapy (NAC) is an effective treatment for locally advanced breast cancer (BC). However, there are no effective biomarkers for evaluating its efficacy. CDR1-AS, well known for its important role in tumorigenesis, is a famous circular RNA involved in the chemosensitivity of cancers other than BC. However, the predictive role of CDR1-AS in the efficacy and prognosis of NAC for BC has not been fully elucidated. We herein aimed to clarify this role.

The present study included patients treated with paclitaxel-cisplatin-based NAC. The expression of CDR1-AS was detected by real-time quantitative reverse transcription polymerase chain reaction testing. The predictive value of CDR1-AS expression was examined in pathological complete response (pCR) after NAC using logistic regression analysis. The relationship between CDR1-AS expression and survival was demonstrated using the Kaplan–Meier method, and tested by log-rank test and Cox proportional hazards regression model.

The present study enrolled 106 patients with BC. Multivariate logistic regression analysis revealed that CDR1-AS expression was an independent predictive factor for pCR (odds ratio [OR] = 0.244; 95% confidence interval [CI] 0.081–0.732; p =  0.012). Furthermore, pCR benefits with low CDR1-AS expression were observed across all subgroups. The Kaplan–Meier curves and log-rank test suggested that the CDR1-AS high-expression group showed significantly better disease-free survival (DFS; log-rank p =  0.022) and relapse-free survival (RFS; log-rank p  = 0.012) than the CDR1-AS low-expression group. Multivariate analysis revealed that CDR1-AS expression was an independent prognostic factor for DFS (adjusted HR = 0.177; 95% CI 0.034–0.928, p  = 0.041), RFS (adjusted HR = 0.061; 95% CI 0.006–0.643, p  = 0.020), and distant disease-free survival (adjusted HR = 0.061; 95% CI 0.006–0.972, p  = 0.047).

Conclusions

CDR1-AS may be a potential novel predictive biomarker of pCR and survival benefit in patients with locally advanced BC receiving NAC. This may help identify specific chemosensitive individuals and build personalized treatment strategies.

Introduction

Breast cancer (BC) is a major cause of cancer-related death among women, becoming the most common malignant tumor worldwide [ 1 ]. Locally advanced breast cancer (LABC) is a huge clinical challenge because most patients with LABC experience a high recurrence rate and shorter survival compared to those with not advanced tumors. Neoadjuvant therapy is a commonly used and effective treatment because it can render inoperable tumors resectable and improve breast conservation rates [ 2 ]. Moreover, neoadjuvant chemotherapy (NAC) can identify individuals at high risk of recurrence who may have residual tumors for subsequent intensive adjuvant therapy, especially for triple-negative and human epidermal growth factor receptor 2 (HER2)-positive BC. Several large clinical trials have confirmed that patients with BC who achieve pathological complete response (pCR) after NAC have a significantly better prognosis than those who do not [ 3 , 4 , 5 ]. However, there are currently no effective biomarkers that can accurately predict the pCR of BC after NAC. Thus, there is an urgent need to discover effective biomarkers to determine the efficacy of NAC for BC.

Circular RNA (circRNA) is a newly discovered type of non-coding RNA with a covalent closed loop structure without a 5′-cap structure or 3′-polyadenylation tail in recent years. CircRNAs can regulate gene expression at epigenetic, transcriptional, and post-transcriptional levels and affect tumorigenesis through diverse mechanisms of action and functional roles, including microRNA sponging, protein interaction, translation, and so on [ 6 ]. Given their high stability and detectability in human tissues and biofluids, circRNAs have been recognized as promising diagnostic and prognostic biomarkers. CDR1-AS (hsa_circ_0001946), a naturally occurring RNA transcribed in antisense to CDR1, which is located on the human chromosome Xq27.1 region, plays various roles through different functions in tumors [ 7 , 8 , 9 ]. Piwecka conducted an analysis of CDR1-AS expression across various mouse tissues, revealing that it is most prominently expressed in neural tissues, abundant expressed in spinal cord, low expressed in lung tissue, skeletal muscle and heart, and almost undetectable in spleen, suggesting a distinct pattern of tissue-specific expression for CDR1-AS [ 10 ]. Growing evidence indicates that circRNAs are highly relevant to drug resistance and metabolism. Yang et al. found that CDR1-AS increased the resistance of BC cells to 5-FU chemotherapy. The main mechanism is that inhibiting the expression of CDR1-AS can upregulate the expression of miR-7, thus suppressing the expression of CCNE1 and finally improving the chemotherapy sensitivity of 5-FU resistant BC cells [ 11 ]. Another study by Yang et al. revealed that CDR1-AS increased the resistance of BC cells to cisplatin chemotherapy. The main mechanism is that the overexpression of CDR1-AS acts as a molecular sponge to adsorb miR-7, thus upregulating REGγ, which is related to poor prognosis of BC and further leads to the resistance of BC cells to cisplatin [ 12 ]. To the best of our knowledge, circRNAs such as CDR1-AS have not been put into practice as biomarkers in clinical decision-making, and proper validation studies involving prospectively collected samples and clinical trials are lacking. The role of CDR1-AS in predicting the efficacy of NAC in patients with BC remains unknown.

Accordingly, this study aimed to evaluate the predictive role of CDR1-AS in the efficacy and prognosis of NAC for BC. We hypothesized that the expression of CDR1-AS might help predict neoadjuvant chemosensitivity in patients with BC, which was illustrated in this retrospective study on our prospective clinical trials.

Materials and methods

Patients and study design.

The patients enrolled in the two registered prospective neoadjuvant clinical trials SHPD001 (NCT02199418) and SHPD002 (NCT02221999) were analyzed in this study. The research protocols of both clinical trials were approved by the Independent Ethics Committee of Renji Hospital, School of Medicine, Shanghai Jiaotong University. Written informed consent was obtained from all patients. In total, 106 patients (SHPD001 trial, N  = 3; SHPD002 trial, N  = 103) with adequate and qualified tissue samples for the detection of CDR1-AS expression from the aforementioned trials were enrolled in this study.

The treatment protocols have been reported previously [ 13 ]. Patients were all qualified with the following inclusion criteria: women aged 18–70 years old, histologically confirmed invasive BC with a tumor size ≥ 2 cm, no prior systemic or loco-regional treatment administered to patients, and sufficient and eligible tissue samples for CDR1-AS expression detection. The NAC regimens were as follows: patients received four cycles of treatment for 28 days per cycle. For every cycle, paclitaxel (80 mg/m 2 ) was administered weekly on days 1, 8, 15, and 22, and cisplatin (25 mg/m 2 ) was administered on days 1, 8, and 15. HER2-positive patients concomitantly received trastuzumab. For patients with ER-or PR-positive BC in SHPD002, endocrine therapy with letrozole in postmenopausal women and ovarian function suppression in premenopausal women were randomized together with chemotherapy, according to their menopausal status. Premenopausal patients with triple-negative BC were randomized to chemotherapy with or without ovarian function suppression in SHPD002. The patients underwent surgery sequentially after completing neoadjuvant therapy.

Tissue samples and clinical data collection

Clinical data was collected prospectively when patients were enrolled in the clinical trial (Table  1 ). Fresh primary cancer tissue specimens were collected from the Department of Breast Surgery, Renji Hospital, School of Medicine, Shanghai Jiaotong University. The tissues were collected by core-needle biopsy before the patients underwent neoadjuvant treatment, frozen immediately in liquid nitrogen, and stored at − 80 °C until RNA extraction. Positive status of estrogen receptor (ER) and progesterone receptor (PR) was defined as ≥ 1% of tumor cells showing positive nuclear staining by immunohistochemistry (IHC). HER2 positivity was defined as IHC 3 + or fluorescence in situ hybridization positivity [ 14 ].

RNA extraction and real-time quantitative reverse transcription polymerase chain reaction (qRT-PCR) testing

Total RNA from BC tissues was extracted with a TRIzol reagent (Invitrogen, NY, USA) and subsequently reverse-transcribed into cDNA using PrimeScript ™ RT Master Mix (Perfect Real Time; Takara, Shiga, Japan) according to the manufacturer’s instructions. qRT-PCR testing was performed using SYBR ® Premix Ex TaqTM (Tli RNaseH Plus; Takara) in a LightCycler ® 480 System (Roche, Basel, Switzerland) according to the manufacturer’s protocol. The specific primers used were as follows: CDR1-AS sense 5′-ACGTCTCCAGTGTGCTGA-3′ and antisense 5′-CTTGACACAGGTGCCATC-3′,  and β-actin sense 5′-CATGTACGTTGCTATCCAGGC-3′ and antisense 5′-CTCCTTAATGTCACGCACGAT-3′. β-Actin was used as an internal control, and gene expression levels were normalized to β-actin using the 2 −ΔCt method [ 15 ]. Each reaction was performed in triplicates.

In silico analysis

The pan-cancer analysis of CDR1-AS expression was shown based on the MiOncoCirc database ( N  = 744; http://mioncocirc.github.io/ ; access on June 30, 2022) [ 16 ]. The comparison of CDR1-AS expression was performed between BC tissues ( N  = 1085) and normal breast tissues ( N  = 112) and tested by student’s t -test within the GEPIA database ( http://gepia.cancer-pku.cn/ ; access on June 30, 2022) [ 17 ].

Statistical analyses

Patients were divided into the high- and low-expression cohorts according to the median expression of CDR1-AS. The relationship between all baseline clinicopathological characteristics and CDR1-AS expression levels was calculated using the chi-squared test. Univariate and multivariate binary logistic regression analyses were utilized to evaluate the association between CDR1-AS expression or clinical characteristics and pCR and investigate the potential interactions between CDR1-AS expression and clinical characteristics of pCR. A nomogram was constructed to predict pCR by combining CDR1-AS with clinical attributes, including clinical T stage, ER status, and HER2 status. A calibration curve was used to evaluate nomogram calibration. Receiver operating characteristic (ROC) curves and decision curve analysis (DCA) were performed to examine whether CDR1-AS could improve the ability to predict a patient’s response to NAC. The Kaplan–Meier method was used to calculate the survival rates, and survival curves were compared using the log-rank test. Univariate and multivariate Cox proportional hazards models were used to investigate independent risk factors for disease-free survival (DFS), relapse-free survival (RFS), and distant disease-free survival (DDFS). DFS was defined as the time from surgery to the first recurrence (local, regional, distant), contralateral BC, second primary non-breast cancer, or death from any cause. RFS was defined as the time from surgery to local, regional, or distant relapse or death. DDFS was defined as the time from surgery to distant recurrence or death. All statistical analyses were performed using Stata version 14.1 (Stata Corp LLC, Texas, USA) and R software version 3.6.1 ( www.r-project.org ). A p -value < 0.05 was considered statistically significant.

CDR1-AS expression and baseline characteristics

The relative CDR1-AS expression in pan-cancer tissues was analyzed based on the MiOncoCirc database [ 16 ] and was found abundant in BRCA group (breast invasive carcinoma) (Fig.  1 A). The relative expression of CDR1-AS in BC was higher than other common cancers, such as lung cancer and hepatocellular carcinoma, which were leading fatal cancer types in China. The expression of CDR1-AS was explored in the GEPIA database [ 17 ] and was found to be significantly downregulated in BC tissues compared to that in normal breast tissues ( p  < 0.01) (Fig.  1 B). A total of 106 patients were included in this study and divided into two groups based on the median expression of CDR1-AS. Thus, both the high and low CDR1-AS expression groups included 53 patients. No significant differences were found in clinico-histopathological characteristics, including age, body mass index (BMI), ER status, PR status, and other factors, between the high and low CDR1-AS expression group (Table  1 ). Thirty-seven patients achieved pCR, and the total pCR rate was 34.91%. Thirteen events occurred in this cohort: one patient died, ten relapsed or progressed, and two had secondary primary cancer.

CDR1-AS expression in cancer tissues. A Relative CDR1-AS expression in pan-cancer tissues based on the MiOncoCirc database. B Relative CDR1-AS expression in BC and normal tissues based on the GEPIA database. * p  < 0.01 (student’s t-test). ACC adrenocortical carcinoma, BLCA bladder urothelial carcinoma, BRCA breast invasive carcinoma, CHOL cholangiocarcinoma, ESCA esophageal carcinoma, GBM Glioblastoma multiforme, HCC hepatocellular carcinoma, HNSC head and neck squamous cell carcinoma, KDNY kidney cancer, LUNG lung cancer, MBL medulloblastoma, NRBL neuroblastoma, OV ovarian serous cystadenocarcinoma, PAAD pancreatic adenocarcinoma, PRAD prostate adenocarcinoma, SARC sarcoma, SECR secretory cancer, and SKCM skin cutaneous melanoma

CDR1-AS expression and pCR outcomes

The median CDR1-AS expression was 0.099 (range, 0.017 to 1.149) and 0.091 (range, 0.002 to 0.614) in the pCR and non-pCR group, respectively (Fig.  2 A). Patients with low CDR1-AS expression achieved a higher pCR rate (41.51%) than those with high CDR1-AS expression (28.30%; Fig.  2 B), although the difference was not significant (odds ratio [OR] = 0.556; 95% confidence interval [CI] 0.247–1.250, p =  0.156; Table  2 ). In univariate logistic regression analysis, negative ER status (OR = 0.182; 95% CI 0.077–0.434, p  < 0.001) and positive HER2 status (OR = 3.286; 95% CI 1.430–7.549, p  < 0.001) favored pCR. Moreover, negative PR status (OR = 0.480; 95% CI 0.209–1.101, p =  0.083) and low BMI status (OR = 0.462; 95% CI 0.197–1.078, p =  0.074) tended to favor pCR. Age (OR = 1.191; 95% CI 0.525–2.700, p =  0.675), clinical tumor stage (OR = 0.486; 95% CI 0.193–1.223, p =  0.125) and ki67 status (OR = 1.506; 95% CI 0.660–3.437, p =  0.331) were not significantly associated with pCR (Table  2 ).

Features of patients with pathological complete response (pCR) and non-pCR. A The relative CDR1-AS expression in the pCR and non-pCR group. B Clinicopathological features of pCR and no-pCR patients. Two-category data (pCR, yes vs. no; CDR1-AS expression, high vs. low; age, ≥ 50 vs. < 50 years, clinical T -stage, T4 vs. T2–3; clinical N stage, N1–3 vs. N0; ER positivity vs. negativity; PR positivity vs. negativity; HER2 positivity vs. negativity; Ki67 > 30% vs. ≤ 30%; and BMI ≥ 24 vs. < 24 kg/m 2 ) are shown in dark and light blue, respectively. pCR pathological complete response, ER estrogen receptor, PR progesterone receptor, HER2 human epidermal growth factor receptor 2, and BMI body mass index

After adjusting for age, clinical tumor stage, ER status, PR status, HER2 status, Ki67 index, and BMI status, multiple logistic regression analysis revealed that low CDR1-AS expression was significantly associated with pCR (OR = 0.244; 95% CI 0.081–0.732, p =  0.012). Meanwhile, patients with lower clinical tumor stage (OR = 0.123; 95% CI 0.029–0.518, p =  0.004), negative ER status (OR = 0.101; 95% CI 0.027–0.375, p =  0.001), and positive HER2 status (OR = 6.668; 95% CI 2.085–21.328, p =  0.001) could achieve pCR more easily (Table  2 ).

Building and assessment of the multivariate model for pCR prediction

According to prior multiple logistic regression analysis, four predictive features, including clinical T stage, ER status, HER2 status, and CDR1-AS, were selected to build a multivariate predictive model. A nomogram was created for predicting pCR (Fig.  3 A). The calibration curves showed a high consistency between the prediction of the nomogram and the actual observed pCR outcomes in our cohort (Fig.  3 B). ROC curves and DCA were used to compare the accuracy of different predictive models with or without CDR1-AS. The area under the curve (AUC) was 0.813 (95% CI 0.727–0.898), achieved by adding CDR1-AS to clinicopathological features, which is better than 0.789 (95% CI 0.700–0.877) for clinicopathological characteristics alone (Fig.  3 C). Moreover, DCA consistently showed more benefits with the model combining CDR1-AS with clinicopathological variables (Fig.  3 D).

Nomogram of the multivariate model for pCR prediction A The nomogram was built using independent predictive factors for pCR. B Calibration curve of nomogram. C Receiver operating characteristic curves of the predictive models with and without CDR1-AS expression (area under the curve, 0.813 vs. 0.789). D Decision curve analysis of the net benefit versus threshold probability. ER estrogen status, HER2 human epidermal growth factor receptor 2, and pCR pathological complete response

Subgroup analysis of pCR rates

Subgroup analysis suggested that pCR outcomes were significantly associated with CDR1-AS in patients aged ≥ 50 years (OR = 0.096; 95% CI 0.013–0.709; p =  0.022) and those with a BMI less than 24 (OR = 0.172; 95% CI 0.038–0.780; p =  0.022), as well as premenopausal (OR = 0.163; 95% CI; 0.027–0.988; p =  0.048), postmenopausal (OR = 0.090; 95% CI 0.010–0.787; p =  0.030), T2-3 (OR = 0.244; 95% CI 0.068–0.880; p =  0.031), stage N1–3 (OR = 0.186; 95% CI 0.055–0.631; p =  0.007), ER-negative (OR = 0.131; 95% CI 0.021–0.805, p =  0.028), PR-negative (OR = 0.072; 95% CI 0.009–0.584; p =  0.014), HER2-positive (OR = 0.156; 95% CI 0.026–0.945; p =  0.043), and ki67 > 30% tumors (OR = 0.133; 95% CI 0.024–0.732, p =  0.020; Fig.  4 ). No interaction was detected between the clinicopathological variables and CDR1-AS for pCR (Fig.  4 ).

Subgroup analysis for pCR according to CDR1-AS expression levels pCR pathological complete response, OR odds ratio, CI confidence interval, ER estrogen receptor, PR progesterone receptor, HER2 human epidermal growth factor receptor 2, and BMI body mass index

CDR1-AS expression and DFS

The median follow-up time for all patients was 30.02 months. Kaplan–Meier curves and log-rank tests were performed to determine DFS according to CDR1-AS expression level. Compared to the CDR1-AS low-expression group, the high-expression group showed significantly better DFS ( N  = 106; log-rank p =  0.022; Fig.  5 A).

Kaplan–Meier plot estimates of survival outcomes according to CDR1-AS expression levels A DFS was estimated using the Kaplan–Meier plot. B RFS estimated using the Kaplan–Meier plot. C DDFS estimated using the Kaplan–Meier plot. DFS disease-free survival, RFS relapse-free survival, DDFS distant disease-free survival, HR hazard ratio, and CI confidence interval

In the univariate analysis, patients with high expression of CDR1-AS had a substantially better DFS than those with a low expression of CDR1-AS (hazard ratio [HR] = 0.202; 95% CI 0.044–0.924; p =  0.039). Simultaneously, multivariate analysis showed that CDR1-AS expression was an independent prognostic factor for DFS (adjusted HR = 0.177; 95% CI 0.034–0.928, p =  0.041). Moreover, T4 clinical tumor stage (adjusted HR = 5.445; 95% CI 1.294–22.907; p =  0.021) and high ki67 index (adjusted HR = 7.576; 95% CI 1.436–39.973; p =  0.017) were significantly associated with worse DFS (Table  3 ).

CDR1-AS expression and RFS

The CDR1-AS high-expression group showed significantly better RFS than the low-expression group ( N  = 106; log-rank p =  0.012; Fig.  5 B). In the univariate analysis, patients with high CDR1-AS expression had substantially better RFS than those with low CDR1-AS expression (HR = 0.112; 95% CI 0.014–0.887; p =  0.038). Multivariate analysis showed that CDR1-AS expression was an independent prognostic factor for RFS (adjusted HR = 0.061; 95% CI 0.006–0.643; p =  0.020). Moreover, T4 clinical tumor stage (adjusted HR = 11.078; 95% CI 2.074–59.164; p =  0.005) and high ki67 index (adjusted HR = 9.880; 95% CI 1.666–58.574; p =  0.012) were significantly associated with worse RFS (Table  4 ).

CDR1-AS expression and DDFS

The CDR1-AS high-expression group was prone to have a better DDFS than the low-expression group ( N =  106; log-rank p =  0.050; Fig.  5 C). In univariate analysis, patients with high expression of CDR1-AS tended to have a better DDFS than patients with low expression of CDR1-AS (HR = 0.158; 95% CI 0.019–1.317; p =  0.088). Multivariate analysis revealed that CDR1-AS expression was an independent prognostic factor for DDFS (adjusted hazard ratio [HR] = 0.061; 95% confidence interval [CI] 0.006–0.972, p =  0.047). Furthermore, T4 clinical tumor stage (adjusted HR = 24.665; 95% CI 2.601–233.992; p =  0.005) and high ki67 index (adjusted HR = 19.134; 95% CI 1.776–206.098; p =  0.015) were significantly associated with worse DDFS (Table  5 ).

To the best of our knowledge, this study is the first to evaluate the value of CDR1-AS expression for predicting efficacy of neoadjuvant therapy in LABC based on data from prospective clinical trials. We unfolded the prognostic value of CDR1-AS for survival outcomes in patients with LABC for the first time.

pCR is still the main indicator for evaluating the efficacy of neoadjuvant chemotherapy (NAC) and predicting prognosis. Various studies have evaluated the prognostic significance of pCR after NAC. A large meta-analysis of 27895 patients found that achieving pCR following NAC was associated with significantly better event-free survival (EFS) and overall survival (OS), particularly for triple-negative and HER2 + BC [ 18 ]. Our multivariate logistic analysis showed that CDR1-AS was an independent predictive factor for pCR in patients with LABC. Patients with low CDR1-AS expression were more likely to achieve pCR. In our clinical study, a neoadjuvant chemotherapy regimen of paclitaxel combined with cisplatin was administered to patients with BC [ 13 ]. Currently, no clinical studies have investigated the role of CDR1-AS in predicting the efficacy of NAC in patients with BC. However, some basic studies have suggested that CDR1-AS is indeed related to the efficacy of chemotherapy in cancer cells, especially those treated with cisplatin. Zhao et al. reported that CDR1-AS is highly expressed in cisplatin-resistant cells and that the CDR1-AS/miR-641/HOXA9 axis promotes NSCLC cisplatin resistance by regulating cancer stem cells [ 19 ]. Meanwhile, a study by Mao et al. documented that the higher expression of CDR1-AS is an independent prognostic factor in lung adenocarcinoma, and is predictive of resistance to pemetrexed and cisplatin since CDR1-AS could induce the EGFR/PI3K signaling pathway [ 20 ]. These studies suggest that CDR1-AS can reduce the sensitivity of tumors to chemotherapy, which may at least partially support our findings.

The results of the subgroup analysis demonstrated that pCR benefits with low CDR1-AS expression were observed across all subgroups, indicating that CDR1-AS was an ideal biomarker and its chemosensitive prediction function was not affected by routine clinicopathological conditions. Particularly, low CDR1-AS expression was related to a higher pCR rate in patients aged ≥ 50 years or with a clinical N1–3 stage, ki67 > 30%, or BMI < 24 sub-populations. The internal mechanism underlying this phenomenon remains unclear and needs to be further elucidated. Future trials should consider CDR1-AS expression as an important factor.

To date, the relationship between CDR1-AS and survival of patients with BC receiving neoadjuvant therapy remains unreported. Our study showed, for the first time, that CDR1-AS expression could be an independent prognostic factor for BC treated with NAC. We found that the expression of CDR1-AS was closely associated with DFS (log-rank p =  0.022) and RFS (log-rank p =  0.012) in patients with BC who had received NAC. After multi-factor adjustment, the DFS, RFS, and DDFS of patients with low CDR1-AS expression were significantly worse than those of patients with high expression ( p =  0.041, p =  0.020, and p =  0.047, respectively). However, the study by K. Uhr et al. suggests that the expression of CDR1-AS is not associated with the metastasis-free survival and overall survival of BC patients [ 21 ]. The survival outcome discrepancy between our study and K. Uhr’s work [ 21 ] may be partially explained by differences in patient selection: 90.6% (96/106) of patients in our study are lymph node-positive, while all patients in K. Uhr’s work are lymph node-negative. Moreover, our patients come from a prospective randomized controlled cohort. The patients in our cohort are LABC patients who had received neoadjuvant therapy. While there are two cohort of patients in K. Uhr’s work, one cohort includes primary BC patients who had not received any systemic (neo)adjuvant therapy, and the other cohort consists of hormone receptor positive metastatic BC patients. The stages and treatments of the patients in the two studies are different. Another reason for the diverse results of the two studies might be the different treatment. Our patient had received systemic (neo)adjuvant therapy while K. Uhr’s cohort had not received any systemic (neo)adjuvant therapy. Furthermore, the patients of both studies were from different regions, which may account for the discrepant result. Despite the shortage of CDR1-AS related clinical data, several basic findings demonstrated that the downregulated expression of CDR1-AS is associated with enhanced malignant biological behavior in tumor cells [ 7 , 8 , 22 ], which is partially consistent with our study. A study by Hanniford et al. showed that the loss of CDR1-AS expression promoted melanoma invasion and metastasis via an IGF2BP3-mediated mechanism [ 8 ]. Another work by Lou et al. revealed that CDR1-AS depletion might play a potent role in promoting tumorigenesis by downregulating p53 expression in patients with glioma [ 7 ]. Moreover, CDR1-AS knockdown facilitated gastric cancer cell migration and invasion in vitro and in vivo by targeting the miR-876-5p/GNG7 Axis [ 22 ]. In summary, our results suggest that CDR1-AS can be used as an effective prognostic marker for survival.

Our study showed that a high pCR rate was achieved in patients with low CDR1-AS expression, and patients with high expression CDR1-AS had better survival. As previously reported, CDR1-AS inhibits tumorigenesis and cancer progression [ 7 , 8 ]. The nature of CDR1-AS is similar to that of the estrogen receptor, of which the positivity is less sensitive to chemotherapy and is related to better prognosis among patients with BC [ 4 ]. Therefore, tumors with higher CDR1-AS expression are well-differentiated, which is related to a lower pCR rate but better prognosis.

The expression pattern of CDR1-AS varies in different tumors. In the case of ovarian and bladder cancers, CDR1-AS expression are found to be reduced, suggesting its role as a suppressor of tumorigenesis [ 23 , 24 ]. Conversely, in colorectal cancer, the levels of CDR1-AS is elevated, and it appears to act as a facilitator of tumor growth [ 25 ]. Our research has also observed that in BC, CDR1-AS expression is lower compared to that in normal tissue. These observations suggest that CDR1-AS may serve dual roles as either a promoter or suppressor of tumor development. Further investigation is warranted to elucidate its complex functions in different cancers.

This study has some limitations. Its sample size was small, and the results require verification in a larger cohort in the future. Our research suggests that patients with low CDR1-AS expression are sensitive to chemotherapy and easily achieve pCR, which implies certain intrinsic regularity and provides guidance for further prospective large-sample clinical studies for validation. The mechanism underlying this phenomenon requires further investigation.

Our research sheds light on the value of CDR1-AS as a potential novel biomarker for predicting pCR and survival benefit in patients with LABC. Our study may help identify specific patient subgroups and guide treatment strategies. However, the role of CDR1-AS in chemoresistance remains unclear.

Availability of data and materials

The datasets reported in the current study are available from the corresponding author upon reasonable request.

Abbreviations

• Neoadjuvant chemotherapy
• Breast cancer

Locally advanced breast cancer

Pathological complete response

Disease-free survival

Relapse-free survival

Distant disease-free survival

• Circular RNA

Event-free survival

Overall survival

Estrogen receptor

Progesterone receptor

Immunohistochemistry

Human epidermal growth factor receptor 2

Receiver operating characteristic

Decision curve analysis

Real-time quantitative reverse transcription polymerase chain reaction

Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;12:e7386–441. https://doi.org/10.3322/caac.21660 .

Article   CAS   Google Scholar  

Tryfonidis K, Senkus E, Cardoso MJ, Cardoso F. Management of locally advanced breast cancer-perspectives and future directions. Nat Rev Clin Oncol. 2015;12:147–62.

Article   PubMed   Google Scholar  

Fisher B, Bryant J, Wolmark N, Mamounas E, Brown A, Fisher ER, et al. Effect of preoperative chemotherapy on the outcome of women with operable breast cancer. J Clin Oncol. 1998;16:2672–85. https://doi.org/10.1200/JCO.1998.16.8.2672 .

Article   CAS   PubMed   Google Scholar  

Bear HD, Anderson S, Brown A, Smith R, Mamounas EP, Fisher B, et al. The effect on tumor response of adding sequential preoperative docetaxel to preoperative doxorubicin and cyclophosphamide: preliminary results from national surgical adjuvant breast and bowel project protocol B-27. J Clin Oncol. 2003;21:4165–74. https://doi.org/10.1200/JCO.2003.12.005 .

Hennessy BT, Hortobagyi GN, Rouzier R, Kuerer H, Sneige N, Buzdar AU, et al. Outcome after pathologic complete eradication of cytologically proven breast cancer axillary node metastases following primary chemotherapy. J Clin Oncol. 2005;23:9304–11. https://doi.org/10.1200/JCO.2005.02.5023 .

Kristensen LS, Jakobsen T, Hager H, Kjems J. The emerging roles of circRNAs in cancer and oncology. Nat Rev Clin Oncol. 2022;19:188–206.

Lou J, Hao Y, Lin K, Lyu Y, Chen M, Wang H, et al. Circular RNA CDR1as disrupts the p53/MDM2 complex to inhibit gliomagenesis. Mol Cancer. 2020;19:138.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Hanniford D, Ulloa-Morales A, Karz A, Berzoti-Coelho MG, Moubarak RS, Sánchez-Sendra B, et al. Epigenetic silencing of CDR1as drives IGF2BP3-mediated melanoma invasion and metastasis. Cancer Cell. 2020;37:55-70.e15.

Mao W, Wang K, Xu B, Zhang H, Sun S, Hu Q, et al. ciRS-7 is a prognostic biomarker and potential gene therapy target for renal cell carcinoma. Mol Cancer. 2021;20:142–7. https://doi.org/10.1186/s12943-021-01443-2 .

Piwecka M, Glažar P, Hernandez-Miranda LR, Memczak S, Wolf SA, Rybak-Wolf A, et al. Loss of a mammalian circular RNA locus causes miRNA deregulation and affects brain function. Science. 2017. https://doi.org/10.1126/science.aam8526 .

Yang W, Gu J, Wang X, Wang Y, Feng M, Zhou D, et al. Inhibition of circular RNA CDR1as increases chemosensitivity of 5-FU-resistant BC cells through up-regulating miR-7. J Cell Mol Med. 2019;23:3166–77. https://doi.org/10.1111/jcmm.14171 .

Yang W, Yang X, Wang X, Gu J, Zhou D, Wang Y, et al. Silencing CDR1as enhances the sensitivity of breast cancer cells to drug resistance by acting as a miR-7 sponge to down-regulate REGγ. J Cell Mol Med. 2019;23:4921–32. https://doi.org/10.1111/jcmm.14305 .

Zhou L, Xu S, Yin W, Lin Y, Du Y, Jiang Y, et al. Weekly paclitaxel and cisplatin as neoadjuvant chemotherapy with locally advanced breast cancer: a prospective, single arm, phase II study. Oncotarget. 2017;8:79305–14.

Article   PubMed   PubMed Central   Google Scholar  

Wolff AC, Hammond MEH, Hicks DG, Dowsett M, McShane LM, Allison KH, et al. Recommendations for human epidermal growth factor receptor 2 testing in breast cancer: American society of clinical oncology/college of American pathologists clinical practice guideline update. J Clin Oncol. 2013. https://doi.org/10.1200/JCO.2013.50.9984 .

Yuan C, Zhu X, Han Y, Song C, Liu C, Lu S, et al. Elevated HOXA1 expression correlates with accelerated tumor cell proliferation and poor prognosis in gastric cancer partly via cyclin D1. J Exp Clin Cancer Res. 2016;35:15. https://doi.org/10.1186/s13046-016-0294-2 .

Vo JN, Cieslik M, Zhang Y, Shukla S, Xiao L, Zhang Y, et al. The landscape of circular RNA in cancer. Cell. 2019;176:869-881.e13. https://doi.org/10.1016/j.cell.2018.12.021 .

Tang Z, Li C, Kang B, Gao G, Li C, Zhang Z. GEPIA: a web server for cancer and normal gene expression profiling and interactive analyses. Nucl Acid Res. 2017;45:W98-102.

Spring LM, Fell G, Arfe A, Sharma C, Greenup R, Reynolds KL, et al. Pathologic complete response after neoadjuvant chemotherapy and impact on breast cancer recurrence and survival: a comprehensive meta-analysis. Clin Cancer Res. 2020;26:2838–48.

Zhao Y, Zheng R, Chen J, Ning D. CircRNA CDR1as/miR-641/HOXA9 pathway regulated stemness contributes to cisplatin resistance in non-small cell lung cancer (NSCLC). Cancer Cell Int. 2020;20:289. https://doi.org/10.1186/s12935-020-01390-w .

Mao Y, Xu R. Circular RNA CDR1-AS contributes to pemetrexed and cisplatin chemoresistance through EGFR/PI3K signaling pathway in lung adenocarcinoma. Biomed Pharmacother. 2019;123:109771. https://doi.org/10.1016/j.biopha.2019.109771 .

Uhr K, Sieuwerts AM, de Weerd V, Smid M, Hammerl D, Foekens JA, et al. Association of microRNA-7 and its binding partner CDR1-AS with the prognosis and prediction of 1st-line tamoxifen therapy in breast cancer. Sci Rep. 2018;8:9657–714.

Jiang J, Li R, Wang J, Hou J, Qian H, Xu W. Circular RNA CDR1as inhibits the metastasis of gastric cancer through targeting miR-876–5p/GNG7 axis. Gastroenterol Res Pract. 2021;2021:5583029.

Chen H, Mao M, Jiang J, Zhu D, Li P. Circular RNA CDR1as acts as a sponge of miR-135b-5p to suppress ovarian cancer progression. OncoTargets Ther. 2019;12:3869–79.

Yuan W, Zhou R, Wang J, Han J, Yang X, Yu H, et al. Circular RNA Cdr1as sensitizes bladder cancer to cisplatin by upregulating APAF1 expression through miR-1270 inhibition. Mol Oncol. 2019;13:1559–76.

Weng W, Wei Q, Toden S, Yoshida K, Nagasaka T, Fujiwara T, et al. Circular RNA ciRS-7-A promising prognostic biomarker and a potential therapeutic target in colorectal cancer. Clin Cancer Res. 2017;23:3918–28. https://doi.org/10.1158/1078-0432.CCR-16-2541 .

Download references

Acknowledgements

We would like to thank all the investigators and patients participating in this study.

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by National Natural Science Foundation of China (No. 82203279, 82203093, 82173115 and 82103695), Clinical Research Plan of Shanghai Hospital Development Center (No. SHDC2020CR3003A), Science and Technology Commission of Shanghai Municipality (No. 20DZ2201600), Shanghai Municipal Key Clinical Specialty, Shanghai ‘Rising Stars of Medical Talent’ Youth Development Program for Outstanding Youth Medical Talents (No. 2018-16), Shanghai Rising-Star Program (No. 22QC1400200), Multidisciplinary Cross Research Foundation of Shanghai Jiao Tong University (No. YG2019QNA28), Clinical Research Innovation Nurturing Fund of Renji Hospital and United Imaging (No. 2021RJLY-002), Nurturing Fund of Renji Hospital (No. PYIII20-09, PY2018-III-15 and RJPY-LX-002), and Postdoctoral Fellowship Program of China Postdoctoral Science Foundation (No. GZC20230153).

Author information

Liheng Zhou, Yaohui Wang & Jinsong Lu

Present address: Department of Breast Surgery, Renji Hospital, Shanghai Jiao Tong University School of Medicine, No. 160 Pujian Road, Shanghai, 200127, People’s Republic of China

Chenwei Yuan and Yaqian Xu have contributed equally to this paper.

Authors and Affiliations

Department of Breast Surgery, Renji Hospital, Shanghai Jiao Tong University School of Medicine, No. 160 Pujian Road, Shanghai, 200127, People’s Republic of China

Chenwei Yuan, Jing Peng, Rui Sha, Yanping Lin, Shuguang Xu, Yumei Ye, Fan Yang, Tingting Yan, Xinrui Dong & Wenjin Yin

Breast Center, Peking University People’s Hospital, No.11 Xizhimen Southern Street, Beijing, 100044, People’s Republic of China

You can also search for this author in PubMed   Google Scholar

Contributions

JS Lu, WJ Yin, LH Zhou and CW Yuan designed and conducted the study. YP Lin, SG Xu, YM Ye, F Yang, and TT Yan collected the clinical data. CW Yuan, J Peng, R Sha and XR Dong carried out RNA extraction and RT-qPCR. CW Yuan and YQ Xu performed data analysis. CW Yuan drafted the manuscript. JS Lu and YH Wang revised the manuscript. All authors have read and approved the final manuscript.

Corresponding authors

Correspondence to Liheng Zhou , Yaohui Wang or Jinsong Lu .

Ethics declarations

Ethics approval and consent to participate.

The research protocols of both clinical trials SHPD001 (NCT02199418) and SHPD002 (NCT02221999) were reviewed and approved by the Independent Ethics Committee of Renji Hospital, School of Medicine, Shanghai Jiaotong University. Written informed consent was obtained from all patients.

Consent for publication

All authors agreed to publish this manuscript.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Yuan, C., Xu, Y., Zhou, L. et al. Value of CDR1-AS as a predictive and prognostic biomarker for patients with breast cancer receiving neoadjuvant chemotherapy in a prospective Chinese cohort. Eur J Med Res 29 , 454 (2024). https://doi.org/10.1186/s40001-024-02015-y

Download citation

Received : 08 May 2023

Accepted : 06 August 2024

Published : 11 September 2024

DOI : https://doi.org/10.1186/s40001-024-02015-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

European Journal of Medical Research

ISSN: 2047-783X

• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 10.9.2024 in Vol 26 (2024)

Prompt Engineering Paradigms for Medical Applications: Scoping Review

Authors of this article:

• Jamil Zaghir 1, 2 * , MSc   ; 
• Marco Naguib 3 * , MSc   ; 
• Mina Bjelogrlic 1, 2 , PhD   ; 
• Aurélie Névéol 3 , PhD   ; 
• Xavier Tannier 4 , PhD   ; 
• Christian Lovis 1, 2 , MPH, MD  

1 Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland

2 Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland

3 Université Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique, Orsay, France

4 Sorbonne Université, INSERM, Université Sorbonne Paris-Nord, Laboratoire d'Informatique Médicale et d'Ingénierie des Connaissances en eSanté, LIMICS, Paris, France

*these authors contributed equally

Corresponding Author:

Jamil Zaghir, MSc

Department of Radiology and Medical Informatics

University of Geneva

Chemin des Mines, 9

Geneva, 1202

Switzerland

Phone: 41 022 379 08 18

Email: [email protected]

Background: Prompt engineering, focusing on crafting effective prompts to large language models (LLMs), has garnered attention for its capabilities at harnessing the potential of LLMs. This is even more crucial in the medical domain due to its specialized terminology and language technicity. Clinical natural language processing applications must navigate complex language and ensure privacy compliance. Prompt engineering offers a novel approach by designing tailored prompts to guide models in exploiting clinically relevant information from complex medical texts. Despite its promise, the efficacy of prompt engineering in the medical domain remains to be fully explored.

Objective: The aim of the study is to review research efforts and technical approaches in prompt engineering for medical applications as well as provide an overview of opportunities and challenges for clinical practice.

Methods: Databases indexing the fields of medicine, computer science, and medical informatics were queried in order to identify relevant published papers. Since prompt engineering is an emerging field, preprint databases were also considered. Multiple data were extracted, such as the prompt paradigm, the involved LLMs, the languages of the study, the domain of the topic, the baselines, and several learning, design, and architecture strategies specific to prompt engineering. We include studies that apply prompt engineering–based methods to the medical domain, published between 2022 and 2024, and covering multiple prompt paradigms such as prompt learning (PL), prompt tuning (PT), and prompt design (PD).

Results: We included 114 recent prompt engineering studies. Among the 3 prompt paradigms, we have observed that PD is the most prevalent (78 papers). In 12 papers, PD, PL, and PT terms were used interchangeably. While ChatGPT is the most commonly used LLM, we have identified 7 studies using this LLM on a sensitive clinical data set. Chain-of-thought, present in 17 studies, emerges as the most frequent PD technique. While PL and PT papers typically provide a baseline for evaluating prompt-based approaches, 61% (48/78) of the PD studies do not report any nonprompt-related baseline. Finally, we individually examine each of the key prompt engineering–specific information reported across papers and find that many studies neglect to explicitly mention them, posing a challenge for advancing prompt engineering research.

Conclusions: In addition to reporting on trends and the scientific landscape of prompt engineering, we provide reporting guidelines for future studies to help advance research in the medical field. We also disclose tables and figures summarizing medical prompt engineering papers available and hope that future contributions will leverage these existing works to better advance the field.

Introduction

In recent years, the development of large language models (LLMs) such as GPT-3 has disrupted the field of natural language processing (NLP). LLMs have demonstrated capabilities in processing and generating human-like text, with applications ranging from text generation and translation to question answering and summarization [ 1 ]. However, harnessing the full potential of LLMs requires careful consideration of how input prompts are formulated and optimized [ 2 ].

Input prompts denote a set of instructions provided to the LLM to execute a task. Prompt engineering, a term coined to describe the strategic design and optimization of prompts for LLMs, has emerged as a crucial aspect of leveraging these models. By crafting prompts that effectively convey tasks or queries, researchers and practitioners can guide LLMs to improve the accuracy and pertinence of responses. The literature defines prompt engineering in various ways: it can be regarded as a prompt structuring process that enhances the efficiency of an LLM to achieve a specific objective [ 3 ] or as the mechanism through which LLMs are programmed by prompts [ 4 ]. Prompt engineering encompasses a plethora of techniques, often separated into distinct categories such as output customization and prompt improvement [ 4 ]. Existing prompt paradigms are presented in more detail in the Methods section.

In the realm of medical NLP, significant advancements have been made, such as the release of LLMs specialized in medical language and the availability of public medical data sets, including in languages other than English [ 5 ]. The unique intricacies of medical language, characterized by its terminological precision, context sensitivity, and domain-specific nuances, demand a dedicated focus and exploration of NLP in health care research. Despite these imperatives, to our knowledge, there is currently no systematic review analyzing prompt engineering applied to the medical domain.

The aim of this scoping review is to shed light on prompt engineering, as it is developed and used in the medical field, by systematically analyzing the literature in the field. Specifically, we examine the definitions, methodologies, techniques, and outcomes of prompt engineering across various NLP tasks. Methodological strengths, weaknesses, and limitations of the current wave of experimentation are discussed. Finally, we provide guidelines for comprehensive reporting of prompt engineering–related studies to improve clarity and facilitate further research in the field. We aspire to furnish insights that will inform both researchers and users about the pivotal role of prompt engineering in optimizing the efficacy of LLMs. By gaining a thorough understanding of the current landscape of prompt engineering research, we can pinpoint areas warranting further investigation and development, thereby propelling the field of medical NLP forward.

Study Design

Our scoping review was conducted following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines for scoping reviews (available in Multimedia Appendix 1 ). In this review, we use terminology to denote emerging technical concepts that lack consensus definitions. We propose the following definitions based on previous use in the literature:

• LLM: Object that models language and can be used to generate text by receiving large-scale language modeling pretraining (Luccioni and Rogers [ 6 ] define an arbitrary threshold at 1 billion tokens of training data). An LLM can be adapted to downstream tasks through transfer learning approaches such as fine-tuning or prompt-based techniques. Following the study of Thirunavukarasu et al [ 7 ] of models for the medical field, we include Bidirectional Encoder Representations From Transformers (BERT)–based and GPT-based models in this definition, although Zhao et al [ 8 ] place BERT models in a separate category.
• Fine-tuning: Approach in which the weights of the pretrained LLM are retrained on new samples. The additional data can be labeled and designed to adapt the LLM to a new downstream task.
• Prompt design (PD) [ 1 , 2 ]: Manually building a prompt (named manual prompt or hard prompt), tailored to guide the LLM toward resolving the task by simply predicting the most probable continuity of the prompt. The prompt is usually a set of task-specific instructions, occasionally featuring a few demonstrations of the task.
• Prompt learning (PL) [ 3 ]: Manually building a prompt and passing it to an LLM, trained via the masked language modeling (MLM) objective, to predict masked tokens. The prompt often features masked tokens, over which the LLM makes predictions. Those are then projected as predictions for a new downstream task. This approach is also referred to as prompt-based learning.
• Prompt tuning (PT) [ 9 ]: Refers to the LLM prompting where part or all the prompt is a trainable vectorial representation (known as continuous prompt or soft prompt) that is optimized with respect to the annotated instances.

Figure 1 illustrates the 4 approaches described above.

Inclusion and Exclusion Criteria

Studies were included if they met the following criteria: focus on prompt engineering, involvement of at least 1 LLM, relevance to the medical field (biomedical or clinical), pertaining to text-based generation (excluding vision-related prompts), and not focusing on prompting for academic writing purposes. Furthermore, as most of the first studies about prompt engineering emerged in 2022 [ 2 ], we added the following constraint: the publication date should be later than 2021.

Screening Process

The initial set of papers retrieved from the searches underwent screening based on titles, abstracts, and keywords. The search strategy is described in Multimedia Appendix 2 . Screening was performed by 2 reviewers (JZ and MN), working in a double-blind process. Interannotator agreement was calculated, with conflicts resolved through discussion.

Data Synthesis

We extracted information on prompt paradigms (PD, PL, and PT), involved LLMs, data sets used, studied language, domain (biomedical or clinical), medical subfield (if any), mentioned prompt engineering techniques, computational complexity, baselines, relative performances, and key findings. Additionally, we extracted journal information and noted instances of PD or PL or PT terminology misuse. Details are available in Multimedia Appendix 3 . Finally, we compile a list of recommendations based on the positive or negative trends we identify from the selected papers.

Screening Results

The systematic search across sources yielded 398 papers. Following the removal of duplicates, 251 papers underwent screening based on title, abstract, and keywords, leading to the exclusion of 94 studies. During this first screening step, 33 conflicts were identified and resolved among the annotators, resulting in an interannotator agreement of 86.8% (n=218). Subsequently, 157 studies remained, and full-text copies were retrieved and thoroughly screened. This process culminated in the inclusion of a total of 114 papers in this scoping review. The detailed process of study selection is shown in Figure 2 . Among the selected papers, 13 are from clinical venues, 33 are from medical informatics sources, 31 are from computer science publications, and 4 are from other sources. Notably, 33 of them are preprints.

Prompt Paradigms and Medical Subfields

Table 1 depicts the number of papers identified within each prompt paradigm along with their associated medical subfields. Some papers may simultaneously involve several (up to 2 in this review) prompt paradigms. Notably, PD emerged as the predominant category, with a total of 78 papers. These papers spanned across various medical fields, with a greater emphasis on clinical (including specialties) rather than biomedical disciplines. The screening yields 29 PL papers and 19 PT papers, with both paradigms maintaining a balanced distribution between biomedical and clinical domains. However, it is noteworthy that unlike PL and PT, PD encompassed a much broader spectrum of clinical specialties, with a particular interest in psychiatry.

Terminology Use

In our review, the consistency of terminology use around prompt engineering was investigated, particularly concerning its 3 paradigms: PD, PL, and PT. Across the papers, we meticulously tracked instances where the terminology was applied differently to the definitions used in the literature and described in the introduction. Notably, PL was used to refer to PD 4 times [ 12 , 13 , 67 , 86 ] and PT once [ 119 ], while PT was used 5 times to describe PL [ 88 , 96 , 97 , 99 , 114 ] and twice for PD [ 23 , 43 ]. Terminology inconsistencies were identified in only 12 studies. Consequently, while there remains some degree of inconsistency, a significant majority of 102 papers adhered to the definitions identified as commonly used terminology.

Language of Study

Considering the latest developments in NLP research encompassing languages beyond English [ 124 ], reporting the language of study is crucial. Several papers do not explicitly state the language of study. In some cases, the language can be inferred from prompt illustrations or examples. In the least informative cases, only the data set of the study is disclosed, indirectly hinting at the language.

Table 2 illustrates the language distribution among the selected papers, noting whether languages are explicitly mentioned, implicitly inferred from prompt illustrations, or simply not stated but implied from the used data set. The language used in 2 papers [ 60 , 68 ] remains unknown.

a Stated in the paper.

b Inferred from prompt figures and examples.

c Inferred from the data set.

Notably, English dominates with 84.2% (n=96) of the selected papers, followed by Chinese at 15.7% (n=18). Then, the other languages are relatively rare, often appearing in studies featuring multiple languages. It is worth mentioning that languages besides English are usually explicitly stated, with the exception of a paper studying Korean [ 63 ]. In total, the language had to be inferred from prompt figures and examples in 48 papers, all in English.

Choice of LLMs

Given the diverse array of LLMs available, spanning general or medical, open-source or proprietary, and monolingual or multilingual models, alongside various architectural configurations (encoder, decoder, or both), our study investigates LLM selection across prompt paradigms.

Figure 3 outlines prevalent LLMs categorized by prompt paradigms, though it is not exhaustive and only includes commonly encountered architectures. For example, while encoder-decoder models are absent in PT in Figure 3 , there are a few instances where they are used [ 95 , 110 ].

ChatGPT’s popularity in PD is unsurprising, given its accessibility. Models from Google, PaLM, and Bard (subsequently rebranded Gemini), all falling under closed models, are also prominent. Among open-source instruct-based LLMs, fewer are used, notably those based on LLaMA-2 with 7 occurrences.

In PL, encoder models, those following the BERT architecture, dominate, covering both general and specialized variants. There are occasional uses of decoder models like GPT-2 in PL-based tasks [ 103 , 105 ]. PT involves all model types, with a preference toward encoders. Further details on the models used are available in Multimedia Appendix 3 .

Topic Domain and NLP Task Trends

Figure 4 [ 16 , 20 , 26 , 41 , 47 , 88 - 123 ] illustrates the target tasks used in the PL and PT papers. PL-focused papers predominantly address classification-based tasks such as text classification, named entity recognition, and relation extraction, with text classification being particularly prominent. This aligns with the nature of PL, which centers around an MLM objective. Among other tasks, a study based on text generation [ 111 ] makes use of PL to predict masked tokens from partial patient records, aiming to generate synthetic electronic health records. Conversely, PT papers tend to exhibit a slightly broader range of tasks.

Figure 5 [ 10 - 87 ] presents the same analysis for PD-based papers. Unlike PL and PT, a prominent trend observed is that several studies focus on real-world board examinations. Notably, these studies predominantly center around tasks involving answering multiple-choice questions (MCQs). It is worth noting that although MCQs might be cast as a classification task, in practice, it is cast as a generation task using causal LLMs. It is interesting to note that none of the selected PD papers propose the task of entity linking, despite the clear opportunity of leveraging LLMs’ in-context learning ability for medical entity linking.

Prompt Engineering Techniques

We extensively investigated the used prompt techniques: among PD papers, 49 studies used zero-shot prompting, 23 used few-shot prompting, and 10 used one-shot prompting. Few shot tends to outperform in MCQs, but its advantage over zero shot is inconsistent in other NLP tasks. We propose a comprehensive summary of the existing techniques in Table 3 .

As shown in Table 3 , chain-of-thought (CoT) prompting [ 2 ] stands as the most common technique, followed by the persona pattern. In medical MCQs, various attempts with CoT can lead to different reasoning pathways and answers. Hence, to improve accuracy, 2 studies [ 19 , 20 ] used self-consistency, a method involving using multiple CoT prompts and selecting the most frequently occurring answer through voting.

Flipped interaction was used for simulation tasks, such as doctor-patient engagement [ 60 ] or to provide clinical training to medical students [ 81 ]. Emotion enhancement was applied in mental health contexts [ 58 , 60 ], allowing the LLM to produce emotional statements.

More innovative prompt engineering techniques include k-nearest neighbor few-shot prompting [ 19 ] and pseudoclassification prompting [ 78 ]. The former uses the k-nearest neighbor algorithm to select the k-closest examples in a large annotated data set based on the input before using them in the prompt, and the latter presents to the LLMs all possible labels, asking the model to respond with a binary output for each provided label. Despite its potential, tree-of-thoughts pattern use was limited, with only 1 instance found among the papers [ 77 ].

Emerging Trends

Figure 6 illustrates a chronological polar pie chart of selected papers and their citation connections, identifying five highly cited papers: (1) Agrawal et al [ 40 ] demonstrate GPT-3’s clinical task performance, especially in named entity recognition and relation extraction through thorough PD. (2) Kung et al [ 36 ] evaluate ChatGPT’s (GPT-3.5) ability for the United States Medical Licensing Examination, shortly after the public release of ChatGPT. (3) Singhal et al [ 20 ] introduce MultiMedQA and HealthSearchQA benchmarks. The paper also presents instruction PT for domain alignment, a novel paradigm that entails learning a soft prompt prior to the LLM general instruction, which is usually written as a hard prompt. Using this approach on FlanPaLM led to the development of Med-PaLM, improving question answering over FlanPaLM. (4) Nori et al [ 27 ] evaluate GPT-4 on the United States Medical Licensing Examination and MultiMedQA, surpassing previous state-of-the-art results, including GPT-3.5 and Med-PaLM. (5) Luo et al [ 26 ] release BioGPT, a fine-tuned variant of GPT-2 for biomedical tasks, achieving state-of-the-art results on 6 biomedical NLP tasks with suffix-based PT.

Trends in PD

As shown in Figure 6 , the PD paradigm presents multiple trends: all papers disseminated in clinical-based venues, and 27 of 33 (82%) of the encountered preprints adhere to this paradigm. Furthermore, we observed a significant focus on work involving frozen LLMs within the PD domain. This trend is likely due to the frequent use of ChatGPT in 74 instances, as depicted in Figure 3 , despite OpenAI offering fine-tuning capabilities for the model. It is worth mentioning that 46 of 78 (59%) PD papers do not include any baseline, including human comparison. This gap will be further explored in a subsequent section.

Trends in PL and PT

Among PL and PT papers, computer science and medical informatics are the most prevalent venues. Although PL has drawn attention to the idea of adapting the MLM objective to downstream tasks without needing to further update the LLM weights, many studies still opt to fine-tune their LLMs, with a nonnegligible amount of them evaluating in few-shot settings [ 89 , 92 , 93 , 112 ]. Unlike PD, PL and PT usually include a baseline, with it often being a traditional fine-tuning version of the evaluated model [ 92 , 93 , 95 ] to compare it against novel prompt-based paradigms. These studies came to a common conclusion, being that PL is a promising alternative to traditional fine-tuning in few-shot scenarios.

There are 2 ways for conducting PL: one involves filling in the blanks within a text, known as cloze prompts, while the other consists in predicting masked tokens at the end of the sequence, referred to as prefix prompts. A distinct advantage of the latter approach is its compatibility with autoregressive models, as they exclusively predict the appended masks. Among the 29 PL papers, 21 (72%) of them propose cloze prompts, while 15 (52%) use prefix prompting. The involved NLP tasks are well-distributed across these 2 prompt patterns. Another crucial component of PL is the verbalizer. As PL revolves around predicting masked tokens, classification-based tasks require mapping manually selected relevant tokens to each class (manual verbalizer). Alternatively, some studies propose a soft verbalizer, akin to soft prompts, which automatically determines the most relevant token embedding for each label through training. Of the 29 PL papers selected, 16 (55%) studies explicitly mention the use of a manual verbalizer, while 2 explored both verbalizers to assess performance [ 101 , 110 ]. Only 1 exclusively used a soft verbalizer [ 89 ]. Another study does not use any verbalizer, as it focuses on generating synthetic data by filling the blanks [ 111 ]. Notably, 8 (28%) studies did not report any mention regarding the verbalizer methodology.

Hard prompts, which are related to PD and PL, involve manually crafted prompts. Regarding PT, optimal prompts are attainable through soft prompting (ie, prompts that are trained on a training data set), yet, determining the appropriate soft prompt length remains obscure. In total, 5 of 19 (26%) PT studies tried various soft prompt lengths and reported their corresponding performances [ 26 , 105 , 118 , 119 , 122 ]. While there is no definitive optimal prompt length, a trend emerges: optimal soft prompt length typically exceeds 10 tokens. Surprisingly, 8 (42%) papers omit reporting the soft prompt length. Regarding the placement of soft prompts in relation to the input and the mask, consensus is lacking. A total of 5 (26%) papers prepend the soft prompt at the input’s outset, while 4 (21%) append it as a suffix. One paper uses both strategies in a single prompt template [ 95 ]. Some innovative methods involve inserting a single soft prompt for each entity that needs to be identified in entity-linking tasks or using token-wise soft prompts, where each token in the textual input is accompanied by a distinct soft prompt. The position of soft prompts remains unreported in 5 (26%) studies. Finally, according to the 6 (32%) studies that used mixed prompts [ 90 , 91 , 95 , 101 , 105 , 110 ] (a combination of hard and soft prompts), it has consistently been reported that mixed prompts lead to a better performance than hard prompts alone.

Baseline Comparison

Only 62 of the screened papers reported comparisons to established baselines. These include traditional deep learning approaches (eg, fine-tuning approach), classical machine learning algorithms (eg, logistic regression), naive systems (eg, majority class), or human annotation. The remaining papers solely explored prompt-related solutions, without including baseline comparisons. Tables 4 - 6 traces the presence of a nonprompt baseline among different prompt categories ( Table 4 ), papers sources ( Table 5 ), and NLP tasks addressed ( Table 6 ).

a Higher or lower indicates that the performance of the proposed prompt-based approach is higher or lower than the baseline.

a NLP: natural language processing.

b Higher or lower indicates that the performance of the proposed prompt-based approach is higher or lower than the baseline.

Nonprompt-related baselines are often featured in studies focused on PL and PT but not PD. Additionally, PL and PT have a tendency to perform better than their respective reported baselines, PD tends to report less conclusive results. More specifically, among the 22 papers using either PL or PT with an identical fine-tuned model as a baseline, 17 indicate superior performance with the prompt-based approach, 3 observed comparable performance, and 2 studies noted inferior performance.

Significantly, papers from computer science venues tend to include more state-of-the-art baselines than those from medical informatics and clinical venues. Specifically, all 13 papers reviewed from clinical venues did not use any nonprompt baselines. Furthermore, there appears to be no consistent link between the type of NLP tasks and the omission of baselines, indicating that the decision to include baselines is more influenced by the evaluation methodology than by feasibility.

Prompt Optimization

Numerous studies in the literature highlight the few-shot learning capabilities of LLMs, often referred to as “few-shot prompting,” wherein they demonstrate proficiency in executing tasks with minimal demonstrations provided, typically through text prompts. However, it is crucial to acknowledge that the annotation cost associated with such frameworks might extend beyond the few annotated demonstrations within the prompt. Many studies claiming to explore few-shot or zero-shot learning through prompt engineering rely on extensive annotated validation data sets to refine PD and formulation. This is, for example, the case in the paper that popularized the term “few-shot learning” [ 1 ]. Among the 45 analyzed papers concentrating on few-shot or zero-shot learning, 5 explicitly detail the optimization of prompt formulation using extensive validation data sets. Conversely, 18 of these papers either do not engage in prompt optimization or test various prompts and document all results. Notably, 22 papers present results using only 1 prompt choice, without clarifying whether this choice was made thanks to additional validation data sets.

Summary of the Findings

This scoping review aimed to map the current landscape of medical prompt engineering, identifying key themes, gaps, and trends within the existing literature. The primary findings of this study reveal a greater prevalence of PD over PL and PT, with ChatGPT dominating the PD domain. Additionally, many studies omit nonprompt-based baselines, do not specify the language of study, or exhibit a lack of consensus in PL (prefix vs cloze prompt) and PT settings (soft prompt lengths and positions). English is notably dominant as the language of study. These findings suggest that while the field is emerging, there is a pressing need for improved research practices.

Costs, Infrastructure, and LLMs in Clinical Settings

Prompt engineering techniques enable competitive performance in scenarios with limited or no resources as well as in environments with low-cost computing infrastructure. As hospital data and infrastructure are often found in this scenario, these approaches hold great promise in the clinical field. Figure 6 shows the absence of PL- and PT-related works in clinical journals. This trend may stem from the widespread accessibility of ChatGPT, favoring PD-focused investigations. Despite efforts like OpenPrompt [ 125 ] to facilitate PL and PT works, the programming barrier likely deters clinical practitioners. Surprisingly, 7 papers use ChatGPT with sensitive clinical data. Despite the recent availability of ChatGPT Enterprise in GPT-4 for secure data handling, it is apparent that most of these studies have not used this feature since they used GPT-3.5. Limited use of local LLMs, especially LLaMA-based, suggests a need for their increased adoption in future clinical PD studies. The lack of local LLMs may be due to clinicians’ limited computational infrastructure.

Prompt Engineering Techniques Effectiveness in Medical Research

In documented prompt engineering techniques, the effectiveness of few-shot prompting compared to zero shot varies by task and scenario. However, CoT shows superior reasoning performance, compelling LLMs to present reasoning pathways and consistently outperforming zero-shot and few-shot methods across PD studies. Its ensemble-based variant, self-consistency, consistently outperforms CoT. Despite the persona pattern’s frequent use, there is a lack of ablation studies on its impact on medical task performance, with only 1 paper reporting negligible improvement [ 61 ]. Prompt engineering is an emerging field of study that still needs to prove its efficacy. However, almost half of the papers focused only on prompt engineering and failed to report any nonprompt-related baseline performance, despite the availability of such baselines for the addressed NLP tasks. On the whole, the results are far from being systematically in favor of LLM-based methods, greatly attenuating the impression of a technological breakthrough that is generally commented on. Selecting a baseline remains a necessary step toward understanding the actual impact of prompt engineering.

Bender Rule

Regarding the languages, while Table 2 shows the dominance of English in medical literature, many papers studying English fail to explicitly mention the language of study. This oversight is more prevalent in computer science and clinical venues, whereas medical informatics exhibits a more favorable trend, as validated by a chi-square test yielding a P value of .02 (Table S1 in Multimedia Appendix 2 ). Notably, languages such as Chinese are consistently mentioned across the 18 selected papers. However, the Bender rule, namely “always name the language(s) you are working on,” seems to be well respected for languages other than English. This finding has already been documented for NLP research in general [ 126 ].

Fine-Tuning Versus Prompt-Based Approaches

While traditional LLM fine-tuning remains a viable method for various NLP tasks, PL and PT are competitive alternatives to fine-tuning, particularly in resource-constrained and low computational scenarios. PL, leveraging predefined prompts to guide model behavior, offers an efficient approach in low-to-no resource environments. Conversely, PT emerges as a viable solution in low computational scenarios, as it requires substantially fewer trainable parameters compared to traditional fine-tuning approaches. Since both prompt-based approaches do not require the LLM to be further trained, they are less prone to catastrophic forgetting [ 127 ].

Recommendations for Future Medical Prompt–Based Studies

For future research in prompt engineering, we propose several recommendations aimed at improving research quality, reporting, and reproducibility. From this review, we identified several trends such as the computational advantages or the lack of evaluations on baselines with a lack of ablation studies to evaluate the performance of the prompting strategies. Some studies do not clearly mention the prompt engineering choices they made. For instance, in PL, choices range from using cloze to prefix prompting and from using manual to soft verbalizer. Similarly, PT is characterized by configurations of soft prompts, such as the length and the positions. To clarify these distinctions and enhance methodological transparency and reproducibility in future research, we have developed reporting guidelines available in Textbox 1 . Adhering to these reporting guidelines will contribute to advancing prompt engineering methodologies and their practical applications in the medical field.

General reporting recommendations

• For sensitive data, local large language models (LLMs) should be preferred to the ones that use an application programming interface or a web service.
• The language of the study used should be explicitly stated.
• The mention of whether the LLM undergoes fine-tuning should be made explicit.
• The prompt optimization process and results should be documented to ensure transparency, whether it is through different tested manual prompts or through a validation data set.
• The terms “few-shot,” “one-shot,” and “zero-shot” should not be used in settings where the prompts have been optimized on annotated examples.
• Experiments should include baseline comparisons or at least mention existing results, particularly when data sets originate from previous medical challenges or benchmarks.

Specific to prompt learning and prompt tuning

• Concepts (such as prompt learning and prompt tuning) should be defined and used consistently with the consensus.
• In prompt learning experiments, the verbalizer used (soft and hard) should be explicitly specified, or a clear justification should be provided if the verbalizer is omitted. Additionally, whether the prompt template follows the cloze or the prefix format should be mentioned.
• In prompt tuning experiments, authors should provide details on soft prompt positions, length, and any variations tested, such as incorporating hard or mixed prompts, as part of the ablation study.

Limitations

A limitation was the large number of papers retrieved during the initial search, which was addressed by limiting the search scope to titles, abstracts, and keywords. Furthermore, since some studies may perform prompt engineering techniques without mentioning any of the 4 prompt-related expressions used in the queries, they might be missed by our searches.

Conclusions

Medical prompt engineering is an emerging field with significant potential for enhancing clinical applications, particularly in resource-constrained environments. Despite the promising capabilities demonstrated, there is a pressing need for standardized research practices and comprehensive reporting to ensure methodological transparency and reproducibility. Consistent evaluation against nonprompt-based baselines, prompt optimization documentation, and prompt settings reporting will be crucial for advancing the field. We hope that a better adherence to the recommended guidelines, in Textbox 1 , will improve our understanding of prompt engineering and enhance the capabilities of LLMs in health care.

Acknowledgments

JZ is financed by the NCCR Evolving Language, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (grant # 51NF40_180888).

Authors' Contributions

JZ and MN performed the screening and data extraction of the papers and synthesized the findings. AN and XT supervised MN. MB and CL supervised JZ. JZ and MN wrote the manuscript with support from MB, AN, XT, and CL. All authors contributed to the analysis of the results. CL conceived the original idea.

Conflicts of Interest

CL is the editor-in-chief of JMIR Medical Informatics . All other authors have no conflict of interest to declare.

PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist.

Search strategy and statistical analysis.

Reading notes and details of the reviewed papers.

• Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language models are few-shot learners. 2020. Presented at: Advances in Neural Information Processing Systems; December 6, 2020:1877-1901; Virtual. URL: https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
• Kojima T, Gu SS, Reid M, Matsuo Y, Iwasawa Y. Large language models are zero-shot reasoners. 2022. Presented at: Advances in Neural Information Processing Systems; November 28, 2022:22199-22213; New Orleans. URL: https:/​/papers.​nips.cc/​paper_files/​paper/​2022/​hash/​8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.​html
• Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput Surv. Jan 16, 2023;55(9):1-35. [ CrossRef ]
• White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, et al. A prompt pattern catalog to enhance prompt engineering with ChatGPT. ArXiv. Preprint posted online on February 21, 2023. [ FREE Full text ] [ CrossRef ]
• Névéol A, Dalianis H, Velupillai S, Savova G, Zweigenbaum P. Clinical natural language processing in languages other than English: opportunities and challenges. J Biomed Semantics. 2018;9(1):12. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Luccioni AS, Rogers A. Mind your language (model): fact-checking LLMs and their role in NLP research and practice. ArXiv. Preprint posted online on June 1, 2024. [ CrossRef ]
• Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930-1940. [ CrossRef ] [ Medline ]
• Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, et al. A survey of large language models. ArXiv. Preprint posted online on November 24, 2023. [ FREE Full text ] [ CrossRef ]
• Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. 2021. Presented at: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing; January 10, 2021:3045-3059; Online and Punta Cana, Dominican Republic. [ CrossRef ]
• Fries J, Weber L, Seelam N, Altay G, Datta D, Garda S, et al. BigBIO: a framework for data-centric biomedical natural language processing. 2022. Presented at: Advances in Neural Information Processing Systems; November 28, 2022:25792-25806; New Orleans. URL: https:/​/proceedings.​neurips.cc/​paper_files/​paper/​2022/​hash/​a583d2197eafc4afdd41f5b8765555c5-Abstract-Datasets_and_Benchmarks.​html
• Weisenthal SJ. ChatGPT and post-test probability. ArXiv. Preprint posted online on July 20, 2024. [ FREE Full text ] [ CrossRef ]
• Li L, Ning W. ProBioRE: a framework for biomedical causal relation extraction based on dual-head prompt and prototypical network. 2023. Presented at: 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); December 5, 2023:2071-2074; Istanbul, Turkiye. URL: https://tinyurl.com/3n45uwdb
• Li Z, Belkadi S, Micheletti N, Han L, Shardlow M, Nenadic G. Large language models and control mechanisms improve text readability of biomedical abstracts. ArXiv. Preprint posted online on March 16, 2024. [ FREE Full text ] [ CrossRef ]
• Li Q, Yang X, Wang H, Liu L, Wang Q, Wang J, et al. From beginner to expert: modeling medical knowledge into general LLMs. ArXiv. Preprint posted online on January 7, 2024. [ FREE Full text ] [ CrossRef ]
• Ateia S, Kruschwitz U. Is ChatGPT a biomedical expert? 2023. Presented at: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023); September 18-21, 2023:73-90; Thessaloniki, Greece. URL: https://ceur-ws.org/Vol-3497/paper-006.pdf
• Belyaeva A, Cosentino J, Hormozdiari F, Eswaran K, Shetty S, Corrado G, et al. Multimodal LLMs for health grounded in individual-specific data. 2023. Presented at: Machine Learning for Multimodal Healthcare Data; July 29, 2023:86-102; Honolulu, Hawaii, United States. [ CrossRef ]
• Chen Q, Sun H, Liu H, Jiang Y, Ran T, Jin X, et al. An extensive benchmark study on biomedical text generation and mining with ChatGPT. Bioinformatics. 2023;39(9):btad557. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mollá D. Large language models and prompt engineering for biomedical query focused multi-document summarisation. ArXiv. Preprint posted online on November 9, 2023. [ FREE Full text ] [ CrossRef ]
• Nori H, Lee YT, Zhang S, Carignan D, Edgar R, Fusi N, et al. Can generalist foundation models outcompete special-purpose tuning? Case study in medicine. ArXiv. Preprint posted online on November 28, 2023. [ FREE Full text ] [ CrossRef ]
• Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tian S, Jin Q, Yeganova L, Lai PT, Zhu Q, Chen X, et al. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Brief Bioinform. 2023;25(1):bbad493. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lim S, Schmälzle R. Artificial intelligence for health message generation: an empirical study using a large language model (LLM) and prompt engineering. Front Commun. 2023;8:1129082. [ CrossRef ]
• Wu YH, Lin YJ, Kao HY. IKM_Lab at BioLaySumm Task 1: longformer-based prompt tuning for biomedical lay summary generation. 2023. Presented at: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks; July 13, 2023; Toronto, Canada. [ CrossRef ]
• Zhang W, Chen C, Wang J, Liu J, Ruan T. A co-adaptive duality-aware framework for biomedical relation extraction. Bioinformatics. 2023;39(5):btad301. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen P, Wang J, Lin H, Zhao D, Yang Z. Few-shot biomedical named entity recognition via knowledge-guided instance generation and prompt contrastive learning. Bioinformatics. 2023;39(8):btad496. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Luo R, Sun L, Xia Y, Qin T, Zhang S, Poon H, et al. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief Bioinform. 2022;23(6):bbac409. [ CrossRef ] [ Medline ]
• Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on medical challenge problems. ArXiv. Preprint posted online on April 12, 2023. [ FREE Full text ] [ CrossRef ]
• Heinz MV, Bhattacharya S, Trudeau B, Quist R, Song SH, Lee CM, et al. Testing domain knowledge and risk of bias of a large-scale general artificial intelligence model in mental health. Digit Health. 2023;9:20552076231170499. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ting Y, Hsieh T, Wang Y, Kuo Y, Chen Y, Chan P, et al. Performance of ChatGPT incorporated chain-of-thought method in bilingual nuclear medicine physician board examinations. Digit Health. 2024;10:20552076231224074. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Casola S, Labruna T, Lavelli A, Magnini B. Testing ChatGPT for stability and reasoning: a case study using Italian medical specialty tests. 2023. Presented at: Proceedings of the 9th Italian Conference on Computational Linguistics; November 30-Decemeber 2, 2023; Venice, Italy. URL: https://ceur-ws.org/Vol-3596/paper13.pdf
• Roemer G, Li A, Mahmood U, Dauer L, Bellamy M. Artificial intelligence model GPT4 narrowly fails simulated radiological protection exam. J Radiol Prot. 2024;44(1):013502. [ CrossRef ] [ Medline ]
• Ali S, Shahab O, Al Shabeeb R, Ladak F, Yang JO, Nadkarni G, et al. General purpose large language models match human performance on gastroenterology board exam self-assessments. MedRxi. Preprint posted online on September 25, 2023. [ FREE Full text ] [ CrossRef ]
• Patel D, Raut G, Zimlichman E, Cheetirala S, Nadkarni G, Glicksberg BS, et al. The limits of prompt engineering in medical problem-solving: a comparative analysis with ChatGPT on calculation based USMLE medical questions. MedRxiv. Preprint posted online on August 9, 2023. [ FREE Full text ] [ CrossRef ]
• Sallam M, Al-Salahat K, Eid H, Egger J, Puladi B. Human versus artificial intelligence: ChatGPT-4 outperforming Bing, Bard, ChatGPT-3.5, and humans in clinical chemistry multiple-choice questions. MedRxiv. Preprint posted online on January 9, 2024. [ FREE Full text ] [ CrossRef ]
• Savage T, Nayak A, Gallo R, Rangan E, Chen JH. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digit Med. 2024;7(1):20. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tanaka Y, Nakata T, Aiga K, Etani T, Muramatsu R, Katagiri S, et al. Performance of generative pretrained transformer on the national medical licensing examination in Japan. PLOS Digit Health. 2024;3(1):e0000433. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rosoł M, Gąsior JS, Łaba J, Korzeniewski K, Młyńczak M. Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish medical final examination. Sci Rep. 2023;13(1):20512. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sivarajkumar S, Kelley M, Samolyk-Mazzanti A, Visweswaran S, Wang Y. An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing. ArXiv. Preprint posted online on September 14, 2023. [ FREE Full text ] [ CrossRef ]
• Agrawal M, Hegselmann S, Lang H, Kim Y, Sontag D. Large language models are few-shot clinical information extractors. 2022. Presented at: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing; May 25, 2022:1998-2022; Abu Dhabi. [ CrossRef ]
• Dong B, Wang Z, Li Z, Duan Z, Xu J, Pan T, et al. Toward a stable and low-resource PLM-based medical diagnostic system via prompt tuning and MoE structure. Sci Rep. 2023;13(1):12595. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gutierrez KLT, Viacrusis PML. Bridging the gap or widening the divide: a call for capacity-building in artificial intelligence for healthcare in the Philippines. JMUST. 2023;7(2):1325-1334. [ CrossRef ]
• Islam KS, Nipu AS, Madiraju P, Deshpande P. Autocompletion of chief complaints in the electronic health records using large language models. 2023. Presented at: 2023 IEEE International Conference on Big Data (BigData); December 15-18, 2023:4912-4921; Sorrento, Italy. URL: https://tinyurl.com/4ajdyddt [ CrossRef ]
• Meoni S, Ryffel T, De La Clergerie É. Annotate French clinical data using large language model predictions. 2023. Presented at: 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI); June 26-29, 2023:550-557; Houston, TX, United States. URL: https://tinyurl.com/yy2b9fe8 [ CrossRef ]
• Meoni S, De la Clergerie E, Ryffel T. Large language models as instructors: a study on multilingual clinical entity extraction. 2023. Presented at: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks; July 2023:178-190; Toronto, Canada. URL: https://aclanthology.org/2023.bionlp-1.15/ [ CrossRef ]
• Wang X, Yang Q. LingX at ROCLING 2023 MultiNER-health task: intelligent capture of Chinese medical named entities by LLMs. 2023. Presented at: Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023); October 20-21, 2023; Taipei City, Taiwan. URL: https://aclanthology.org/2023.rocling-1.44.pdf
• Yang Y, Li X, Wang H, Guan Y, Jiang J. Modeling clinical thinking based on knowledge hypergraph attention network and prompt learning for disease prediction. SSRN. Preprint posted online on June 30, 2023. [ FREE Full text ] [ CrossRef ]
• Yao Z, Jaafar A, Wang B, Zhu Y, Yang Z, Yu H. Do physicians know how to prompt? The need for automatic prompt optimization help in clinical note generation. ArXiv. Preprint posted online on July 5, 2024. [ FREE Full text ] [ CrossRef ]
• van Zandvoort D, Wiersema L, Huibers T, van Dulmen S, Brinkkemper S. Enhancing summarization performance through transformer-based prompt engineering in automated medical reporting. ArXiv. Preprint posted online on January 19, 2024. [ FREE Full text ] [ CrossRef ]
• Zhang B, Mishra R, Teodoro D. DS4DH at MEDIQA-Chat 2023: leveraging SVM and GPT-3 prompt engineering for medical dialogue classification and summarization. 2023. Presented at: Proceedings of the 5th Clinical Natural Language Processing Workshop; June 12, 2023:536-545; Toronto, Canada. [ CrossRef ]
• Zhu W, Wang X, Chen M, Tang B. Overview of the PromptCBLUE Shared Task in CHIP2023. ArXiv. Preprint posted online on December 29, 2023. [ FREE Full text ] [ CrossRef ]
• Caruccio L, Cirillo S, Polese G, Solimando G, Sundaramurthy S, Tortora G. Can ChatGPT provide intelligent diagnoses? A comparative study between predictive models and ChatGPT to define a new medical diagnostic bot. Expert Syst Appl. 2024;235:121186. [ CrossRef ]
• Lee Y, Chen C, Chen C, Lee C, Chen P, Wu C, et al. Unlocking the secrets behind advanced artificial intelligence language models in deidentifying Chinese-English mixed clinical text: development and validation study. J Med Internet Res. 2024;26:e48443. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bhaumik R, Srivastava V, Jalali A, Ghosh S, Chandrasekaran R. Mindwatch: a smart cloud-based AI solution for suicide ideation detection leveraging large language models. MedRxiv. Preprint posted online on September 26, 2023. [ FREE Full text ] [ CrossRef ]
• Heston TF. Safety of large language models in addressing depression. Cureus. 2023:15. [ FREE Full text ] [ CrossRef ]
• Grabb D. The impact of prompt engineering in large language model performance: a psychiatric example. J Med Artif Intell. 2023;6:20. [ FREE Full text ] [ CrossRef ]
• Santos WR, Paraboni I. Prompt-based mental health screening from social media text. ArXiv. Preprint posted online on May 11, 2024. [ FREE Full text ] [ CrossRef ]
• Yang K, Ji S, Zhang T, Xie Q, Kuang Z, Ananiadou S. Towards interpretable mental health analysis with large language models. 2023. Presented at: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing; January 16, 2023:6056-6077; Singapore. [ CrossRef ]
• Xu X, Yao B, Dong Y, Gabriel S, Yu H, Hendler J, et al. Mental-LLM: leveraging large language models for mental health prediction via online text data. Proc ACM Interact Mob Wearable Ubiquitous Technol. 2024;8(1):1-32. [ CrossRef ]
• Chen S, Wu M, Zhu KQ, Lan K, Zhang Z, Cui L. LLM-empowered chatbots for psychiatrist and patient simulation: application and evaluation. ArXiv. Preprint posted online on May 23, 2023. [ FREE Full text ] [ CrossRef ]
• Qi H, Zhao Q, Li J, Song C, Zhai W, Dan L, et al. Supervised learning and large language model benchmarks on mental health datasets: cognitive distortions and suicidal risks in Chinese social media. ResearchSquare. Preprint posted online on November 02, 2023. [ FREE Full text ] [ CrossRef ]
• Sambath V. Advancements of artificial intelligence in mental health applications?: A comparative analysis of ChatGPT 3.5 and ChatGPT 4. ResearchGate. Preprint posted online on December, 2023. [ FREE Full text ] [ CrossRef ]
• Choi HS, Song JY, Shin KH, Chang JH, Jang B. Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer. Radiat Oncol J. 2023;41(3):209-216. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lee DT, Vaid A, Menon KM, Freeman R, Matteson DS, Marin MP, et al. Development of a privacy preserving large language model for automated data extraction from thyroid cancer pathology reports. MedRxiv. Preprint posted online on November 8, 2023. [ FREE Full text ] [ CrossRef ]
• Dennstädt F, Hastings J, Putora PM, Vu E, Fischer GF, Süveg K, et al. Exploring capabilities of large language models such as ChatGPT in radiation oncology. Adv Radiat Oncol. 2024;9(3):101400. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhu S, Gilbert M, Ghanem AI, Siddiqui F, Thind K. Feasibility of using zero-shot learning in transformer-based natural language processing algorithm for key information extraction from head and neck tumor board notes. Int J Radiat Oncol Biol Phys. 2023;117(2):e500. [ CrossRef ]
• Zhao X, Zhang M, Ma M, Su C, Wang M, Qiao X, et al. HW-TSC at SemEval-2023 task 7: exploring the natural language inference capabilities of ChatGPT and pre-trained language model for clinical trial. 2023. Presented at: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023); July 10, 2023:1603-1608; Toronto, Canada. [ CrossRef ]
• Nazary F, Deldjoo Y, Di Noia T. ChatGPT-HealthPrompt. Harnessing the power of XAI in prompt-based healthcare decision support using ChatGPT. 2023. Presented at: Artificial Intelligence. ECAI 2023 International Workshops; September 30-October 4, 2023:382-397; Kraków, Poland. [ CrossRef ]
• Wang B, Lai J, Cao H, Jin F, Tang M, Yao C, et al. Enhancing real-world data extraction in clinical research: evaluating the impact of the implementation of large language models in hospital setting. ResearchSquare. Preprint posted online on November 29, 2023. [ FREE Full text ] [ CrossRef ]
• Mishra V, Sarraju A, Kalwani NM, Dexter JP. Evaluation of prompts to simplify cardiovascular disease information using a large language model. MedRxiv. Preprint posted online on November 9, 2023. [ FREE Full text ] [ CrossRef ]
• Feng R, Brennan KA, Azizi Z, Goyal J, Pedron M, Chang HJ, et al. Optimizing ChatGPT to detect VT recurrence from complex medical notes. Circulation. 2023;148(Suppl 1):A16401. [ CrossRef ]
• Chowdhury M, Lim E, Higham A, McKinnon R, Ventoura N, He Y, et al. Can large language models safely address patient questions following cataract surgery? 2023. Presented at: Proceedings of the 5th Clinical Natural Language Processing Workshop; June 10, 2023:131-137; Toronto, Canada. [ CrossRef ]
• Kleinig O, Gao C, Kovoor JG, Gupta AK, Bacchi S, Chan WO. How to use large language models in ophthalmology: from prompt engineering to protecting confidentiality. Eye (Lond). 2024;38(4):649-653. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arsenyan V, Bughdaryan S, Shaya F, Small K, Shahnazaryan D. Large language models for biomedical knowledge graph construction: information extraction from EMR notes. ArXiv. Preprint posted online on December 9, 2023. [ FREE Full text ] [ CrossRef ]
• Kwon T, Ong KT, Kang D, Moon S, Lee JR, Hwang D, et al. Large language models are clinical reasoners: reasoning-aware diagnosis framework with prompt-generated rationales. 2024. Presented at: Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence; February 20, 2024:18417-18425; Vancouver. [ CrossRef ]
• Wang C, Liu S, Li A, Liu J. Text dialogue analysis for primary screening of mild cognitive impairment: development and validation study. J Med Internet Res. 2023;25:e51501. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li J, Wang L, Chen X, Deng X, Wen H, You M, et al. Are you asking GPT-4 medical questions properly?—Prompt engineering in consistency and reliability with evidence-based guidelines for ChatGPT-4: a pilot study. ResearchSquare. Posted online on October 3, 2023. 2023:1-20. [ FREE Full text ] [ CrossRef ]
• Zaidat B, Lahoti YS, Yu A, Mohamed KS, Cho SK, Kim JS. Artificially intelligent billing in spine surgery: an analysis of a large language model. Global Spine J. 2023:21925682231224753. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Datta S, Lee K, Paek H, Manion FJ, Ofoegbu N, Du J, et al. AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models. J Am Med Inform Assoc. 2024;31(2):375-385. [ FREE Full text ] [ CrossRef ] [ Medline ]
• White R, Peng T, Sripitak P, Rosenberg Johansen A, Snyder M. CliniDigest: a case study in large language model based large-scale summarization of clinical trial descriptions. 2023. Presented at: Proceedings of the 2023 ACM Conference on Information Technology for Social Good; September 6-8, 2023; Lisbon, Portugal. [ CrossRef ]
• Scherr R, Halaseh FF, Spina A, Andalib S, Rivera R. ChatGPT interactive medical simulations for early clinical education: case study. JMIR Med Educ. 2023;9:e49877. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Akinci D'Antonoli T, Stanzione A, Bluethgen C, Vernuccio F, Ugga L, Klontzas ME, et al. Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions. Diagn Interv Radiol. 2024;30(2):80-90. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wiest IC, Ferber D, Zhu J, van Treeck M, Meyer SK, Juglan SK, et al. From text to tables: a local privacy preserving large language model for structured information retrieval from medical documents. MedRxiv. Preprint posted online on December 8, 2023. [ FREE Full text ] [ CrossRef ]
• Hamed E, Eid A, Alberry M. Exploring ChatGPT's potential in facilitating adaptation of clinical guidelines: a case study of diabetic ketoacidosis guidelines. Cureus. 2023;15(5):e38784. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Leypold T, Schäfer B, Boos A, Beier JP. Can AI think like a plastic surgeon? Evaluating GPT-4's clinical judgment in reconstructive procedures of the upper extremity. Plast Reconstr Surg Glob Open. 2023;11(12):e5471. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yang J, Liu C, Deng W, Wu D, Weng C, Zhou Y, et al. Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. Patterns (NY). 2024;5(1):100887. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xiong L, Zeng Q, Deng W, Luo W, Liu R. A novel approach to nursing clinical intelligent decision-making: integration of large language models and local knowledge bases. ResearchSquare. Preprint posted online on December 8, 2023. [ FREE Full text ] [ CrossRef ]
• Zeng Q, Liu Y, He P. A medical question classification approach based on prompt tuning and contrastive learning. 2023. Presented at: The Thirty Fifth International Conference on Software Engineering and Knowledge Engineering (SEKE 2023); July 1-10, 2023:632-635; San Francisco, CA, United States. [ CrossRef ]
• Zhao D, Yang Y, Chen P, Meng J, Sun S, Wang J, et al. Biomedical document relation extraction with prompt learning and KNN. J Biomed Inform. 2023;145:104459. [ CrossRef ] [ Medline ]
• Zhu T, Qin Y, Feng M, Chen Q, Hu B, Xiang Y. BioPRO: context-infused prompt learning for biomedical entity linking. IEEE/ACM Trans Audio Speech Lang Process. 2024;32(2023):374-385. [ CrossRef ]
• Liu C, Zhang S, Li C, Zhao H. CPK-Adapter: infusing medical knowledge into K-adapter with continuous prompt. 2023. Presented at: 2023 8th International Conference on Intelligent Computing and Signal Processing (ICSP); April 21-23, 2023:1017-1023; Xi'an, China. [ CrossRef ]
• Yeh HS, Lavergne T, Zweigenbaum P. Decorate the examples: a simple method of prompt design for biomedical relation extraction. 2022. Presented at: Proceedings of the Thirteenth Language Resources and Evaluation Conference; August 13, 2024:3780-3787; Marseille, France. URL: https://aclanthology.org/2022.lrec-1.403
• Su Z, Yu X, Chen P. EPTQA: a Chinese medical prompt learning method based on entity pair type question answering. SSRN. 2023:24. [ CrossRef ]
• Xu H, Zhang J, Wang Z, Zhang S, Bhalerao M, Liu Y, et al. GraphPrompt: graph-based prompt templates for biomedical synonym prediction. 2023. Presented at: Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence; February 7, 2023:10576-10584; Washington, DC, United States. [ CrossRef ]
• Chen T, Stefanidis A, Jiang Z, Su J. Improving biomedical claim detection using prompt learning approaches. 2023. Presented at: 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML); August 4-6, 2023:369-376; Urumqi, China. [ CrossRef ]
• Xu Z, Chen Y, Hu B. Improving biomedical entity linking with cross-entity interaction. 2023. Presented at: Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence; February 7, 2023:13869-13877; Washington. [ CrossRef ]
• Wang Y, Wang Y, Peng Z, Zhang F, Zhou L, Yang F. Medical text classification based on the discriminative pre-training model and prompt-tuning. Digit Health. 2023;9:1-14. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tian X, Wang P, Mao S. Open-world biomedical knowledge probing and verification. 2023. Presented at: Proceedings of The 12th International Joint Conference on Knowledge Graphs (IJCKG-23); December 8-9, 2023; Tokyo, Japan. URL: https://ijckg2023.knowledge-graph.jp/pages/proc/paper_3.pdf
• Lu K, Potash P, Lin X, Sun Y, Qian Z, Yuan Z, et al. Prompt discriminative language models for domain adaptation. 2023. Presented at: Proceedings of the 5th Clinical Natural Language Processing Workshop; July 14, 2023:247-258; Toronto, Canada. [ CrossRef ]
• Hu Y, Chen Y, Xu H. Towards more generalizable and accurate sentence classification in medical abstracts with less data. J Healthc Inform Res. 2023;7(4):542-556. [ CrossRef ] [ Medline ]
• Taylor N, Zhang Y, Joyce DW, Gao Z, Kormilitzin A, Nevado-Holgado A. Clinical prompt learning with frozen language models. IEEE Trans Neural Netw Learn Syst. 2023:1-11. [ CrossRef ] [ Medline ]
• Landi I, Alleva E, Valentine AA, Lepow LA, Charney AW. Clinical text deduplication practices for efficient pretraining and improved clinical tasks. ArXiv. Preprint posted online on September 29, 2023. [ FREE Full text ] [ CrossRef ]
• Sivarajkumar S, Wang Y. HealthPrompt: a zero-shot learning paradigm for clinical natural language processing. AMIA Annu Symp Proc. 2022;2022:972-981. [ FREE Full text ] [ Medline ]
• Sivarajkumar S, Wang Y. Evaluation of healthprompt for zero-shot clinical text classification. 2023. Presented at: 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI); June 26-29, 2023:492-494; Houston, TX, United States. [ CrossRef ]
• Zhang L, Liu J. Intent-aware prompt learning for medical question summarization. 2022. Presented at: 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); December 6-8, 2022:672-679; Las Vegas, NV, United States. [ CrossRef ]
• Alleva E, Landi I, Shaw LJ, Böttinger E, Fuchs TJ, Ensari I. Keyword-optimized template insertion for clinical information extraction via prompt-based learning. ArXiv. Preprint posted online on October 31, 2023. [ FREE Full text ] [ CrossRef ]
• Cui Z, Yu K, Yuan Z, Dong X, Luo W. Language inference-based learning for low-resource Chinese clinical named entity recognition using language model. J Biomed Inform. 2024;149:104559. [ CrossRef ] [ Medline ]
• Ahmed A, Zeng X, Xi R, Hou M, Shah SA. MED-Prompt: a novel prompt engineering framework for medicine prediction on free-text clinical notes. J King Saud Univ Comput Inf Sci. 2024;36(2):1-17. [ CrossRef ]
• Lu Y, Liu X, Du Z, Gao Y, Wang G. MedKPL: a heterogeneous knowledge enhanced prompt learning framework for transferable diagnosis. J Biomed Inform. 2023;143:104417. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cui Y, Han L, Nenadic G. MedTem2.0: prompt-based temporal classification of treatment events from discharge summaries. 2023. Presented at: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop); July 10-12, 2023:160-183; Toronto, Canada. [ CrossRef ]
• Wang Z, Sun J. PromptEHR: conditional electronic healthcare records generation with prompt learning. 2022. Presented at: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics; October 11, 2022:2873-2855; Abu Dhabi, United Arab Emirates. [ CrossRef ]
• Wang S, Tang L, Majety A, Rousseau JF, Shih G, Ding Y, et al. Trustworthy assertion classification through prompting. J Biomed Inform. 2022;132:104139. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yao Z, Tsai J, Liu W, Levy DA, Druhl E, Reisman JI, et al. Automated identification of eviction status from electronic health record notes. J Am Med Inform Assoc. 2023;30(8):1429-1437. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kwon S, Wang X, Liu W, Druhl E, Sung ML, Reisman JI, et al. ODD: a benchmark dataset for the natural language processing based opioid related aberrant behavior detection. 2024. Presented at: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; June 16, 2024:1-22; Mexico City. [ CrossRef ]
• Su J, Zhang J, Peng P, Wang H. EGDE: a framework for bridging the gap in medical zero-shot relation triplet extraction. 2023. Presented at: 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); December 5-8, 2023; Istanbul, Turkiye. [ CrossRef ]
• Li Q, Wang Y, You T, Lu Y. BioKnowPrompt: incorporating imprecise knowledge into prompt-tuning verbalizer with biomedical text for relation extraction. Inf Sci. 2022;617:346-358. [ CrossRef ]
• Peng C, Yang X, Chen A, Yu Z, Smith KE, Costa AB, et al. Generative large language models are all-purpose text analytics engines: text-to-text learning is all your need. J Am Med Inform Assoc. Sep 01, 2024;31(9):1892-1903. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Peng C, Yang X, Smith KE, Yu Z, Chen A, Bian J, et al. Model tuning or prompt tuning? A study of large language models for clinical concept and relation extraction. J Biomed Inform. 2024;153:104630. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Duan J, Lu F, Liu J. MVP: optimizing multi-view prompts for medical dialogue summarization. 2023. Presented at: 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); December 5-8, 2023; Istanbul, Turkiye. [ CrossRef ]
• Rohanian O, Jauncey H, Nouriborji M, Kumar V, Gonalves BP, Kartsonaki C, et al. Using bottleneck adapters to identify cancer in clinical notes under low-resource constraints. 2023. Presented at: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks; July 23, 2023:6239-6278; Toronto, Canada. [ CrossRef ]
• Elfrink A, Vagliano I, Abu-Hanna A, Calixto I. Soft-prompt tuning to predict lung cancer using primary care free-text Dutch medical notes. In: Artificial Intelligence in Medicine. Cham. Springer Nature Switzerland; 2023:193-198.
• Singh Rawat BP, Yu H. Parameter efficient transfer learning for suicide attempt and ideation detection. 2022. Presented at: Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI); September 7, 2022:108-115; Abu Dhabi, United Arab Emirates. [ CrossRef ]
• Xu S, Wan X, Hu S, Zhou M, Xu T, Wang H, et al. COSSUM: towards conversation-oriented structured summarization for automatic medical insurance assessment. 2022. Presented at: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 14-18, 2022:4248-4256; Washington, DC, United States. [ CrossRef ]
• Shaitarova A, Zaghir J, Lavelli A, Krauthammer M, Rinaldi F. Exploring the latest highlights in medical natural language processing across multiple languages: a survey. Yearb Med Inform. 2023;32(1):230-243. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ding N, Hu S, Zhao W, Chen Y, Liu Z, Zheng HT, et al. OpenPrompt: an open-source framework for prompt-learning. ArXiv. Preprint posted online on November 3, 2021. [ FREE Full text ] [ CrossRef ]
• Ducel F, Fort K, Lejeune G, Lepage Y. Do we name the languages we study? The #BenderRule in LREC and ACL articles. 2022. Presented at: Proceedings of the Thirteenth Language Resources and Evaluation Conference; June 20-25, 2022:564-573; Marseille, France. URL: https://aclanthology.org/2022.lrec-1.60
• Luo Y, Yang Z, Meng F, Li Y, Zhou J, Zhang Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. ArXiv. Preprint posted online on April 2, 2024. [ FREE Full text ] [ CrossRef ]

Abbreviations

Edited by T de Azevedo Cardoso; submitted 14.05.24; peer-reviewed by B Bhasuran, D Hu, A Jain; comments to author 03.07.24; revised version received 09.07.24; accepted 22.07.24; published 10.09.24.

©Jamil Zaghir, Marco Naguib, Mina Bjelogrlic, Aurélie Névéol, Xavier Tannier, Christian Lovis. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 10.09.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.