systematic review paper research pdf

• Open access
• Published: 01 August 2019

A step by step guide for conducting a systematic review and meta-analysis with simulation data

• Gehad Mohamed Tawfik 1 , 2 ,
• Kadek Agus Surya Dila 2 , 3 ,
• Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
• Dao Ngoc Hien Tam 2 , 5 ,
• Nguyen Dang Kien 2 , 6 ,
• Ali Mahmoud Ahmed 2 , 7 &
• Nguyen Tien Huy 8 , 9 , 10  

Tropical Medicine and Health volume  47 , Article number:  46 ( 2019 ) Cite this article

811k Accesses

302 Citations

94 Altmetric

Metrics details

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With all of the words: ebola virus

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

With all of the words: EVD

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Availability of data and materials

Not applicable.

Abbreviations

Network meta-analysis

Principal investigator

Population, Intervention, Comparison, Outcome

Preferred Reporting Items for Systematic Review and Meta-analysis statement

Quality assessment

Sample, Phenomenon of Interest, Design, Evaluation, Research type

Systematic review and meta-analyses

Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.

Article   Google Scholar  

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.

PubMed   Google Scholar  

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.

Article   CAS   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.

Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.

Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.

Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.

Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.

Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.

Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.

Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.

Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.

RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.

Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.

Google Scholar  

Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.

Download references

Acknowledgements

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Author information

Authors and affiliations.

Faculty of Medicine, Ain Shams University, Cairo, Egypt

Gehad Mohamed Tawfik

Online research Club http://www.onlineresearchclub.org/

Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed

Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia

Kadek Agus Surya Dila

Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Muawia Yousif Fadlelmola Mohamed

Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Dao Ngoc Hien Tam

Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Nguyen Dang Kien

Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Ali Mahmoud Ahmed

Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Nguyen Tien Huy

Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

Corresponding author

Correspondence to Nguyen Tien Huy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Additional file 2:

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Additional file 3:

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Additional file 4:

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Additional file 5:

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Additional file 6:

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Additional file 7:

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6

Download citation

Received : 30 January 2019

Accepted : 24 May 2019

Published : 01 August 2019

DOI : https://doi.org/10.1186/s41182-019-0165-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Tropical Medicine and Health

ISSN: 1349-4147

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Easy guide to conducting a systematic review

Affiliations.

• 1 Discipline of Child and Adolescent Health, University of Sydney, Sydney, New South Wales, Australia.
• 2 Department of Nephrology, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• 3 Education Department, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• PMID: 32364273
• DOI: 10.1111/jpc.14853

A systematic review is a type of study that synthesises research that has been conducted on a particular topic. Systematic reviews are considered to provide the highest level of evidence on the hierarchy of evidence pyramid. Systematic reviews are conducted following rigorous research methodology. To minimise bias, systematic reviews utilise a predefined search strategy to identify and appraise all available published literature on a specific topic. The meticulous nature of the systematic review research methodology differentiates a systematic review from a narrative review (literature review or authoritative review). This paper provides a brief step by step summary of how to conduct a systematic review, which may be of interest for clinicians and researchers.

Keywords: research; research design; systematic review.

© 2020 Paediatrics and Child Health Division (The Royal Australasian College of Physicians).

Publication types

• Systematic Review
• Research Design*

• Open access
• Published: 01 June 2024

Biomarkers for personalised prevention of chronic diseases: a common protocol for three rapid scoping reviews

• E Plans-Beriso   ORCID: orcid.org/0000-0002-9388-8744 1 , 2   na1 ,
• C Babb-de-Villiers 3   na1 ,
• D Petrova 2 , 4 , 5 ,
• C Barahona-López 1 , 2 ,
• P Diez-Echave 1 , 2 ,
• O R Hernández 1 , 2 ,
• N F Fernández-Martínez 2 , 4 , 5 ,
• H Turner 3 ,
• E García-Ovejero 1 ,
• O Craciun 1 ,
• P Fernández-Navarro 1 , 2 ,
• N Fernández-Larrea 1 , 2 ,
• E García-Esquinas 1 , 2 ,
• V Jiménez-Planet 7 ,
• V Moreno 2 , 8 , 9 ,
• F Rodríguez-Artalejo 2 , 10 , 11 ,
• M J Sánchez 2 , 4 , 5 ,
• M Pollan-Santamaria 1 , 2 ,
• L Blackburn 3 ,
• M Kroese 3   na2 &
• B Pérez-Gómez 1 , 2   na2  

Systematic Reviews volume  13 , Article number:  147 ( 2024 ) Cite this article

338 Accesses

4 Altmetric

Metrics details

Introduction

Personalised prevention aims to delay or avoid disease occurrence, progression, and recurrence of disease through the adoption of targeted interventions that consider the individual biological, including genetic data, environmental and behavioural characteristics, as well as the socio-cultural context. This protocol summarises the main features of a rapid scoping review to show the research landscape on biomarkers or a combination of biomarkers that may help to better identify subgroups of individuals with different risks of developing specific diseases in which specific preventive strategies could have an impact on clinical outcomes.

This review is part of the “Personalised Prevention Roadmap for the future HEalThcare” (PROPHET) project, which seeks to highlight the gaps in current personalised preventive approaches, in order to develop a Strategic Research and Innovation Agenda for the European Union.

To systematically map and review the evidence of biomarkers that are available or under development in cancer, cardiovascular and neurodegenerative diseases that are or can be used for personalised prevention in the general population, in clinical or public health settings.

Three rapid scoping reviews are being conducted in parallel (February–June 2023), based on a common framework with some adjustments to suit each specific condition (cancer, cardiovascular or neurodegenerative diseases). Medline and Embase will be searched to identify publications between 2020 and 2023. To shorten the time frames, 10% of the papers will undergo screening by two reviewers and only English-language papers will be considered. The following information will be extracted by two reviewers from all the publications selected for inclusion: source type, citation details, country, inclusion/exclusion criteria (population, concept, context, type of evidence source), study methods, and key findings relevant to the review question/s. The selection criteria and the extraction sheet will be pre-tested. Relevant biomarkers for risk prediction and stratification will be recorded. Results will be presented graphically using an evidence map.

Inclusion criteria

Population: general adult populations or adults from specific pre-defined high-risk subgroups; concept: all studies focusing on molecular, cellular, physiological, or imaging biomarkers used for individualised primary or secondary prevention of the diseases of interest; context: clinical or public health settings.

Systematic review registration

https://doi.org/10.17605/OSF.IO/7JRWD (OSF registration DOI).

Peer Review reports

In recent years, innovative health research has moved quickly towards a new paradigm. The ability to analyse and process previously unseen sources and amounts of data, e.g. environmental, clinical, socio-demographic, epidemiological, and ‘omics-derived, has created opportunities in the understanding and prevention of chronic diseases, and in the development of targeted therapies that can cure them. This paradigm has come to be known as “personalised medicine”. According to the European Council Conclusion on personalised medicine for patients (2015/C 421/03), this term defines a medical model which involves characterisation of individuals’ genotypes, phenotypes and lifestyle and environmental exposures (e.g. molecular profiling, medical imaging, lifestyle and environmental data) for tailoring the right therapeutic strategy for the right person at the right time, and/or to determine the predisposition to disease and/or to deliver timely and targeted prevention [ 1 , 2 ]. In many cases, these personalised health strategies have been based on advances in fields such as molecular biology, genetic engineering, bioinformatics, diagnostic imaging and new’omics technologies, which have made it possible to identify biomarkers that have been used to design and adapt therapies to specific patients or groups of patients [ 2 ]. A biomarker is defined as a substance, structure, characteristic, or process that can be objectively quantified as an indicator of typical biological functions, disease processes, or biological reactions to exposure [ 3 , 4 ].

Adopting a public health perspective within this framework, one of the most relevant areas that would benefit from these new opportunities is the personalisation of disease prevention. Personalised prevention aims to delay or avoid the occurrence, progression and recurrence of disease by adopting targeted interventions that take into account biological information, environmental and behavioural characteristics, and the socio-economic and cultural context of individuals. These interventions should be timely, effective and equitable in order to maintain the best possible balance in lifetime health trajectory [ 5 ].

Among the main diseases that merit specific attention are chronic noncommunicable diseases, due to their incidence, their mortality or disability-adjusted life years [ 6 , 7 , 8 , 9 ]. Within the European Union (EU), in 2021, one-third of adults reported suffering from a chronic condition [ 10 ]. In addition, in 2019, the leading causes of mortality were cardiovascular disease (CVD) (35%), cancer (26%), respiratory disease (8%), and Alzheimer's disease (5%) [ 11 ]. For all of the above, in 2019, the PRECeDI consortium recommended the identification of biomarkers that could be used for the prevention of chronic diseases to integrate personalised medicine in the field of chronicity. This will support the goal of stratifying populations by indicating an individuals’ risk or resistance to disease and their potential response to drugs, guiding primary, secondary and tertiary preventive interventions [ 12 ]; understanding primary prevention as measures taken to prevent the occurrence of a disease before it occurs, secondary prevention as actions aimed at early detection, and tertiary prevention as interventions to prevent complications and improve quality of life in individuals already affected by a disease [ 4 ].

The “Personalised Prevention roadmap for the future HEalThcare” (PROPHET) project, funded by the European Union’s Horizon Europe research and innovation program and linked to ICPerMed, seeks to assess the effectiveness, clinical utility, and existing gaps in current personalised preventive approaches, as well as their potential to be implemented in healthcare settings. It also aims to develop a Strategy Research and Innovation Agenda (SRIA) for the European Union. This protocol corresponds to one of the first steps in the PROPHET, namely a review that aims to map the evidence and highlight the evidence gaps in research or the use of biomarkers in personalised prevention in the general adult population, as well as their integration with digital technologies, including wearable devices, accelerometers, and other appliances utilised for measuring physical and physiological functions. These biomarkers may be already available or currently under development in the fields of cancer, CVD, and neurodegenerative diseases.

There is already a significant body of knowledge about primary and secondary prevention strategies for these diseases. For example, hypercholesterolemia or dyslipidaemia, hypertension, smoking, diabetes mellitus and obesity or levels of physical activity are known risk factors for CVD [ 6 , 13 ] and neurodegenerative diseases [ 14 , 15 , 16 ]; for cancer, a summary of lifestyle preventive actions with good evidence is included in the European code against cancer [ 17 ]. The question is whether there is any biomarker or combination of biomarkers that can help to better identify subgroups of individuals with different risks of developing a particular disease, in which specific preventive strategies could have an impact on clinical outcomes. Our aim in this context is to show the available research in this field.

Given the context and time constraints, the rapid scoping review design is the most appropriate method for providing landscape knowledge [ 18 ] and provide summary maps, such as Campbell evidence and gap map [ 19 ]. Here, we present the protocol that will be used to elaborate three rapid scoping reviews and evidence maps of research on biomarkers investigated in relation to primary or secondary prevention of cancer, cardiovascular and neurodegenerative diseases, respectively. The results of these three rapid scoping reviews will contribute to inform the development of the PROPHET SRIA, which will guide the future policy for research in this field in the EU.

Review question

What biomarkers are being investigated in the context of personalised primary and secondary prevention of cancer, CVD and neurodegenerative diseases in the general adult population in clinical or public health settings?

Three rapid scoping reviews are being conducted between February and June 2023, in parallel, one for each disease group included (cancer, CVD and neurodegenerative diseases), using a common framework and specifying the adaptations to each disease group in search terms, data extraction and representation of results.

This research protocol, designed according to Joanna Briggs Institute (JBI) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist [ 20 , 21 , 22 ] was uploaded to the Open Science Framework for public consultation [ 23 ], with registration DOI https://doi.org/ https://doi.org/10.17605/OSF.IO/7JRWD . The protocol was also reviewed by experts in the field, after which modifications were incorporated.

Eligibility criteria

Following the PCC (population, concept and context) model [ 21 , 22 ], the included studies will meet the following eligibility criteria (Table  1 ):

Rationale for performing a rapid scoping review

As explained above, these scoping reviews are intended to be one of the first materials produced in the PROPHET project, so that they can inform the first draft of the SRIA. Therefore, according to the planned timetable, the reviews should be completed in only 4 months. Thus, following recommendations from the Cochrane Rapid Review Methods Group [ 24 ] and taking into account the large number of records expected to be assessed, according to the preliminary searches, and in order to meet these deadlines, specific restrictions were defined for the search—limited to a 3-year period (2020–2023), in English only, and using only MEDLINE and EMBASE as possible sources—and it was decided that the title-abstract and full-text screening phase would be carried out by a single reviewer, after an initial training phase with 10% of the records assessed by two reviewers to ensure concordance between team members. This percentage could be increased if necessary.

Rationale for population selection

These rapid scoping reviews are focused on the general adult population. In addition, they give attention to studies conducted among populations that present specific risk factors relevant to the selected diseases or that include these factors among those considered in the study.

For cancer, these risk (or preventive) factors include smoking [ 25 ], obesity [ 26 ], diabetes [ 27 , 28 , 29 ], Helicobacter pylori infection/colonisation [ 30 ], human papillomavirus (HPV) infection [ 30 ], human immunodeficiency virus (HIV) infection [ 30 ], alcohol consumption [ 31 ], liver cirrhosis and viral (HVB, HVC, HVD) hepatitis [ 32 ].

For CVD, we include hypercholesterolemia or dyslipidaemia, arterial hypertension, smoking, diabetes mellitus, chronic kidney disease, hyperglycaemia and obesity [ 6 , 13 ].

Risk groups for neurodegenerative diseases were defined based on the following risk factors: obesity [ 15 , 33 ], arterial hypertension [ 15 , 33 , 34 , 35 ], diabetes mellitus [ 15 , 33 , 34 , 35 ], dyslipidaemia [ 33 ], alcohol consumption [ 36 , 37 ] and smoking [ 15 , 16 , 33 , 34 ].

After the general search, only relevant and/or disease-specific subpopulations will be used for each specific disease. On the other hand, pregnancy is an exclusion criterion, as the very specific characteristics of this population group would require a specific review.

Rationale for disease selection

The search is limited to diseases with high morbidity and mortality within each of the three disease groups:

Cancer type

Due to time constraints, we only evaluate those malignant neoplasms with the greatest mortality and incidence rates in Europe, which according to the European Cancer Information System [ 38 ] are breast, prostate, colorectum, lung, bladder, pancreas, liver, stomach, kidney, and corpus uteri. Additionally, cervix uteri and liver cancers will also be included due to their preventable nature and/or the existence of public health screening programs [ 30 , 31 ].

We evaluate the following main causes of deaths: ischemic heart disease (49.2% of all CVD deaths), stroke (35.2%) (this includes ischemic stroke, intracerebral haemorrhage and subarachnoid haemorrhage), hypertensive heart disease (6.2%), cardiomyopathy and myocarditis (1.8%), atrial fibrillation and flutter (1.7%), rheumatic heart disease (1.6%), non-rheumatic valvular heart disease (0.9%), aortic aneurism (0.9%), peripheral artery disease (0.4%) and endocarditis (0.4%) [ 6 ].

In this scoping review, specifically in the context of CVD, rheumatic heart disease and endocarditis are not considered because of their infectious aetiology. Arterial hypertension is a risk factor for many cardiovascular diseases and for the purposes of this review is considered as an intermediary disease that leads to CVD.

• Neurodegenerative diseases

The leading noncommunicable neurodegenerative causes of death are Alzheimer’s disease or dementia (20%), Parkinson’s disease (2.5%), motor neuron diseases (0.4%) and multiple sclerosis (0.2%) [ 8 ]. Alzheimer’s disease, vascular dementia, frontotemporal dementia and Lewy body disease will be specifically searched, following the pattern of European dementia prevalence studies [ 39 ]. Additionally, because amyotrophic lateral sclerosis is the most common motor neuron disease, it is also included in the search [ 8 , 40 , 41 ].

Rationale for context

Public health and clinical settings from any geographical location are being considered. The searches will only consider the period between January 2020 and mid-February 2023 due to time constraints.

Rationale for type of evidence

Qualitative studies are not considered since they cannot answer the research question. Editorials and opinion pieces, protocols, and conference abstracts will also be excluded. Clinical practice guidelines are not included since the information they contain should be in the original studies and in reviews on which they are based.

Pilot study

We did a pilot study to test and refine the search strategies, selection criteria and data extraction sheet as well as to get used to the software—Covidence [ 42 ]. The pilot study consisted of selecting from the results of the preliminary search matrix 100 papers in order of best fit to the topic, and 100 papers at random. The team comprised 15 individual reviewers (both in the pilot and final reviews) who met daily to revise, enhance, and reach consensus on the search matrices, criteria, and data extraction sheets.

Regarding the selected databases and the platforms used, we conducted various tests, including PubMed/MEDLINE and Ovid/MEDLINE, as well as Ovid/Embase and Elsevier/Embase. Ultimately, we chose Ovid as the platform for accessing both MEDLINE and Embase, utilizing thesaurus Mesh and EmTrees. We manually translated these thesauri to ensure consistency between them. Given that the review team was spread across the UK and Spain, we centralised the search results within the UK team's access to the Ovid license to ensure consistency. Additionally, using Ovid exclusively for accessing both MEDLINE and Embase streamlined the process and allowed for easier access to preprints, which represent the latest research in this rapidly evolving field.

Identification of research

The searches are being conducted in MEDLINE via Ovid, Embase via Ovid and Embase preprints via Ovid. We also explored the feasibility of searching in CDC-Authored Genomics and Precision Health Publications Databases [ 43 ] . However, the lack of advanced tools to refine the search, as well as the unavailability of bulk downloading prevented the inclusion of this data source. Nevertheless, a search with 15 records for each disease group showed a full overlap with MEDLINE and/or Embase.

Search strategy definition

An initial limited search of MEDLINE via PubMed and Ovid was undertaken to identify relevant papers on the topic. In this step, we identified keytext words in their titles and abstracts, as well as thesaurus terms. The SR-Accelerator, Citationchaser, and Yale Mesh Analyzer tools were used to assist in the construction of the search matrix. With all this information, we developed a full search strategy adapted for each included database and information source, optimised by research librarians.

Study evidence selection

The complete search strategies are shown in Additional file 3. The three searches are being conducted in parallel. When performing the search, no limits to the type of study or setting are being applied.

Following each search, all identified citations will be collated and uploaded into Covidence (Veritas Health Innovation, Melbourne, Australia, available at www.covidence.org ) with the citation details, and duplicates will be removed.

In the title-abstract and full-text screening phase, the first 10% of the papers will be evaluated by two independent reviewers (accounting for 200 or more papers in absolute numbers in the title-abstract phase). Then, a meeting to discuss discrepancies will lead to adjusting inclusion and exclusion criteria and to acquire consistency between reviewers’ decisions. After that, the full screening of the search results will be performed by a single reviewer. Disagreements that arise between reviewers at each stage of the selection process will be resolved through discussion, or with additional reviewers. We maintain an active forum to facilitate permanent contact among reviewers.

The results of the searches and the study inclusion processes will be reported and presented in a flow diagram following the PRISMA-ScR recommendations [ 22 ].

Expert consultation

The protocol has been refined after consultation with experts in each field (cancer, CVD, and neurodegenerative diseases) who gave input on the scope of the reviews regarding the diverse biomarkers, risk factors, outcomes, and types of prevention relevant to their fields of expertise. In addition, the search strategies have been peer-reviewed by a network of librarians (PRESS-forum in pressforum.pbworks.com) who kindly provided useful feedback.

Data extraction

We have developed a draft data extraction sheet, which is included as Additional file 4, based on the JBI recommendations [ 21 ]. Data extraction will include citation details, study design, population type, biomarker information (name, type, subtype, clinical utility, use of AI technology), disease (group, specific disease), prevention (primary or secondary, lifestyle if primary prevention), and subjective reviewer observations. The data extraction for all papers will be performed by two reviewers to ensure consistency in the classification of data.

Data analysis and presentation

The descriptive information about the studies collected in the previous phase will be coded according to predefined categories to allow the elaboration of visual summary maps that can allow readers and researchers to have a quick overview of their main results. As in the previous phases, this process will be carried out with the aid of Covidence.

Therefore, a summary of the extracted data will be presented in tables as well as in static and, especially, through interactive evidence gap maps (EGM) created using EPPI-Mapper [ 44 ], an open-access web application developed in 2018 by the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) and Digital Solution Foundry, in partnership with the Campbell Collaboration, which has become the standard software for producing visual evidence gap maps.

Tables and static maps will be made by using R Studio, which will also be used to clean and prepare the database for its use in EPPI-Mapper by generating two Excel files: one containing the EGM structure (i.e. what will be the columns and rows of the visual table) and coding sets, and another containing the bibliographic references and their codes that reviewers had added. Finally, we will use a Python script to produce a file in JSON format, making it ready for importation into EPPI-Reviewer.

The maps are matrixes with biomarker categories/subcategories defining the rows and diseases serving as columns. They define cells, which contain small squares, each one representing each paper included in it. We will use a code of colours to reflect the study design. There will be also a second sublevel in the columns, depending on the map. Thus, for each group of diseases, we will produce three interactive EGMs: two for primary prevention and one for secondary prevention. For primary prevention, the first map will stratify the data to show whether any or which lifestyle has been considered in each paper in combination with the studied biomarker. The second map for primary prevention and the map for secondary prevention will include, as a second sublevel, the subpopulations in which the biomarker has been used or evaluated, which are disease-specific (i.e. cirrhosis for hepatic cancer) researched. The maps will also include filters that allow users to select records based on additional features, such as the use of artificial intelligence in the content of the papers. Furthermore, the EGM, which will be freely available online, will enable users to view and export selected bibliographic references and their abstracts. An example of these interactive maps with dummy data is provided in Additional file 5.

Finally, we will elaborate on two scientific reports for PROPHET. The main report, which will follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) recommendations, will summarise the results of the three scoping reviews, will provide a general and global interpretation of the results and will comment on their implication for the SRIA, and will discuss the limitations of the process. The second report will present the specific methodology for the dynamic maps.

This protocol summarises the procedure to carry out three parallel rapid scoping reviews to provide an overview of the available research and gaps in the literature on biomarkers for personalised primary and secondary prevention for the three most common chronic disease groups: cancer, CVD and neurodegenerative diseases. The result will be a common report for the three scoping reviews and the online publication of interactive evidence gap maps to facilitate data visualisation.

This work will be complemented, in a further step of the PROPHET project, by a subsequent mapping report on the scientific evidence for the clinical utility of biomarkers. Both reports are part of an overall mapping effort to characterise the current knowledge and environment around personalised preventive medicine. In this context, PROPHET will also map personalised prevention research programs, as well as bottlenecks and challenges in the adoption of personalised preventive approaches or in the involvement of citizens, patients, health professionals and policy-makers in personalised prevention. The overall results will contribute to the development of the SRIA concept paper, which will help define future priorities for personalised prevention research in the European Union.

In regard to this protocol, one of the strengths of this approach is that it can be applied in the three scoping reviews. This will improve the consistency and comparability of the results between them, allowing for better leveraging of efforts; it also will facilitate the coordination among the staff conducting the different reviews and will allow them to discuss them together, providing a more global perspective as needed for the SRIA. In addition, the collaboration of researchers with different backgrounds, the inclusion of librarians in the research team, and the specific software tools used have helped us to guarantee the quality of the work and have shortened the time invested in defining the final version of this protocol. Another strength is that we have conducted a pilot study to test and refine the search strategy, selection criteria and data extraction sheet. In addition, the selection of the platform of access to the bibliographic databases has been decided after a previous evaluation process (Ovid-MEDLINE versus PubMed MEDLINE, Ovid-Embase versus Elsevier-Embase, etc.).

Only 10% of the papers will undergo screening by two reviewers, and if time permits, we will conduct kappa statistics to assess reviewer agreement during the screening phases. Additionally, ongoing communication and the exchange and discussion of uncertainties will ensure a high level of consensus in the review process.

The main limitation of this work is the very broad field it covers: personalised prevention in all chronic diseases; however, we have tried to maintain decisions to limit it to the chronic diseases with the greatest impact on the population and in the last 3 years, making a rapid scoping review due to time constraints following recommendations from the Cochrane Rapid Review Methods Group [ 24 ]; however, as our aim is to identify gaps in the literature in an area of growing interest (personalisation and prevention), we believe that the records retrieved will provide a solid foundation for evaluating available literature. Additionally, systematic reviews, which may encompass studies predating 2020, have the potential to provide valuable insights beyond the temporal constraints of our search.

Thus, this protocol reflects the decisions set by the PROPHET's timetable, without losing the quality and rigour of the work. In addition, the data extraction phase will be done by two reviewers in 100% of the papers to ensure the consistency of the extracted data. Lastly, extending beyond these three scoping reviews, the primary challenge resides in amalgamating their findings with those from numerous other reviews within the project, ultimately producing a cohesive concept paper in the Strategy Research and Innovation Agenda (SRIA) for the European Union, firmly rooted in evidence-based conclusions.

Council of European Union. Council conclusions on personalised medicine for patients (2015/C 421/03). Brussels: European Union; 2015 dic. Report No.: (2015/C 421/03). Disponible en: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52015XG1217(01)&from=FR .

Goetz LH, Schork NJ. Personalized medicine: motivation, challenges, and progress. Fertil Steril. 2018;109(6):952–63.

Article   PubMed   PubMed Central   Google Scholar  

FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) Resource. Silver Spring (MD): Food and Drug Administration (US); 2016 [citado 3 de febrero de 2023]. Disponible en: http://www.ncbi.nlm.nih.gov/books/NBK326791/ .

Porta M, Greenland S, Hernán M, dos Silva I S, Last JM. International Epidemiological Association, editores. A dictionary of epidemiology. 6th ed. Oxford: Oxford Univ. Press; 2014. p. 343.

Google Scholar  

PROPHET. Project kick-off meeting. Rome. 2022.

Roth GA, Mensah GA, Johnson CO, Addolorato G, Ammirati E, Baddour LM, et al. Global burden of cardiovascular diseases and risk factors, 1990–2019. J Am College Cardiol. 2020;76(25):2982–3021.

Article   Google Scholar  

GBD 2019 Cancer Collaboration, Kocarnik JM, Compton K, Dean FE, Fu W, Gaw BL, et al. Cancer incidence, mortality, years of life lost, years lived with disability, and disability-adjusted life years for 29 cancer groups from 2010 to 2019: a systematic analysis for the global burden of disease study 2019. JAMA Oncol. 2022;8(3):420.

Feigin VL, Vos T, Nichols E, Owolabi MO, Carroll WM, Dichgans M, et al. The global burden of neurological disorders: translating evidence into policy. The Lancet Neurology. 2020;19(3):255–65.

Article   PubMed   Google Scholar  

GBD 2019 Collaborators, Nichols E, Abd‐Allah F, Abdoli A, Abosetugn AE, Abrha WA, et al. Global mortality from dementia: Application of a new method and results from the Global Burden of Disease Study 2019. A&D Transl Res & Clin Interv. 2021;7(1). Disponible en: https://onlinelibrary.wiley.com/doi/10.1002/trc2.12200 . [citado 7 de febrero de 2023].

Eurostat. ec.europa.eu. Self-perceived health statistics. European health interview survey (EHIS). 2022. Disponible en: https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Self-perceived_health_statistics . [citado 7 de febrero de 2023].

OECD/European Union. Health at a Glance: Europe 2022: State of Health in the EU Cycle. Paris: OECD Publishing; 2022. Disponible en: https://www.oecd-ilibrary.org/social-issues-migration-health/health-at-a-glance-europe-2022_507433b0-en .

Boccia S, Pastorino R, Ricciardi W, Ádány R, Barnhoorn F, Boffetta P, et al. How to integrate personalized medicine into prevention? Recommendations from the Personalized Prevention of Chronic Diseases (PRECeDI) Consortium. Public Health Genomics. 2019;22(5–6):208–14.

Visseren FLJ, Mach F, Smulders YM, Carballo D, Koskinas KC, Bäck M, et al. 2021 ESC Guidelines on cardiovascular disease prevention in clinical practice. Eur Heart J. 2021;42(34):3227–337.

World Health Organization. Global action plan on the public health response to dementia 2017–2025. Geneva: WHO Document Production Services; 2017. p. 27.

Norton S, Matthews FE, Barnes DE, Yaffe K, Brayne C. Potential for primary prevention of Alzheimer’s disease: an analysis of population-based data. Lancet Neurol. 2014;13(8):788–94.

Mentis AFA, Dardiotis E, Efthymiou V, Chrousos GP. Non-genetic risk and protective factors and biomarkers for neurological disorders: a meta-umbrella systematic review of umbrella reviews. BMC Med. 2021;19(1):6.

Schüz J, Espina C, Villain P, Herrero R, Leon ME, Minozzi S, et al. European Code against Cancer 4th Edition: 12 ways to reduce your cancer risk. Cancer Epidemiol. 2015;39:S1-10.

Tricco AC, Langlois EtienneV, Straus SE, Alliance for Health Policy and Systems Research, World Health Organization. Rapid reviews to strengthen health policy and systems: a practical guide. Geneva: World Health Organization; 2017. Disponible en: https://apps.who.int/iris/handle/10665/258698 . [citado 3 de febrero de 2023].

White H, Albers B, Gaarder M, Kornør H, Littell J, Marshall Z, et al. Guidance for producing a Campbell evidence and gap map. Campbell Systematic Reviews. 2020;16(4). Disponible en: https://onlinelibrary.wiley.com/doi/10.1002/cl2.1125 . [citado 3 de febrero de 2023].

Aromataris E, Munn Z. editores. JBI: JBI Manual for Evidence Synthesis; 2020.

Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18(10):2119–26.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73.

OSF. Open Science Framework webpage. Disponible en: https://osf.io/ . [citado 8 de febrero de 2023].

Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, et al. Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. Journal Clin Epidemiol. 2021;130:13–22.

Leon ME, Peruga A, McNeill A, Kralikova E, Guha N, Minozzi S, et al. European code against cancer, 4th edition: tobacco and cancer. Cancer Epidemiology. 2015;39:S20-33.

Anderson AS, Key TJ, Norat T, Scoccianti C, Cecchini M, Berrino F, et al. European code against cancer 4th edition: obesity, body fatness and cancer. Cancer Epidemiology. 2015;39:S34-45.

Barone BB, Yeh HC, Snyder CF, Peairs KS, Stein KB, Derr RL, et al. Long-term all-cause mortality in cancer patients with preexisting diabetes mellitus: a systematic review and meta-analysis. JAMA. 2008;300(23):2754–64.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Barone BB, Yeh HC, Snyder CF, Peairs KS, Stein KB, Derr RL, et al. Postoperative mortality in cancer patients with preexisting diabetes: systematic review and meta-analysis. Diabetes Care. 2010;33(4):931–9.

Noto H, Tsujimoto T, Sasazuki T, Noda M. Significantly increased risk of cancer in patients with diabetes mellitus: a systematic review and meta-analysis. Endocr Pract. 2011;17(4):616–28.

Villain P, Gonzalez P, Almonte M, Franceschi S, Dillner J, Anttila A, et al. European code against cancer 4th edition: infections and cancer. Cancer Epidemiology. 2015;39:S120-38.

Scoccianti C, Cecchini M, Anderson AS, Berrino F, Boutron-Ruault MC, Espina C, et al. European Code against Cancer 4th Edition: Alcohol drinking and cancer. Cancer Epidemiology. 2016;45:181–8.

El-Serag HB. Epidemiology of viral hepatitis and hepatocellular carcinoma. Gastroenterology. 2012;142(6):1264-1273.e1.

Li XY, Zhang M, Xu W, Li JQ, Cao XP, Yu JT, et al. Midlife modifiable risk factors for dementia: a systematic review and meta-analysis of 34 prospective cohort studies. CAR. 2020;16(14):1254–68.

Ford E, Greenslade N, Paudyal P, Bremner S, Smith HE, Banerjee S, et al. Predicting dementia from primary care records: a systematic review and meta-analysis Forloni G, editor. PLoS ONE. 2018;13(3):e0194735.

Xu W, Tan L, Wang HF, Jiang T, Tan MS, Tan L, et al. Meta-analysis of modifiable risk factors for Alzheimer’s disease. J Neurol Neurosurg Psychiatry. 2015;86(12):1299–306.

PubMed   Google Scholar  

Guo Y, Xu W, Liu FT, Li JQ, Cao XP, Tan L, et al. Modifiable risk factors for cognitive impairment in Parkinson’s disease: A systematic review and meta-analysis of prospective cohort studies. Mov Disord. 2019;34(6):876–83.

Jiménez-Jiménez FJ, Alonso-Navarro H, García-Martín E, Agúndez JAG. Alcohol consumption and risk for Parkinson’s disease: a systematic review and meta-analysis. J Neurol agosto de. 2019;266(8):1821–34.

ECIS European Cancer Information System. Data explorer | ECIS. 2023. Estimates of cancer incidence and mortality in 2020 for all cancer sites. Disponible en: https://ecis.jrc.ec.europa.eu/explorer.php?$0-0$1-AE27$2-All$4-2$3-All$6-0,85$5-2020,2020$7-7,8$CEstByCancer$X0_8-3$CEstRelativeCanc$X1_8-3$X1_9-AE27$CEstBySexByCancer$X2_8-3$X2_-1-1 . [citado 22 de febrero de 2023].

Bacigalupo I, Mayer F, Lacorte E, Di Pucchio A, Marzolini F, Canevelli M, et al. A systematic review and meta-analysis on the prevalence of dementia in Europe: estimates from the highest-quality studies adopting the DSM IV diagnostic criteria Bruni AC, editor. JAD. 2018;66(4):1471–81.

Barceló MA, Povedano M, Vázquez-Costa JF, Franquet Á, Solans M, Saez M. Estimation of the prevalence and incidence of motor neuron diseases in two Spanish regions: Catalonia and Valencia. Sci Rep. 2021;11(1):6207.

Ng L, Khan F, Young CA, Galea M. Symptomatic treatments for amyotrophic lateral sclerosis/motor neuron disease. Cochrane Neuromuscular Group, editor. Cochrane Database of Systematic Reviews. 2017;2017(1). Disponible en: http://doi.wiley.com/10.1002/14651858.CD011776.pub2 . [citado 13 de febrero de 2023].

Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation; 2023. Disponible en: https://www.covidence.org .

Centre for Disease Control and Prevention. Public Health Genomics and Precision Health Knowledge Base (v8.4). 2023. Disponible en: https://phgkb.cdc.gov/PHGKB/specificPHGKB.action?action=about .

Digital Solution Foundry and EPPI Centre. EPPI Centre. UCL Social Research Institute: University College London; 2022.

Download references

Acknowledgements

We are grateful for the library support received from Teresa Carretero (Instituto de Salud Carlos III, ISCIII) and, from Concepción Campos-Asensio (Hospital Universitario de Getafe, Comité ejecutivo BiblioMadSalud) for the seminar on the Scoping Reviews methodology and for their continuous teachings through their social networks.

Also, we would like to thank Dr. Héctor Bueno (Centro Nacional de Investigaciones Cardiovasculares (CNIC), Hospital Universitario 12 de Octubre) and Dr. Pascual Sánchez (Fundación Centro de Investigación de Enfermedades Neurológicas (CIEN)) for their advice in their fields of expertise.

The PROPHET project has received funding from the European Union’s Horizon Europe research and innovation program under grant agreement no. 101057721. UK participation in Horizon Europe Project PROPHET is supported by UKRI grant number 10040946 (Foundation for Genomics & Population Health).

Author information

Plans-Beriso E and Babb-de-Villiers C contributed equally to this work.

Kroese M and Pérez-Gómez B contributed equally to this work.

Authors and Affiliations

Department of Epidemiology of Chronic Diseases, National Centre for Epidemiology, Instituto de Salud Carlos III, Madrid, Spain

E Plans-Beriso, C Barahona-López, P Diez-Echave, O R Hernández, E García-Ovejero, O Craciun, P Fernández-Navarro, N Fernández-Larrea, E García-Esquinas, M Pollan-Santamaria & B Pérez-Gómez

CIBER of Epidemiology and Public Health (CIBERESP), Madrid, Spain

E Plans-Beriso, D Petrova, C Barahona-López, P Diez-Echave, O R Hernández, N F Fernández-Martínez, P Fernández-Navarro, N Fernández-Larrea, E García-Esquinas, V Moreno, F Rodríguez-Artalejo, M J Sánchez, M Pollan-Santamaria & B Pérez-Gómez

PHG Foundation, University of Cambridge, Cambridge, UK

C Babb-de-Villiers, H Turner, L Blackburn & M Kroese

Instituto de Investigación Biosanitaria Ibs. GRANADA, Granada, Spain

D Petrova, N F Fernández-Martínez & M J Sánchez

Escuela Andaluza de Salud Pública (EASP), Granada, Spain

Cambridge University Medical Library, Cambridge, UK

National Library of Health Sciences, Instituto de Salud Carlos III, Madrid, Spain

V Jiménez-Planet

Oncology Data Analytics Program, Catalan Institute of Oncology (ICO), L’Hospitalet de Llobregat, Barcelona, 08908, Spain

Colorectal Cancer Group, ONCOBELL Program, Institut de Recerca Biomedica de Bellvitge (IDIBELL), L’Hospitalet de Llobregat, Barcelona, 08908, Spain

Department of Preventive Medicine and Public Health, Universidad Autónoma de Madrid, Madrid, Spain

F Rodríguez-Artalejo

IMDEA-Food Institute, CEI UAM+CSIC, Madrid, Spain

You can also search for this author in PubMed   Google Scholar

Contributions

BPG and MK supervised and directed the project. EPB and CBV coordinated and managed the development of the project. CBL, PDE, ORH, CBV and EPB developed the search strategy. All authors reviewed the content, commented on the methods, provided feedback, contributed to drafts and approved the final manuscript.

Corresponding author

Correspondence to E Plans-Beriso .

Ethics declarations

Competing interests.

There are no conflicts of interest in this project.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: glossary., additional file 2: glossary of biomarkers that may define high risk groups., additional file 3: search strategy., additional file 4: data extraction sheet., additional file 5: example of interactive maps in cancer and primary prevention., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Plans-Beriso, E., Babb-de-Villiers, C., Petrova, D. et al. Biomarkers for personalised prevention of chronic diseases: a common protocol for three rapid scoping reviews. Syst Rev 13 , 147 (2024). https://doi.org/10.1186/s13643-024-02554-9

Download citation

Received : 19 October 2023

Accepted : 03 May 2024

Published : 01 June 2024

DOI : https://doi.org/10.1186/s13643-024-02554-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Personalised prevention
• Precision Medicine
• Precision prevention
• Cardiovascular diseases
• Chronic diseases

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Family Med Prim Care
• v.2(1); Jan-Mar 2013

Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare

S. gopalakrishnan.

Department of Community Medicine, SRM Medical College, Hospital and Research Centre, Kattankulathur, Tamil Nadu, India

P. Ganeshkumar

Healthcare decisions for individual patients and for public health policies should be informed by the best available research evidence. The practice of evidence-based medicine is the integration of individual clinical expertise with the best available external clinical evidence from systematic research and patient's values and expectations. Primary care physicians need evidence for both clinical practice and for public health decision making. The evidence comes from good reviews which is a state-of-the-art synthesis of current evidence on a given research question. Given the explosion of medical literature, and the fact that time is always scarce, review articles play a vital role in decision making in evidence-based medical practice. Given that most clinicians and public health professionals do not have the time to track down all the original articles, critically read them, and obtain the evidence they need for their questions, systematic reviews and clinical practice guidelines may be their best source of evidence. Systematic reviews aim to identify, evaluate, and summarize the findings of all relevant individual studies over a health-related issue, thereby making the available evidence more accessible to decision makers. The objective of this article is to introduce the primary care physicians about the concept of systematic reviews and meta-analysis, outlining why they are important, describing their methods and terminologies used, and thereby helping them with the skills to recognize and understand a reliable review which will be helpful for their day-to-day clinical practice and research activities.

Introduction

Evidence-based healthcare is the integration of best research evidence with clinical expertise and patient values. Green denotes, “Using evidence from reliable research, to inform healthcare decisions, has the potential to ensure best practice and reduce variations in healthcare delivery.” However, incorporating research into practice is time consuming, and so we need methods of facilitating easy access to evidence for busy clinicians.[ 1 ] Ganeshkumar et al . mentioned that nearly half of the private practitioners in India were consulting more than 4 h per day in a locality,[ 2 ] which explains the difficulty of them in spending time in searching evidence during consultation. Ideally, clinical decision making ought to be based on the latest evidence available. However, to keep abreast with the continuously increasing number of publications in health research, a primary healthcare professional would need to read an insurmountable number of articles every day, covered in more than 13 million references and over 4800 biomedical and health journals in Medline alone. With the view to address this challenge, the systematic review method was developed. Systematic reviews aim to inform and facilitate this process through research synthesis of multiple studies, enabling increased and efficient access to evidence.[ 1 , 3 , 4 ]

Systematic reviews and meta-analyses have become increasingly important in healthcare settings. Clinicians read them to keep up-to-date with their field and they are often used as a starting point for developing clinical practice guidelines. Granting agencies may require a systematic review to ensure there is justification for further research and some healthcare journals are moving in this direction.[ 5 ]

This article is intended to provide an easy guide to understand the concept of systematic reviews and meta-analysis, which has been prepared with the aim of capacity building for general practitioners and other primary healthcare professionals in research methodology and day-to-day clinical practice.

The purpose of this article is to introduce readers to:

• The two approaches of evaluating all the available evidence on an issue i.e., systematic reviews and meta-analysis,
• Discuss the steps in doing a systematic review,
• Introduce the terms used in systematic reviews and meta-analysis,
• Interpret results of a meta-analysis, and
• The advantages and disadvantages of systematic review and meta-analysis.

Application

What is the effect of antiviral treatment in dengue fever? Most often a primary care physician needs to know convincing answers to questions like this in a primary care setting.

To find out the solutions or answers to a clinical question like this, one has to refer textbooks, ask a colleague, or search electronic database for reports of clinical trials. Doctors need reliable information on such problems and on the effectiveness of large number of therapeutic interventions, but the information sources are too many, i.e., nearly 20,000 journals publishing 2 million articles per year with unclear or confusing results. Because no study, regardless of its type, should be interpreted in isolation, a systematic review is generally the best form of evidence.[ 6 ] So, the preferred method is a good summary of research reports, i.e., systematic reviews and meta-analysis, which will give evidence-based answers to clinical situations.

There are two fundamental categories of research: Primary research and secondary research. Primary research is collecting data directly from patients or population, while secondary research is the analysis of data already collected through primary research. A review is an article that summarizes a number of primary studies and may draw conclusions on the topic of interest which can be traditional (unsystematic) or systematic.

Terminologies

Systematic review.

A systematic review is a summary of the medical literature that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and random errors.[ 7 ] To this end, systematic reviews may or may not include a statistical synthesis called meta-analysis, depending on whether the studies are similar enough so that combining their results is meaningful.[ 8 ] Systematic reviews are often called overviews.

The evidence-based practitioner, David Sackett, defines the following terminologies.[ 3 ]

• Review: The general term for all attempts to synthesize the results and conclusions of two or more publications on a given topic.
• Overview: When a review strives to comprehensively identify and track down all the literature on a given topic (also called “systematic literature review”).
• Meta-analysis: A specific statistical strategy for assembling the results of several studies into a single estimate.

Systematic reviews adhere to a strict scientific design based on explicit, pre-specified, and reproducible methods. Because of this, when carried out well, they provide reliable estimates about the effects of interventions so that conclusions are defensible. Systematic reviews can also demonstrate where knowledge is lacking. This can then be used to guide future research. Systematic reviews are usually carried out in the areas of clinical tests (diagnostic, screening, and prognostic), public health interventions, adverse (harm) effects, economic (cost) evaluations, and how and why interventions work.[ 9 ]

Cochrane reviews

Cochrane reviews are systematic reviews undertaken by members of the Cochrane Collaboration which is an international not-for-profit organization that aims to help people to make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

Cochrane Primary Health Care Field is a systematic review of primary healthcare research on prevention, treatment, rehabilitation, and diagnostic test accuracy. The overall aim and mission of the Primary Health Care Field is to promote the quality, quantity, dissemination, accessibility, applicability, and impact of Cochrane systematic reviews relevant to people who work in primary care and to ensure proper representation in the interests of primary care clinicians and consumers in Cochrane reviews and review groups, and in other entities. This field would serve to coordinate and promote the mission of the Cochrane Collaboration within the primary healthcare disciplines, as well as ensuring that primary care perspectives are adequately represented within the Collaboration.[ 10 ]

Meta-analysis

A meta-analysis is the combination of data from several independent primary studies that address the same question to produce a single estimate like the effect of treatment or risk factor. It is the statistical analysis of a large collection of analysis and results from individual studies for the purpose of integrating the findings.[ 11 ] The term meta-analysis has been used to denote the full range of quantitative methods for research reviews.[ 12 ] Meta-analyses are studies of studies.[ 13 ] Meta-analysis provides a logical framework to a research review where similar measures from comparable studies are listed systematically and the available effect measures are combined wherever possible.[ 14 ]

The fundamental rationale of meta-analysis is that it reduces the quantity of data by summarizing data from multiple resources and helps to plan research as well as to frame guidelines. It also helps to make efficient use of existing data, ensuring generalizability, helping to check consistency of relationships, explaining data inconsistency, and quantifies the data. It helps to improve the precision in estimating the risk by using explicit methods.

Therefore, “systematic review” will refer to the entire process of collecting, reviewing, and presenting all available evidence, while the term “meta-analysis” will refer to the statistical technique involved in extracting and combining data to produce a summary result.[ 15 ]

Steps in doing systematic reviews/meta-analysis

Following are the six fundamental essential steps while doing systematic review and meta-analysis.[ 16 ]

Define the question

This is the most important part of systematic reviews/meta-analysis. The research question for the systematic reviews may be related to a major public health problem or a controversial clinical situation which requires acceptable intervention as a possible solution to the present healthcare need of the community. This step is most important since the remaining steps will be based on this.

Reviewing the literature

This can be done by going through scientific resources such as electronic database, controlled clinical trials registers, other biomedical databases, non-English literatures, “gray literatures” (thesis, internal reports, non–peer-reviewed journals, pharmaceutical industry files), references listed in primary sources, raw data from published trials and other unpublished sources known to experts in the field. Among the available electronic scientific database, the popular ones are PUBMED, MEDLINE, and EMBASE.

Sift the studies to select relevant ones

To select the relevant studies from the searches, we need to sift through the studies thus identified. The first sift is pre-screening, i.e., to decide which studies to retrieve in full, and the second sift is selection which is to look again at these studies and decide which are to be included in the review. The next step is selecting the eligible studies based on similar study designs, year of publication, language, choice among multiple articles, sample size or follow-up issues, similarity of exposure, and or treatment and completeness of information.

It is necessary to ensure that the sifting includes all relevant studies like the unpublished studies (desk drawer problem), studies which came with negative conclusions or were published in non-English journals, and studies with small sample size.

Assess the quality of studies

The steps undertaken in evaluating the study quality are early definition of study quality and criteria, setting up a good scoring system, developing a standard form for assessment, calculating quality for each study, and finally using this for sensitivity analysis.

For example, the quality of a randomized controlled trial can be assessed by finding out the answers to the following questions:

• Was the assignment to the treatment groups really random?
• Was the treatment allocation concealed?
• Were the groups similar at baseline in terms of prognostic factors?
• Were the eligibility criteria specified?
• Were the assessors, the care provider, and the patient blinded?
• Were the point estimates and measure of variability presented for the primary outcome measure?
• Did the analyses include intention-to-treat analysis?

Calculate the outcome measures of each study and combine them

We need a standard measure of outcome which can be applied to each study on the basis of its effect size. Based on their type of outcome, following are the measures of outcome: Studies with binary outcomes (cured/not cured) have odds ratio, risk ratio; studies with continuous outcomes (blood pressure) have means, difference in means, standardized difference in means (effect sizes); and survival or time-to-event data have hazard ratios.

Combining studies

Homogeneity of different studies can be estimated at a glance from a forest plot (explained below). For example, if the lower confidence interval of every trial is below the upper of all the others, i.e., the lines all overlap to some extent, then the trials are homogeneous. If some lines do not overlap at all, these trials may be said to be heterogeneous.

The definitive test for assessing the heterogeneity of studies is a variant of Chi-square test (Mantel–Haenszel test). The final step is calculating the common estimate and its confidence interval with the original data or with the summary statistics from all the studies. The best estimate of treatment effect can be derived from the weighted summary statistics of all studies which will be based on weighting to sample size, standard errors, and other summary statistics. Log scale is used to combine the data to estimate the weighting.

Interpret results: Graph

The results of a meta-analysis are usually presented as a graph called forest plot because the typical forest plots appear as forest of lines. It provides a simple visual presentation of individual studies that went into the meta-analysis at a glance. It shows the variation between the studies and an estimate of the overall result of all the studies together.

Forest plot

Meta-analysis graphs can principally be divided into six columns [ Figure 1 ]. Individual study results are displayed in rows. The first column (“study”) lists the individual study IDs included in the meta-analysis; usually the first author and year are displayed. The second column relates to the intervention groups and the third column to the control groups. The fourth column visually displays the study results. The line in the middle is called “the line of no effect.” The weight (in %) in the fifth column indicates the weighting or influence of the study on the overall results of the meta-analysis of all included studies. The higher the percentage weight, the bigger the box, the more influence the study has on the overall results. The sixth column gives the numerical results for each study (e.g., odds ratio or relative risk and 95% confidence interval), which are identical to the graphical display in the fourth column. The diamond in the last row of the graph illustrates the overall result of the meta-analysis.[ 4 ]

Interpretation of meta-analysis[ 4 ]

Thus, the horizontal lines represent individual studies. Length of line is the confidence interval (usually 95%), squares on the line represent effect size (risk ratio) for the study, with area of the square being the study size (proportional to weight given) and position as point estimate (relative risk) of the study.[ 7 ]

For example, the forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults is shown in Figure 2 .[ 17 ]

Forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults[ 17 ]

The overall effect is shown as diamond where the position toward the center represents pooled point estimate, the width represents estimated 95% confidence interval for all studies, and the black plain line vertically in the middle of plot is the “line of no effect” (e.g., relative risk = 1).

Therefore, when examining the results of a systematic reviews/meta-analysis, the following questions should be kept in mind:

• Heterogeneity among studies may make any pooled estimate meaningless.
• The quality of a meta-analysis cannot be any better than the quality of the studies it is summarizing.
• An incomplete search of the literature can bias the findings of a meta-analysis.
• Make sure that the meta-analysis quantifies the size of the effect in units that you can understand.

Subgroup analysis and sensitivity analysis

Subgroup analysis looks at the results of different subgroups of trials, e.g., by considering trials on adults and children separately. This should be planned at the protocol stage itself which is based on good scientific reasoning and is to be kept to a minimum.

Sensitivity analysis is used to determine how results of a systematic review/meta-analysis change by fiddling with data, for example, what is the implication if the exclusion criteria or excluded unpublished studies or weightings are assigned differently. Thus, after the analysis, if changing makes little or no difference to the overall results, the reviewer's conclusions are robust. If the key findings disappear, then the conclusions need to be expressed more cautiously.

Advantages of Systematic Reviews

Systematic reviews have specific advantages because of using explicit methods which limit bias, draw reliable and accurate conclusions, easily deliver required information to healthcare providers, researchers, and policymakers, help to reduce the time delay in the research discoveries to implementation, improve the generalizability and consistency of results, generation of new hypotheses about subgroups of the study population, and overall they increase precision of the results.[ 18 ]

Limitations in Systematic Reviews/Meta-analysis

As with all research, the value of a systematic review depends on what was done, what was found, and the clarity of reporting. As with other publications, the reporting quality of systematic reviews varies, limiting readers’ ability to assess the strengths and weaknesses of those reviews.[ 5 ]

Even though systematic review and meta-analysis are considered the best evidence for getting a definitive answer to a research question, there are certain inherent flaws associated with it, such as the location and selection of studies, heterogeneity, loss of information on important outcomes, inappropriate subgroup analyses, conflict with new experimental data, and duplication of publication.

Publication Bias

Publication bias results in it being easier to find studies with a “positive” result.[ 19 ] This occurs particularly due to inappropriate sifting of the studies where there is always a tendency towards the studies with positive (significant) outcomes. This effect occurs more commonly in systematic reviews/meta-analysis which need to be eliminated.

The quality of reporting of systematic reviews is still not optimal. In a recent review of 300 systematic reviews, few authors reported assessing possible publication bias even though there is overwhelming evidence both for its existence and its impact on the results of systematic reviews. Even when the possibility of publication bias is assessed, there is no guarantee that systematic reviewers have assessed or interpreted it appropriately.[ 20 ]

To overcome certain limitations mentioned above, the Cochrane reviews are currently reported in a format where at the end of every review, findings are summarized in the author's point of view and also give an overall picture of the outcome by means of plain language summary. This is found to be much helpful to understand the existing evidence about the topic more easily by the reader.

A systematic review is an overview of primary studies which contains an explicit statement of objectives, materials, and methods, and has been conducted according to explicit and reproducible methodology. A meta-analysis is a mathematical synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way. Although meta-analysis can increase the precision of a result, it is important to ensure that the methods used for the reviews were valid and reliable.

High-quality systematic reviews and meta-analyses take great care to find all relevant studies, critically assess each study, synthesize the findings from individual studies in an unbiased manner, and present balanced important summary of findings with due consideration of any flaws in the evidence. Systematic review and meta-analysis is a way of summarizing research evidence, which is generally the best form of evidence, and hence positioned at the top of the hierarchy of evidence.

Systematic reviews can be very useful decision-making tools for primary care/family physicians. They objectively summarize large amounts of information, identifying gaps in medical research, and identifying beneficial or harmful interventions which will be useful for clinicians, researchers, and even for public and policymakers.

Source of Support: Nil

Conflict of Interest: None declared.

Market research ethics: New practices but no new ideas

• Theory/Conceptual
• Open access
• Published: 07 June 2024

Cite this article

You have full access to this open access article

• Robert Cluley   ORCID: orcid.org/0000-0002-0827-4538 1 &
• William Green   ORCID: orcid.org/0000-0003-4161-4490 2  

The ethical issues involved with marketing research are receiving increased public scrutiny, prompting calls for marketing scholars and research practitioners to revisit the issue. To support researchers and practitioners, this paper provides a systematic scoping review of research on the ethics of market research developed across a range of literatures ( N  = 134). It demonstrates that, over 70 years, marketing scholars have explored the ethics of market research from normative , descriptive , theoretical and technical approaches. But, while marketing scholars were once at the forefront of theorising the ethics of marketing research, the field is increasingly fragmented and specialized. The result is that, following a series of theoretical innovations in the 1980s, progress has all but ended. We ask why marketing scholars have turned away from the ethics of marketing research given the importance of the topic in practice.

Avoid common mistakes on your manuscript.

Introduction

Market research “continually raises ethical concerns” (Castleberry et al., 1993 : 39). Reflecting this, the ethics of marketing research has attracted “special attention” from marketing scholars (Skinner et al., 1988 : 212). As Segal and Giacobbe ( 2007 : 35) explain, this is “largely due to the research industry’s self-interest”. Practitioners must “be at the forefront of being able to manage ethical issues’ to maintain research clients” confidence in the objectivity, independence, and rigour of market research results (Yallop & Mowatt, 2016 : 381) and “ensure participants” wellbeing and safety (Dodds et al., 2023 : 14).

Yet, this interest is decreasing. A decade ago, Aggarwal et al. argued that the “time has come” for marketing scholars “to revisit the issue of ethical perceptions in marketing research” ( 2012 :463). But, despite recent high-profile controversies involving consumer privacy (Nunan & Domenico, 2016 ), neuromarketing (Ulman et al., 2015 ), data trading, ad fraud (Cluley, 2019 ) and artificial intelligence (Nunan & Domenico, 2013 ), the call for further research remains largely unanswered within the marketing field. The ethics of commercial marketing research is increasingly treated as a technical specialism and discussed outside of marketing scholarship. Indeed, the last article addressing the ethics of commercial marketing research in the Journal of Marketing was Sparks and Hunt ( 1998 ).

This presents us with an increasingly fragmented and outdated research base. In response, in this paper we take stock of knowledge about marketing research ethics via a systematic literature review. Our objective is to provide a holistic framework to make sense of the literature, consider the need for fresh thinking and identify opportunities for theory development. We aim to encourage a debate about how to put marketing research ethics back on the agenda (Hunt, 2013 ).

We highlight four research themes from a systematically reviewed corpus ( N  = 134). The normative theme examines the boundaries of ethical behaviour with a focus on codes of ethics. The descriptive theme tracks practitioners’ attitudes towards controversial techniques. The theoretical theme engages with psychological thinking to model market researchers' ethical decision-making processes. The technical theme explores the ethical issues and conflicts inherent in market research techniques and tools, with a focus on new technologies. Overall, we find that the literature has moved towards technical discussions of specific ethical issues, typically published in specialist outlets.

The result is that theoretical progress has largely come to a standstill. This may be because business ethics and marketing scholars’ attention centres on consumption issues such as the significant body of research exploring the gap between consumers’ espoused ethical commitments and their actual behaviours in the market (Sun & Govind, 2022 ). It might be because discussions of ethics have been rebranded around sustainability, responsibility, and social justice. It may also reflect the facts that the market research industry “is increasingly defining itself in other ways” (Nunan, 2016 : 518). Much commercial market research is “not being created and run by people who call themselves ‘market researchers’” (Poynter, 2021 : 403–404). System designers and programmers who administer online data collection services might not think to consult codes of conduct written for marketing researchers.

There is a risk that in approaching ethical issues from a range of specialist perspectives, often divorced from core marketing concerns, such practitioners may ignore existing knowledge and miss readily discernible ethical pitfalls and vice versa. For instance, scholars are well-versed in debates about data fraud in their own research, meanwhile fraudulent practices in online data collection marketplaces have flourished, with little attempt to take lessons from academic marketing research into commercial settings. Through this article, then, we seek to contribute to the attempt to “resolve the perceived lack of relevance of scholarly marketing work” (Wieland et al., 2021 : 268).

The paper adopts a systematic scoping review approach to identify a comprehensive body of research that addresses the ethics of market research from a range of disciplines (Tranfield et al., 2003 ; Arksey & O'Malley, 2005 ). Although not guaranteed to identify all relevant work, as literature may be missing or excluded based on inclusion criteria, systematically reviewing literature has a number of advantages in comparison to other review methods. It promotes “reproducibility and transparency in the review process” (Domenico et al., 2021 : 330). Most importantly given our aim to support practice, evidence from systematic reviews has been shown to have a “real impact” with “practicing managers, policy-makers or researchers” (Macpherson & Jones, 2010 : 110).

To conduct our review, we follow Littell et al.’s ( 2008 ) five step protocol: identify a review panel, formulate the topic, determine the review characteristics, retrieve literature, and synthesize and report. This approach also reflects those defined for scoping reviews in the medical and health sciences (Arksey & O'Malley, 2005 ; Peters, et al., 2015 ). Our interest in the topic emerged through action research conducted by the authors with a large multinational market research company. This brought our attention to news stories about consumer privacy, data management and artificial intelligence. Following Rotfeld and Taylor’s ( 2009 ) call for marketing academics to move beyond headline stories, we formulated our concern with the ethics of market research. This led us to ask:

How has the marketing research ethics literature developed?

What are the key research issues in the marketing research ethics literature?

How does marketing research ethics relate to marketing practice, consumers and society?

Search strategy and article selection process

To ensure our review retrieved as many relevant sources as possible, we adopted a broad definition of commercial market research. We used ESOMAR's ( 2021 ) definition of commercial market research as the collecting, analyzing and interpreting “data to build information and knowledge that can be used to predict, for example, future events, actions or behaviours” for clients in consumer, industry, policy and civic sectors as the basis of our search criteria. To ensure an inclusive search strategy, we expanded the search terms to include 32 “descriptions of core business activity” of commercial market research reported by Nunan ( 2016 : 512). See Supplementary Table  1 for the results of the search queries and results.

We searched Web of Science for all dates until November 29th, 2022. Forty-eight duplicates were removed from the initial 769 articles. Article titles, keywords and abstracts were reviewed to ensure they met the definitions of commercial market research and included discussions of ethics. We assumed that if “ethic*” was mentioned in the title or abstract, ethics formed a significant part of the subsequent paper. In total, 134 articles were identified for full article review (Fig.  1 ). The complete bibliography is available in Supplementary Table 2 .

Systematic review flow diagram (adapted from Page et al., 2021 )

Analysis and synthesis

To scope the field, we conducted a thematic synthesis (Popay et al., 2006 ). Practically, the author/s read the papers in chronological order and engaged in a first round of descriptive coding to summarize the research topics. This settled on an analytical scheme comprising four themes, with each theme operating as an “ideal type” and, collectively, representing essential features of the literature on the ethics of marketing research. We proceeded to thematically code each text identified in the systematic review according to these four themes. We placed each text in what we judged to be the most applicable theme. We recognize that many papers speak to multiple themes and could therefore have been positioned in multiple themes. As we aim to expose broad trends in the literature and distinguish characteristics from the literature, we feel justified in our approach. The themes are illustrated in the “ Thematic analysis ” section.

Publication year

Figure  2 presents the frequency of published contributions to the field of marketing research ethics. The earliest contributions were published in the 1960s as the American Marketing Academy (AMA) Standards Committee was creating a code of ethics for market research practitioners (Blankenship, 1964 ; Twedt, 1963 ). The number of publications has subsequently increased on an upward trend as market research has expanded into new fields (Tybout & Zaltman, 1974 ) and adopted new technologies (Nunan & Di Domenico, 2013 ).

Publications by year

Journal orientation

As illustrated in Table  1 , the literature has moved from general marketing-oriented outlets such as the Journal of Marketing Research and Journal of Marketing towards business ethics-centred outlets such as Journal of Business Ethics and, more recently, to specialist outlets for neuroethicitians, statisticians and engineers. Despite publishing the earliest contributions, the Journal of Marketing has not published a paper on the ethics of marketing research since Sparks and Hunt ( 1998 ). Perhaps this explains Aggarwal et al.’s ( 2012 ) call for fresh perspectives and empirical contributions within marketing theory.

The scope of the marketing research ethics literature covers normative, descriptive, theoretical and technical contributions. The distinguishing features of themes are summarized in Table  2 .

The prevalence of the themes is represented in Table  3 . Starting in the 1960s, the normative theme established academic interest in the topic. In the early 1960s, a number of industry bodies were developing codes of conduct (AMA, American Association for Public Opinion Research and Consumer Psychology Division of the American Psychological Association) and this appears to have prompted discussions about the content of codes. Key contributions here emphasize the importance of understanding market research practitioners’ experiences and attitudes towards unethical practices. This resulted in a descriptive literature that gathered momentum in the 1970s through to the early 2000s. To explain differences in evidenced in practitioners’ attitudes and behaviours, from the 1980s researchers began to theorize market research practitioners’ ethical judgements using general ethical decision-making models and concepts. Thus, we saw the emergence of a theoretical turn in the literature. Most recently, the technical theme has explored the potential ethical issues inherent in individual market research techniques such as big data and neuromarketing.

Contribution of disciplines to themes

Table 4 illustrates the contribution of different facets of marketing scholarships to the development of each theme. It demonstrates that technical research has tended to be published in journals targeting specialist and  non-marketing audiences. General marketing outlets have been more focused on exploring the normative justification for ethical decisions and describing changes in practice.

Thematic analysis

The normative theme.

In line with Hunt and Vitell ( 1986 ), we use the term normative to refer to contributions that primarily define the boundaries of ethical behaviour for commercial marketing research practitioners. Such work seeks to establish the appropriate moral foundations, principles and norms for market research practice. It considers how to develop ethical cultures in market research settings and how to enforce moral standards—primarily, it focuses on codes of conduct.

The first article that emerged in our systematic review represents this approach. Working with the American Marketing Academy (AMA) Standards Committee, Twedt ( 1963 : 48) argues in favour of a deontological ethics, writing that “the public must be protected against market research abuses” because “the public has an inherent right to expect fair play.” At the same time, Twedt contests that practitioners have a teleological business incentive against unethical behaviour because they rely on public participation for their work. Consequently, Twedt calls for the AMA to improve ethical standards by encouraging a scientific approach among professionals. In Twedt’s view, such an approach would prioritize the virtue of truth and devalue deceptive practices.

Shortly after, Blankenship ( 1964 ) argues that the market research industry cannot rely solely on codes of ethics to enhance ethical behaviours. For Blankenship ( 1964 ), codes of conduct cannot address the most controversial research practices because they emerge from the competing demands placed on research practitioners and are not inherent in any particular research technique or technology. Virtues might be simple on paper but difficult to put into practice. For example, researchers’ relationships with clients “are fraught with pressures that might easily lead less scrupulous people to unethical behaviour” (Blankenship, 1964 : 27). Needing to keep a client’s business, a researcher “may find it difficult to tell his (sic) president or his (sic) customer that he (sic) is wrong” (Blankenship, 1964 : 29). In response, Blankenship argues that market research industries must find other ways to raise ethical conduct rather than expect codes to be sufficient.

A decade later, Tybout and Zaltman ( 1974 , 1975 ) argue that, as market research moves into new domains beyond consumer-settings, ethical values relating to “education, family planning and municipal government” will change the nature of ethical dilemmas in market research practice ( 1974 : 357). They also note that, as the range of techniques used in market research expands, new ethical questions are likely to emerge. Based on these observations, Tybout and Zaltman ( 1974 ) criticize the AMA code for focusing on survey techniques and commercial researcher-client relations. They argue for new standards relating to the ethics of care that researchers owe respondents. These include informed consent; the right to safe treatment; protecting anonymity and privacy; ensuring relief and support for potential stress and anxiety caused by research tasks; and honest research techniques as a default.

This prompted a direct response from Day ( 1975 ), who argues that ethical duties other than a commitment to scientific objectivity have no place in market research. However, later discussions accept that the boundaries of ethical practice must reflect the changing the nature of market research, methods and social conventions. For example, Sojka and Spangenberg observe that “the broadened domain of consumer research”, which introduces qualitative inquiry through consumer culture theory and consumer anthropology, “makes it necessary to look beyond marketing guidelines alone for prescriptions to ethical questions” ( 1994 : 393). Turning to cognizant literature, they emphasize two ethical duties: deception and confidentiality.

More recently, Alsmadi ( 2008 , 2010 ) argues for greater ethical consideration of respondents and wider society. Nunan and Di Domenico ( 2013 ) advocate changing industry guidelines to accommodate new ways of collecting, storing, and analysing big data. They argue that codes must include the right to be forgotten, data expiry and users’ and respondents’ maintaining ownership of their data and sharing any economic value it creates.

Many of these more recent contributions do not explicitly draw on ethical systems to justify the rights of respondents nor duties of market researchers—although, many implicitly adopt a deontological standpoint. But this is not to say that no contributors to the normative theme incorporate specific philosophical frameworks to argue for particular ethical imperatives. Boulanouar et al. ( 2017 ) address research ethics specific to Muslim women. Adler et al. ( 2022 ) draw explicitly on virtue ethics to derive four conditions (sophisticated, suppressed, contagion, and impeded) in which teams can overcome ethical dilemmas faced in market research practice. Michaelides and Gibbs ( 2006 : 50) use phenomenological hermeneutics “to enhance the self-awareness of market researchers”. They argue that marketing researchers' tendency to dehumanize individual participants does “not permit caring for the research subjects by the researcher who following the market system has no time to engage fully with them as people, and rather treats them as consumer objects” (Michaelides & Gibbs, 2006 : 50). Wilson et al. ( 2008 ) use a Rawlisian ethics of free will to examine neuromarketing.

In the 2000s, the normative literature took an empirical turn to explore the relationship between codes of ethics and research practice. Here, Giacobbe and Segal ( 2000 ) compare the ethical norms included in seven national codes of conduct against commercial research practitioners' attitudes in the USA and Canada. They identify notable differences and conclude that “market research codes of ethics do not reflect the perspectives of the very group they intend to direct” ( 2000 : 240). Developing this issue through semi-structured interviews with 29 research practitioners recruited through purposive sampling, Yallop and Mowatt find “that ethical codes do not appear to have a significant effect on ethical behaviour” ( 2016 : 392). They argue that the ineffectiveness of codes of ethics may be explained by the lack of awareness of their existence, a lack of familiarity of their content and a lack of proactive enforcement (see also Dutka, 1994 ). Consequently, Yallop and Mowatt conclude that “codes of ethics have the potential to influence behaviour, but only when there is adequate awareness of and familiarity with codes’ content, and when they are understood, used and enforced in organizations” ( 2016 : 395).

In summary, the normative theme explores the boundaries of ethical practices. Early contributions assumed that market researchers and industries have an implicit incentive to behave ethically because of their need to maintain public trust. Consequently, there was a focus on the content of codes of conduct and an assumption that practitioners would follow them. Later work, testing these assumptions empirically, suggests that codes of ethics have a marginal impact on practitioners’ ethical decisions and struggle to reflect the pertinent ethical decisions that practitioners face in their work. Substantively, then, a distinguishing feature of this theme is its focus on setting the boundaries of ethical behaviors for commercial market researchers. Here, much of the literature is anchored to notions of scientific objectivity as the key ethical value that should guide research practitioners’ behaviours. While later contributions test the assumptions in this theme empirically, much of the literature is discursive in nature—with key contributors making value-laden arguments for particular ethical values.

The descriptive theme

The descriptive theme focuses on commercial marketing research practitioners’ attitudes towards controversial practices. Crawford ( 1970 ) offers a founding contribution. He invites 700 market researchers to express their approval for 30 fictitious vignettes relating to research practice, the role of the marketing director and contemporary social concerns and asks participants if they judged the practices described in the vignettes to be ethical or not. He reveals consistent attitudes concerning the boundaries of acceptable practice, with senior managers and directors having even highest levels of agreement.

Crawford ( 1970 ) forms the basis of an ongoing empirical analysis of researchers' attitudes (Murphy & Laczniak, 1992 ). Building on this work and testing the key ethical dilemmas research practitioners experience through an open-ended survey item, Hunt et al. ( 1984 ) argue that the ethics of market research identified by practitioners revolve around: research integrity; treating outside clients fairly; research confidentiality; marketing mix and social issues; personnel issues; treating respondents fairly; treating others in one’s company fairly; interviewer dishonesty; gifts, bribes, and entertainment; treating suppliers fairly; legal issues; misuse of funds. They do “not find many researchers indicating fundamental conflicts involving the rights of respondents or subjects used in marketing research” ( 1984 : 318).

Akaah and Riordan ( 1989 ) explicitly replicate Crawford’s ( 1970 ) work. They find 15 significant shifts (out of 22) in comparison to the attitudes uncovered by Crawford ( 1970 ). Some of these relate to technical aspects of market research but, overall, suggest greater concern for social issues and a lessening of concern for ethical conflicts involving clients. Akaah and Riordan suggest these results “defy easy explanation” but most likely “derive from societal and industry changes” ( 1989 : 119). Aggarwal et al. ( 2012 ) retain six of Akaah and Riordan’s ( 1989 ) ethical scenarios and develop five new scenarios to reflect contemporary controversial practices. Summarizing their results, they conclude: “one trend that is quite clear is that disapproval for unethical research conduct has grown across the board in the last 20 years among professionals (both managers and researchers)” ( 2012 : 473). They then compare market researchers’ judgements against a general consumer sample. Here, Aggarwal et al. ( 2012 : 474) note: “Surprisingly, and with a couple of important exceptions, the general public is more tolerant of [unethical] conduct (as indicated by lower disapproval ratings) compared to the marketing professionals”.

Despite this consensus among descriptive studies, in the pages of a practitioner outlet, Whalen ( 1984 ) reports on a study of 500 research practitioners from blue-chip companies conducted by a private research firm that uncovers much higher levels of acceptance for unethical practices among commercial researchers than previously thought. Whalen ( 1984 ) highlights three potential reasons: business respondents are becoming more knowledgeable about market research techniques and are going to greater lengths to withhold proprietary information from researchers; deregulation removing reporting requirements making it harder for researchers to access business data; and increased competition in the market research industries leading to aggressive tactics. Whalen ( 1984 ) also reports an “ethical gap” whereby respondents indicate they believe their competitors are more willing to engage in unethical practices than they are. This perspective is developed by Pallister et al. ( 1999 )—who argue that unethical practices might be more profitable than ethical ones.

A second component of the descriptive theme examines the antecedents of market research practitioners’ attitudes. In a highly influential contribution, Hunt et al. ( 1984 ) argue that ethical issues that emerge in commercial marketing research practice are caused by conflicts between the values and responsibilities among clients, employing organizations, colleagues, self, industry competitors, and society. Informed by this theory, researchers have investigated cross-cultural differences between national industries. Here, Akaah notes that the majority of previous studies focus on the “viewpoints of ‘domestic’ (United States) marketing professionals” ( 1990 : 46), and, in response, compares Australian, British and American respondents using a similar survey instrument to Akaah and Riordan ( 1989 ). Akaah finds that practitioners from different industries “do not differ in terms of their research ethics attitudes” ( 1990 : 52).

Others explore the organizational antecedents of ethical attitudes and behaviours. Crawford ( 1970 ) had suggested a potential role modelling effect whereby senior executives could influence the ethical behaviour for junior researchers within an organization. This is confirmed by Segal and Giacombe ( 2007 ). Based on 86 survey responses from Australian practitioners, they find that practitioners attribute improved ethical behaviour to ethical leadership displayed by senior leaders in their firms, followed by immediate supervisors and, to a lesser degree, legal departments. However, Akaah and Riordan report that “code of ethics, organizational rank, and industry category lack significance as correlates of research ethics judgments” ( 1989 : 119). Ferrell and Skinner ( 1988 ) study the effect of organizational bureaucracy on market researcher ethics in three types of marketing organizations (data subcontractors, marketing research firms and corporate research departments). Drawing on 550 survey responses from members of a market research association, they find that increased levels of bureaucracy improve ethical behaviour in all settings.

Finally, studies have considered the role of demographics on researchers' ethical judgements. Ferrell and Skinner report that gender “is a significant predictor of ethical behavior” in data subcontractors and research firms but not corporate research departments ( 1988 : 106). This is confirmed by Kelley et al. ( 1990 ). Analyzing 602 surveys, they explore 10 ethical dilemmas and report significant differences between sex, age, education level, job title, and job tenure. Specifically, they tell us that “the perceptions of female researchers are more ethical than their male counterparts” and the “self-ratings of older marketing researchers were found to be significantly more ethical than their younger associates” (Kelley et al., 1990 : 687). Explaining their results, Kelley et al. ( 1990 : 688) ask: “Have younger, less experienced marketers been socialized into a different value system than were their older counterparts?”

In summary, the descriptive theme examines practitioners’ perceptions of controversial practices. It finds that variables such as gender, age, job tenure, level of seniority, organizational culture and type have significant effects on commercial marketing researchers' attitudes. Compared chronologically, it suggests that market researchers have become less tolerant of controversial practices and may even hold themselves to higher ethical standards than the general public. This theme can be distinguished from the normative theme, then, in the sense that contributors rarely advocate a specific ethical position. Rather than seeking to define the boundaries of acceptable behavior, contributors explore how market researchers themselves define ethical practices. Typically, this is achieved via survey methods and, in focusing on researchers' attitudes, tends to assume congruence between market researchers' attitudes and behaviours.

The theoretical theme

A third theme of research theorizes the ethical decision-making processes of market research practitioners. It focuses on why, when and how research practitioners engage in ethical decision making. The emergence of this stream of the literature can perhaps be explained as a consequence of the influential general theorizations of ethical decision-making among marketing practitioners (e.g. Hunt & Vitell, 1986 ).

Ferrell and Gresham ( 1985 ) exemplify this strand of thinking. Their contingency model suggests that individual marketers’ ethical decision-making will be determined by contextual and organizational factors. Contextual factors include the opportunity to make ethical decisions. Organizational factors include “managers' everyday performances in achieving company goals” ( 1985 : 90). To build this model, Ferrell and Gresham integrate several perspectives covering philosophies such as rights theory and sociological thinking including differential association theory and role set theory. They do not, however, empirically test the model. Their aim is, rather, to provide a “starting point for the development of a theory of ethical/unethical actions in organizational environments” ( 1985 : 87).

A prominent strand of the theoretical theme turns to psychological theories (Toy et al., 2001 ). For example, Castlebury et al. ( 1993 ) explore the moral reasoning research practitioners use to make ethical judgements. To do so, they compare the moral reasoning of advertising and market researchers and the general public. They find that research practitioners utilize “a level of moral reasoning that emphasizes social utility rather than personal reward” and that researchers’ moral reasoning follows that of the general population ( 1993 : 45).

Recognizing changes in the industry relations, a second strand of the theoretical theme models the relationships between the stakeholders within market research practice. Following Hunt et al. ( 1984 ), much of this work assumes that ethical dilemmas emerge from the competing interests of different stakeholders in market research (e.g. Malhotra & Peterson, 2001 ; Skinner et al., 1988 ). In this regard, Ferrell et al. ( 1998 ) use metaphors to theorize the similarities and differences between various stakeholders in relation to a code of ethics. They argue for a two-communities theorization of ethics in market research, suggesting that market research firms and data subcontractors should be considered a single community in terms of their ethical values and that this community should be distinguished from commercial research departments as a second community. It has less awareness of and enforcement of codes of ethics. Malhotra and Miller ( 1998 ) develop an integrated model setting out relations between the public, project commitments, employing firm, the industry, clients, the individual research, wider culture and respondents. These provide a context within which individual researchers work through ethical decision-making processes.

In summary, the theoretical theme assumes that commercial marketing research practice involves unique ethical decision-making not covered by general decision-making theories. Research reveals that research practitioners have consistent and distinct moral reasoning underpinning their ethical judgements and are uniquely sensitive to ethical issues and conflicts. Much like the descriptive theme, this theme is largely agnostic towards particular ethical values and ethical frameworks. It is distinguished, instead, by its psychological-focus. That is to say, rather than describing market researchers’ perceptions and attitudes, this theme aims to explain how researchers make ethical decisions. To achieve this aim, research tends to adopt a more experimental approach to test the effects of variables on researchers’ ethical decision-making.

The technical theme

The technical theme focuses on the ethical issues with research techniques, tools, data, and working practices. One stream of this literature crosses into normative research as it uses philosophical frameworks to evaluate specific techniques and tools. However, much of the research in this area evaluates ethical issues inherent to a technique or tool without relying explicitly on an ethical perspective. Kimmel ( 2001 ) and Kimmel and Smith ( 2001 ), for example, highlight issues with deceptive research practices. Signal et al. ( 2017 ), Nairn and Clarke ( 2012 ), Nairn ( 2006 ) and Ahuja et al. ( 2001 ) discuss the ethics of researching children. Phillips ( 2010 ) looks into observational techniques.

Emerging digital technologies have attracted much interest in this theme. Strother et al. ( 2009 ) discuss corporate blogging; Boyd and Crawford ( 2012 ) discuss the social consequences of big data; Nunan and Di Domenico ( 2013 ) highlight the ethics of privacy, security, and issues emerging from the volume of big data and the ability of firms to store, analyze and combine data in ways that were not covered by original informed consent; Tan and Salo ( 2023 ) discuss the block chain. Others consider the impact of digital technologies for traditional research practices and ask whether existing ethical ideas are sufficient in these new contexts and applications. Here, Hair and Clark ( 2007 ) focus on the challenges of ethnographic research in a digital context; Nunan and Di Domenico ( 2016 ) discuss the ethics of anonymization.

A prominent strand of the technical theme focuses on ethical considerations of neuromarketing (Murphy et al., 2008 ). As Bakardjieva and Kimmel ( 2017 : 182) explain: “the emergence of neuromarketing has rekindled some of the early concerns relating to the intrusiveness of physiological measurement and the potential applications of research findings”. Much of this work is informed by neuroethics. For example, Olteanu ( 2015 ) uses neuroethics to highlight neuromarketers’ responsibilities to research subjects, consumers and researchers including informed consent; enhancing public understanding of neuromarketing; assuring the public that neuromarketing is not used for manipulation.

Some studies adopt a more descriptive approach to explore attitudes towards neuromarketing. For instance, Eser et al. ( 2011 ) survey 111 marketing academics, 52 neurologists, and 56 marketing professionals with a 14-item survey asking participants to rate their agreement with statements such as “Neuromarketing techniques are ethical” and “Neuromarketing is a manipulative way to sell unnecessary goods and services”. They report that neurologists and practitioners hold a more positive view of neuromarketing than marketing academics. Hensel et al. ( 2017 ) examine the ethical dimensions of neuromarketing through 10 expert interviews with practitioners. They find agreement for 12 ethical principles as the basis of an ethical use of neuroscience within marketing including appropriate incentives, no manipulation of consumer behaviours, protection for vulnerable groups and the importance of informed consent. Bakardjieva and Kimmel ( 2017 ) conduct two online questionnaires with student samples from North America and France and find that a positive attitude towards neuromarketing and science was related to respondents classing neuromarketing as an ethical activity.

The technical theme is, then, distinguished from other approaches by an interest in understanding the ethics of particular techniques and technologies which have been or could be used within market research. The aim is to identify potential issues that might emerge through the use of certain techniques or tools. It is implicitly informed by a media and cultural studies view that sees ethical issues as inherent within a technology or technique. Work in the technical theme does not typically rely on a specific ethical framework, nor specific method other than a broadly case study approach that focuses on individual techniques and tools.

Critical analysis

There are common threads that run across the four research themes. First, there is agreement on the object of study. The four themes assume that market research practice and practitioners have unique ethical concerns and values. Second, there is agreement that the ethics of market research derive “from a researcher's relations with the parties in the research process, including respondents, clients of research agencies, clients of research departments (as in the case of large organizations), and the general public. Each of the parties is owed duties and responsibilities” (Akaah & Riordan, 1989 : 113). As Akaah writes: “To the extent that the fulfilment of these responsibilities creates a conflict, a research ethics problem arises” ( 1990 : 45). Thus, across the four themes, there is an interest in understanding the views, attitudes and ethical decision making of the public, consumers, academicians and as well as commercial marketing researchers themselves. Finally, there is agreement on the objectives of the research. The four themes share a desire to raise ethical standards in market research. In some cases, this aim is driven by pragmatic reasons, in others specific ethical frameworks. But to what extent has the literature delivered on these aims?

How to impact market research practice?

It is generally assumed that codes of ethics offer the most efficient way to affect market research practitioners’ behaviours. This means that the effectiveness of codes should be measured by the extent to which commercial marketing research practitioners follow them. Yet, research in the normative and descriptive themes show that marketing researchers’ ethical judgements often differ from those in codes and that researchers’ ethical decisions are more often influenced by other factors. Some argue that the problem is not codes per se but their content. It is essential they reflect real ethical dilemmas that practitioners actually encounter in practice. These might not be the most obvious to outsiders. This is indicated by the research. It shows that students and the general public rate protections for market research respondents more highly than market research practitioners. Equally, the most difficult ethical dilemmas practitioners encounter may not be the most frequently encountered. As Hunt et al. explain: “The most difficult ethical problem is not the same as the problem that occurs most frequently” ( 1984 : 318).

Put simply, studies of marketing research have yet to fully account for the role of codes of conduct. This is different to other areas of ethical research. The wider marketing ethics literature tells us that the real purpose of codes of ethics is to signal that a profession upholds ethical standards and, as a result, to maintain public acceptance of the practice. This is particularly true in fields such as advertising where unethical practices might benefit individual practitioners (Cluley, 2022 ; Harker, 1998 ; Boddewyn, 1989 ; Zaltman & Moorman, 1988 ; LaBarbera, 1983 ). Accordingly, codes of conduct in marketing have been theorised as an accommodation among adversaries (Kanter, 1974 ). They represent a marketing industries’ collective need to maintain public trust. But this means there can be a gap between the signalling effect of codes and their impact on practice. Codes, in other words, can serve their true purpose even if they are not followed.

To date, research on marketing ethics has little concern for theorizing codes of conduct beyond their content and effect on practitioners. It rarely questions what incentives researchers are offered for engaging in unethical practices. We need research to explore the ways codes are produced and enforced by researchers and clients, the operation of systems of redress and their utility to wider society. If the main function of a code of conduct is to support the public acceptance of market research, research also needs to establish how well codes map to wider public concerns. Notably, here, descriptive research could expand beyond its focus on market researchers and, following empirical studies in the technical theme, needs to consider wider views about controversial research practices. Research could draw on social contract theory to establish the most ethical basis on which to create codes.

Given findings about the limited effects of codes of conduct in commercial marketing research, we might need to develop a better understanding of the mechanism through which we can raise ethical standards. This raises questions for research practitioners and industry bodies as it may suggest a different role for marketing education and professional training. If ethical issues can only be resolved when stakeholders in the industry develop shared values, it is necessary to train market researchers to engage in reflective practice and to borrow for other sectors where different attitudes to ethical conflicts exist. For example, the airline industry prioritizes psychological safety and no-blame cultures to allow stakeholders to raise ethical concerns and deal with issues openly. Clearly, such changes require leadership from industry bodies and senior executives.

Are ethical practices good business?

There are almost no empirical explorations of why some market researchers engage in practices they and others deem controversial. There is a lack of studies exploring whether unethical practices produce more insightful findings or produce greater returns for research firms or their clients. It is, though, entirely possible that controversial practices produce better research.

Indeed, a vast amount of research into the ethics of commercial marketing research seeks to identify “readily discernible and easily avoidable ethical pitfalls” (Malhotra & Miller, 1998 : 264). But, as the literature has moved away from normative explorations, there is less concern for establishing what makes a research practice controversial. Practitioners have a key role to play here in driving the scholarly agenda. They can identify controversial practices but, by understanding the normative frameworks which influence their judgements, we can infer potential issues and conflicts on the horizon.

Such research has the potential to reveal unexpected ethical issues and behaviours within market research. For example, commenting in Whalen ( 1984 : 31) on a study showing a high acceptance of unethical practices, Jack Richardson, who drafted the code of ethics for the AMA’s New York Chapter, stated that “the results surprises the heck out of me…I realize this isn't the type of thing we talk about in professional conferences, but I have never even heard of some of those practices discussed, even at social functions”. More recently, Green et al. ( 2020 ) describe how research practitioners are using smartphones to access consumers' experiences in ways that challenge existing ethical conventions. In one case, they explain how a brand recruited families to record themselves (including children) consuming toothpaste at home. Granting consumers freedom to reveal their experiences in this way handed control of research design to participants and opened unforeseen ethical problems for the research practitioners.

We should expect new ethical issues will emerge and existing ethical values to become outdated (Poynter, 2021 ). But these are not necessarily the ones that scholars pay the most attention to. Hype around new research techniques can often attract academics’ attention before emerging technologies are adopted in practice. This creates a risk that scholarly resources are diverted towards new techniques and tools which, in reality, have limited adaption. The expansive literature on neuromarketing, for example, appears to have been based on the possibility, not the reality, of neuroscience within market research. We can compare this against a much more limited discussion of facial recognition and facial coding even though this technique is widely adopted in advertising testing (Cluley, 2022 ).

Indeed, research shows us that new technologies can have counterintuitive effects once adopted in commercial research practice. Cluley et al. ( 2020 ), for example, show how the adoption of data-driven marketing research, which has long been assumed to prioritize quantitative over qualitative interpretation, has actually led to a need for more “brand journalists” and fewer “quants”. This has implications for the idea that market research is centered around a scientific ethics of independence and objectivity as is often assumed in the normative literature.

Is it time for an update?

There is little doubt that the market research industry has undergone significant change since the publication of foundational accounts and theories of marketing research ethics (Lewis, 2012 ; Nunan, 2016 ). Marketing research methods have changed thanks to the rise of big data and digital technology. Industry relations and working practices have favoured automation, internationalization and outsourcing. In this regard, we cannot assume that practitioners and stakeholders are organized, act nor perceive ethical problems in the same ways as they once did. Indeed, responding to Yallop and Mowatt ( 2016 ) on behalf of the Research Association of New Zealand, Curran et al. ( 2017 ) highlight the need to update thinking. They write that descriptive studies offer “a springboard for discussion around ethical issues and practices in market research” but that we cannot assume that past studies present “an accurate picture of the current market research industry” ( 2017 : 280). We agree. In this section, we set out some pressing areas that demand attention.

Digital technology has fundamentally altered the relationships within marketing research (Cluley et al., 2020 ). This provides a route for knowledge exchange as, in responding to these issues, new relationships are being formed. For example, the Global Data Quality ( 2024 ) mission represents a partnership of marketing research associations who have come together to tackle the risks presented through artificial intelligence and machine learning. It brings together the Canadian Research Insights Council (CRIC), ESOMAR, Insights Association, Market Research Society (MRS), The Research Society (TRS), SampleCon, and The Association of Market Research Austria (VMÖ)).

As we move in the age of generative AI, complex and profound new ethics questions are likely to emerge (Candelon et al., 2021 ; Oliver & Vayre, 2015 ). We are entering an age where brands may use AI systems to conduct brand journalism and desk-based research tasks rather than commissioning marketing research firms. Likewise, marketing research companies may train large language models to replace human data analysts. Data providers might generate artificial panel members using AI technology.

Marketing scholarship must document and understand these practices. It can support practice by interrogating the ethical issues that such new technologies involve in a marketing research setting. This may help avert ethical controversy, legal risks and industrial action of the kinds seen in other industries such as publishing and entertainment (Bedingfield, 2023 ). A large-scale screenwriters strike, for example, was resolved when entertainment companies changed the compensation they offer and agreed to limit the role of AI (Hastings, 2024 ; Nicoletti & Bass, 2023 ). As well as taking direct lessons from other areas, marketing scholarship needs to consider what is unique about the application of these technologies in marketing practice. The risk of cyclical, self-defeating bias in AI-generated respondents, for instance, might be relatively specific to commercial research applications (Leffer, 2023 ).

At the same time as practice is changing, new ethical concerns and new ethical frameworks have risen to prominence. The social justice movement has shifted many consumer brands’ relationship to moral issues (Peñaloza et al., 2023 ; Rhodes, 2021 ). Although socially-responsible investment has been a concern for over a century, environmental ethics have more recently inspired attention beyond financial measures such as ESG (environmental, social and governance) metrics (Székely & Knirsch, 2005 ; UN Global Compact, 2004 ). We need to explore how these concerns might affect commercial research. For example, should also research tasks include environmental impact assessments?

Posthuman ethics challenges us to consider the effects of business activity beyond human interests. They have motivated, among other things, concerns for animal welfare in production and research and a concern for understanding the role of technology in social action (Giraud, 2019 ). Existing frameworks and vocabulary may be insufficient to account for these emerging issues and ethical positions. Thinking in terms of deontological or teleological bases may have less resonance in an era of climate crisis where it is impossible to ignore the greater good (Gasparin et al., 2020 ).

To respond to these practical and theoretical challenges we need methodological innovation. It is notable, in this regard, that there are only a handful of non-survey based research studies found in this review—especially among studies published in marketing journals. Such “check-the-box” survey methods have two significant weaknesses (Sparks & Hunt, 1998 : 106). First, they can limit the practices to which participants offer their opinion. In fact, many survey-based studies only explore a small pool of research practices, frequently adapted from Crawford ( 1970 )—as Table  5 shows. Second, they may be influenced by social desirability bias. On this point, Crawford acknowledged: “There is no way of being certain that what respondents said is what they truly believe” ( 1970 : 46).

There are ways around these weaknesses. Sparks and Hunt ( 1998 ) use a vignette to prompt practitioners to identify hypothetical ethical issues. Recently, though, there has been renewed interest in examining marketing practice through ethnomethodological approaches. As Zeithaml et al. ( 2020 : 49) explain, “if the field of marketing is to continue to have relevance for the practice of marketing, we must develop ideas, concepts, and theories whose central focus is the study of marketing in its natural environment”. Technology can facilitate these approaches both insitu and remotely (Cluley & Green, 2019 ; Green et al., 2020 ). Participant observation studies, for example, in which marketing scholars go to work with practitioners, allow scholars to establish first-hand how practitioners make ethical decisions (Moeran, 2005 ). They offer an emic perspective on the ways that practitioners identify, discuss and neutralize ethical issues on the frontline of marketing work. This allows practitioners to reveal things about their work not set out by the researcher in advance (Homburg et al., 2000 ; Hult, 2011 ; Moorman & Day, 2016 ).

In short, the world of marketing research has changed, and is changing, since foundational theories have been published. We need to revisit the area with fresh empirical studies that draw on a range of methods and incorporate a range of ethical perspectives. We must be willing to test, reject and develop well-established theories. In so doing, marketing scholars can help develop market research ethics and, potentially, lead to improvements in other areas of marketing theory.

We can summarize the current state of knowledge as follows. We know that ethical issues emerge in market research due to the conflicting values of different stakeholders. These create tension and conflicts for practitioners as they design, execute and report their findings. Second, we know that some research techniques and technologies have inherent ethical issues when they are used within market research. Third, we know that codes of conduct do not offer an effective way to directly raise ethical standards in market research as they do not reflect the full range of ethical dilemmas practitioners face. Finally, we know that research practitioners’ attitudes towards controversial techniques change. They tend to homogenize, with research practitioners tending to be more ethically sensitive than the general public and demonstrate greater-levels of social-responsibilities when making ethical decisions.

In terms of what we know we do not know about the ethics of market research, there are still many uncertainties. We do not know conclusively what influences market research practitioners' ethical attitudes nor how these attitudes influence their actual behaviours. There are very few studies looking at the resolution of ethical issues nor is there much work exploring market researchers’ actual ethical decisions and behaviours. There are also uncertainties about the future of the industry, new working practices and techniques. To address these gaps, we not only need to understand and consider responsibly (Cluley & Green, 2024 ) new techniques and technology but also to find ways to study practice when research activities are increasingly dispersed among people who do not consider themselves market researchers. This work is becoming more pertinent as the market research industries have faced increasing public and commercial scrutiny in recent decades.

Here, this paper makes an important intervention. It provides a central point of reference on the ethics of market research. We have expanded beyond the normative (Hunt & Vitell, 1986 ; Laczniak, 1983 ; Laczniak & Murphy, 2006 ) and descriptive (Crawford, 1970 ; Ferrell & Gresham, 1985 ; Hunt & Vitell, 1986 ) approaches that have long been the staple in the ethical marketing decision making literature through a grounded, systematic, and holistic approach to marketing research ethics (Ferrell et al., 2013 ). Two novel approaches have emerged: Theoretical and Technical. This sets a research agenda for an integrated research project that can update and develop existing thinking. It combines disparate areas of research which have, in their own way, explored the ethics of market research.

In closing, we note the following limitations to our study. First, as with any systematic review, our results are dependent on the search terms. If we had adopted a different definition of market research or synonymous terms for ethics, we may have uncovered different literatures. We attempted to use a broad and inclusive definition of market research to expose literature from all relevant areas. Indeed, the range of literature we did uncover convinces us that, while we may have found additional contributions, the general impression of siloed discussions would not alter. Similarly, investigating further databases may have deepened the granularity of our review. One reason we adopt a systematic approach is to facilitate replication to deal with these issues.

Adler, T. R., Pittz, T. G., Strevel, H. B., Denney, D., Steiner, S. D., & Adler, E. S. (2022). Team over-empowerment in market research: A virtue-based ethics approach. Journal of Business Ethics, 176 , 159–173.

Article   Google Scholar  

Aggarwal, P., Vaidyanathan, R., & Castleberry, S. (2012). Managerial and public attitudes toward ethics in marketing research. Journal of Business Ethics, 109 (4), 463–481.

Ahuja, R. D., Walker, M., & Tadepalli, R. (2001). Paternalism, limited paternalism and the Pontius Pilate plight when researching children. Journal of Business Ethics, 32 (1), 81–92.

Akaah, I. P. (1990). Attitudes of marketing professionals toward ethics in marketing research: A cross-national comparison. Journal of Business Ethics, 9 (1), 45–53.

Akaah, I. P., & Riordan, E. A. (1989). Judgments of marketing professionals about ethical issues in marketing research: A replication and extension. Journal of Marketing Research, 26 (1), 112–120.

Alsmadi, S. (2008). Marketing research ethics: Researcher’s obligations toward human subjects. Journal of Academic Ethics, 6 (2), 153–160.

Alsmadi, S. (2010). Marketing research and social responsibility: Ethical obligations toward the society. Journal of Accounting, Business & Management, 17 (1), 42–47.

Google Scholar  

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19–32.

Bakardjieva, E., & Kimmel, A. J. (2017). Neuromarketing research practices: Attitudes, ethics, and behavioral intentions. Ethics & Behavior, 27 (3), 179–200.

Bedingfield, W. (2023, 2024). Hollywood writers reached an AI deal that will rewrite history. Wired . Accessed 15th April, 2024. https://www.wired.co.uk/article/us-writers-strike-ai-provisions-precedents

Blankenship, A. B. (1964). Some aspects of ethics in marketing research. Journal of Marketing Research, 1 (2), 26–31.

Boddewyn, J. J. (1989). Advertising self-regulation: True purpose and limits. Journal of Advertising, 18 (2), 19–27.

Boulanouar, A. W., Aitken, R., Boulanouar, Z., & Todd, S. J. (2017). Imperatives for research designs with Muslim women. Marketing Intelligence & Planning, 35 (1), 2–17.

Boyd, D., & Crawford, K. (2012). Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information, Communication & Society, 15 (5), 662–679.

Candelon, F., di Carlo, R. C., De Bondt, M., & Evgeniou, T. (2021). AI regulation is coming: How to prepare for the inevitable. Harvard Business Review, 99 (5), 103–111.

Castleberry, S. B., French, W., & Carlin, B. A. (1993). The ethical framework of advertising and marketing research practitioners: A moral development perspective. Journal of Advertising, 22 (2), 39–46.

Cluley, R. (2019). The politics of consumer data. Marketing Theory, 20 (1), 45–63.

Cluley, R. (2022). Interesting numbers: An ethnographic account of quantification, marketing analytics and facial coding data. Marketing Theory, 22 (1), 3–20.

Cluley, R., & Green, W. (2019). Social representations of marketing work: Advertising workers and social media. European Journal of Marketing, 53 , 830–847.

Cluley, R., & Green, W. (2024). Responsible Technology in Marketing: Theory, Adoption, Practice. In M. Saren, L. M. Hassan, M. McGowan, N. C. Smith, E. Surman, & R. Varman (Eds.), Responsible Marketing for Well-being and Society: A Research Companion (1st ed., pp. 175–192). London: Routledge Research Companions in Business and Economics, Routledge.

Chapter   Google Scholar  

Cluley, R., Green, W., & Owen, R. (2020). The changing role of the marketing researcher in the age of digital technology: Practitioner perspectives on the digitization of marketing research. International Journal of Market Research, 62 (1), 27–42.

Crawford, C. M. (1970). Attitudes of marketing executives toward ethics in marketing research. Journal of Marketing, 34 (2), 46–52.

Curran, K., Bramley, K., & Henderson, W. (2017). Letters regarding ‘investigating market research ethics: An empirical study of codes of ethics in practice and their effect on ethical behaviour’, Anca C. Yallop and Simon Mowatt, published in IJMR 58/3, 2016. International Journal of Market Research, 59 (3), 279–280.

Day, R. L. (1975). A comment on ‘ethics in marketing research.’ Journal of Marketing Research, 12 (2), 232–233.

Di Domenico, G., Sit, J., Ishizaka, A., & Nunan, D. (2021). Fake news, social media and marketing: A systematic review. Journal of Business Research, 124 , 329–341.

Dodds, S., Finsterwalder, J., Prayag, G., & Subramanian, I. (2023). Transformative service research methodologies for vulnerable participants. International Journal of Market Research, 65 (2–3), 279–296.

Dutka, A. F. (1994). The maintenance of ethical principles in marketing research. American statistical association 1994 proceedings of the section on survey research methods, vols i and ii.

Eser, Z., Isin, F. B., & Tolon, M. (2011). Perceptions of marketing academics, neurologists, and marketing professionals about neuromarketing. Journal of Marketing Management, 27 (7–8), 854–868.

ESOMAR. (2021). Market research explained. https://www.esomar.org/market-research-explained . Accessed: 3rd September 2021.

Ferrell, O. C., Crittenden, V. L., Ferrell, L., & Crittenden, W. F. (2013). Theoretical development in ethical marketing decision making. AMS Review, 3 , 51–60.

Ferrell, O. C., & Gresham, L. G. (1985). A contingency framework for understanding ethical decision making in marketing. Journal of Marketing, 49 (3), 87–96.

Ferrell, O. C., Hartline, M. D., & McDaniel, S. W. (1998). Codes of ethics among corporate research departments, marketing research firms, and data subcontractors: An examination of a three-communities metaphor. Journal of Business Ethics, 17 (5), 503–516.

Ferrell, O. C., & Skinner, S. J. (1988). Ethical behavior and bureaucratic structure in marketing research organizations. Journal of Marketing Research, 25 (1), 103–109.

Fisher, J. (2001). Lessons for business ethics from bioethics. Journal of Business Ethics, 34 (1), 15–24.

Gasparin, M., Brown, S. D., Green, W., Hugill, A., Lilley, S., Quinn, M., .., & Zalasiewicz, J. (2020). The business school in the Anthropocene: Parasite logic and pataphysical reasoning for a working earth. Academy of Management Learning & Education, 19 (3), 385–405.

Giacobbe, R. W., & Segal, M. N. (2000). A comparative analysis of ethical perceptions in marketing research: USA vs. Canada. Journal of Business Ethics, 27 (3), 229–245.

Giraud, E. H. (2019). What comes after entanglement?: Activism, anthropocentrism, and an ethics of exclusion . Duke University Press.

Book   Google Scholar  

Global Data Quality. (2024). “Our mission”. Global data quality: Tackling data fraud in market research. Accessed 15th April, 2024. https://www.globaldataquality.org

Green, W., Cluley, R., & Gasparin, M. (2020). Mobile market research applications as a new voice of customer method: Implications for innovation and design management. Research-Technology Management, 63 (1), 49–55.

Hair, N., & Clark, M. (2007). The ethical dilemmas and challenges of ethnographic research in electronic communities. International Journal of Market Research, 49 (6), 1–13.

Harker, D. (1998). Achieving acceptable advertising. International Marketing Review, 15 (2), 101–118.

Hastings, J. (2024). Preventing harm from non-conscious bias in medical generative AI. The Lancet Digital Health, 6 (1), e2–e3.

Hensel, D., Iorga, A., Wolter, L., & Znanewitz, J. (2017). Conducting neuromarketing studies ethically-practitioner perspectives. Cogent Psychology, 4 (1), 1320858.

Homburg, C., Workman, J. P., & Jensen, O. (2000). Fundamental changes in marketing organization: The movement toward a customer-focused organizational structure. Journal of the Academy of Marketing Science, 28 (4), 459–478.

Hult, G. T. M. (2011). Toward a theory of the boundary-spanning marketing organization and insights from 31 organization theories. Journal of the Academy of Marketing Science, 39 (4), 509–536.

Hunt, S. D. (2013). The inductive realist model of theory generation: Explaining the development of a theory of marketing ethics. AMS Review, 3 , 61–73.

Hunt, S. D., Chonko, L. B., & Wilcox, J. B. (1984). Ethical problems of marketing researchers. Journal of Marketing Research, 21 (3), 309–324.

Hunt, S. D., & Vitell, S. (1986). A general theory of marketing ethics. Journal of Macromarketing, 6 (1), 5–16.

Kanter, D. L. (1974). Psychological considerations in advertising regulation. California Management Review, 16 (3), 73–78.

Kelley, S. W., Ferrell, O. C., & Skinner, S. J. (1990). Ethical behavior among marketing researchers: An assessment of selected demographic characteristics. Journal of Business Ethics, 9 (8), 681–688.

Kimmel, A. J. (2001). Ethical trends in marketing and psychological research. Ethics & Behavior, 11 (2), 131–149.

Kimmel, A. J., & Smith, N. C. (2001). Deception in marketing research: Ethical, methodological, and disciplinary implications. Psychology & Marketing, 18 (7), 663–689.

Kozinets, R. V. (2023). Immersive netnography: A novel method for service experience research in virtual reality, augmented reality and metaverse contexts. Journal of Service Management, 34 (1), 100–125.

LaBarbera, P. A. (1983). The diffusion of trade association advertising self-regulation. Journal of Marketing, 47 (1), 58–67.

Laczniak, G. R. (1983). Framework for analyzing marketing ethics. Journal of Macromarketing, 3 (1), 7–18.

Laczniak, G. R., & Murphy, P. E. (2006). Normative perspectives for ethical and socially responsible marketing. Journal of Macromarketing, 26 (2), 154–177.

Leffer, L. (2023). Humans absorb bias from AI—and keep it after they stop using the algorithm. Scientific American. Accessed 15th April, 2024. https://www.scientificamerican.com/article/humans-absorb-bias-from-ai-and-keep-it-after-they-stop-using-the-algorithm/

Lewis, I. (2012). The future of market research. International Journal of Market Research, 54 (1), 11–13.

Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis . Oxford University Press.

Macpherson, A., & Jones, O. (2010). Strategies for the development of international journal of management reviews. International Journal of Management Reviews, 12 (2), 107–113.

Malhotra, N. K., & Miller, G. L. (1998). An integrated model for ethical decisions in marketing research. Journal of Business Ethics, 17 (3), 263–280.

Malhotra, N. K., & Peterson, M. (2001). Marketing research in the new millennium: Emerging issues and trends. Marketing Intelligence & Planning, 19 (4), 216–232.

Michaelides, P., & Gibbs, P. (2006). Technical skills and the ethics of market research. Business Ethics: A European Review, 15 (1), 44–52.

Moeran, B. (2005). The business of ethnography: Strategic exchanges, people and organizations . Routledge.

Moorman, C., & Day, G. S. (2016). Organizing for marketing excellence. Journal of Marketing, 80 (6), 6–35.

Murphy, P. E., & Laczniak, G. R. (1992). Traditional ethical issues facing marketing researchers. Marketing Research, 4 (1), 8–21.

Murphy, E. R., Illes, J., & Reiner, P. B. (2008). Neuroethics of neuromarketing. Journal of Consumer Behaviour: An International Research Review, 7 (4–5), 293–302.

Nairn, A. (2006). Commercialisation of childhood? The ethics of research with primary school children. International Journal of Market Research, 48 (2), 113–114.

Nairn, A., & Clarke, B. (2012). Researching children: Are we getting it right?: A discussion of ethics. International Journal of Market Research, 54 (2), 177–198.

Nicoletti, L., & Bass, D. (2023). Humans are biased. Generative AI is even worse. Bloomberg. June 2023. Accessed 15th April, 2024. https://www.bloomberg.com/graphics/2023-generative-ai-bias/#:~:text=Like%20many%20AI%20models%2C%20what,found%20in%20the%20real%20world

Nunan, D. (2016). The declining use of the term market research: An empirical analysis. International Journal of Market Research, 58 , 503–522.

Nunan, D., & Di Domenico, M. (2013). Market research and the ethics of big data. International Journal of Market Research, 55 (4), 505–520.

Nunan, D., & Di Domenico, M. (2016). Exploring reidentification risk: Is anonymisation a promise we can keep? International Journal of Market Research, 58 (1), 19–34.

Oliver, M. A., & Vayre, J. S. (2015). Big data and the future of knowledge production in marketing research: Ethics, digital traces, and abductive reasoning. Journal of Marketing Analytics, 3 (1), 5–13.

Olteanu, M. D. B. (2015). Neuroethics and responsibility in conducting neuromarketing research. Neuroethics, 8 (2), 191–202.

Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ..., & McKenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ , 372 , n160. (1–36).

Pallister, J., Nancarrow, C., & Brace, I. (1999). Navigating the righteous course: A quality issue. Journal of the Market Research Society, 41 (3), 327–342.

Peñaloza, L., Prothero, A., McDonagh, P., & Pounders, K. (2023). The past and future of gender research in marketing: Paradigms, stances, and value-based commitments. Journal of Marketing, 87 (6), 847–868.

Peters, M. D., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015). Guidance for conducting systematic scoping reviews. International Journal of Evidence-Based Healthcare, 13 (3), 141–146.

Phillips, A. (2010). Researchers, snoopers and spies–the legal and ethical challenges facing observational research. International Journal of Market Research, 52 (2), 275–278.

Popay, J., et al. (2006). Guidance on the conduct of narrative synthesis in systematic reviews: A product from the ESRC methods programme . ESRC Methods Programme.

Poynter, R. (2021). Market research: A state of the nation review. International Journal of Market Research, 63 (4), 403–407.

Rhodes, C. (2021). Woke capitalism: How corporate morality is sabotaging democracy . Policy Press.

Rotfeld, H. J., & Taylor, C. R. (2009). The advertising regulation and self-regulation issues ripped from the headlines with (sometimes missed) opportunities for disciplined multidisciplinary research. Journal of Advertising, 38 (4), 5–14.

Segal, M. N., & Giacobbe, R. W. (2007). Ethical issues in Australian marketing research services: An empirical investigation. Services Marketing Quarterly, 28 (3), 33–53.

Signal, L. N., Smith, M. B., Barr, M., Stanley, J., Chambers, T. J., Zhou, J., ..., & Mhurchu, C. N. (2017). Kids’ cam: An objective methodology to study the world in which children live. American Journal of Preventive Medicine, 53 (3), e89–e95.

Skinner, S. J., Dubinsky, A. J., & Ferrell, O. C. (1988). Organizational dimensions of marketing-research ethics. Journal of Business Research, 16 (3), 209–223.

Sojka, J., & Spangenberg, E. R. (1994). Ethical concerns in marketing research. ACR North American Advances, VOL XXI.

Sparks, J. R., & Hunt, S. D. (1998). Marketing researcher ethical sensitivity: Conceptualization, measurement, and exploratory investigation. Journal of Marketing, 62 (2), 92–109.

Strother, J. B., Fazal, Z., & Millsap, M. (2009). Legal and ethical issues of the corporate blogosphere. IEEE Transactions on Professional Communication, 52 (3), 243–253.

Sun, W., & Govind, R. (2022). A new understanding of marketing and “doing good”: Marketing’s power in the TMT and corporate social responsibility. Journal of Business Ethics, 176 , 89–109.

Székely, F., & Knirsch, M. (2005). Responsible leadership and corporate social responsibility: Metrics for sustainable performance. European Management Journal, 23 (6), 628–647.

Tan, T. M., & Salo, J. (2023). Ethical marketing in the blockchain-based sharing economy: Theoretical integration and guiding insights. Journal of Business Ethics, 183 , 1113–1140.

Toy, D., Wright, L., & Olson, J. (2001). A conceptual framework for analyzing deception and debriefing effects in marketing research. Psychology & Marketing, 18 (7), 691–719.

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14 (3), 207–222.

Twedt, D. W. (1963). Why a marketing research code of ethics? Journal of Marketing, 27 (4), 48–50.

Tybout, A. M., & Zaltman, G. (1974). Ethics in marketing research: Their practical relevance. Journal of Marketing Research, 11 (4), 357–368.

Tybout, A. M., & Zaltman, G. (1975). A reply to comments on “ethics in marketing research: Their practical relevance.” Journal of Marketing Research, 12 (2), 234–237.

Ulman, Y. I., Cakar, T., & Yildiz, G. (2015). Ethical issues in neuromarketing: “I consume, therefore I am!” Science and Engineering Ethics, 21 (5), 1271–1284.

UN Global Compact. (2004). UN Global Compact (2004) who cares wins: Connecting financial markets to a changing world. Global compact. Accessed 15th April, 2024. www.globalcompact.org

Whalen, B. (1984). Research rise in use of ‘JR’ research tactics. Marketing News, 18 (May), 1.

Wieland, H., Nariswari, A., & Akaka, M. A. (2021). On managerial relevance: Reconciling the academic-practitioner divide through market theorizing. AMS Review, 11 , 252–271.

Wilson, M., Gaines, J., & Hill, R. (2008). Neuromarketing and consumer free will. Journal of Consumers Affair, 32 , 389–410.

Yallop, A. C., & Mowatt, S. (2016). Investigating market research ethics: An empirical study of codes of ethics in practice and their effect on ethical behaviour. International Journal of Market Research, 58 (3), 381–400.

Zaltman, G., & Moorman, C. (1988). The importance of personal trust in the use of research. Journal of Advertising Research, 28 (5), 16–24.

Zeithaml, V. A., Jaworski, B. J., Kohli, A. K., Tuli, K. R., Ulaga, W., & Zaltman, G. (2020). A theories-in-use approach to building marketing theory. Journal of Marketing, 84 (1), 32–51.

Download references

Author information

Authors and affiliations.

Department of Marketing, University of Birmingham, Birmingham, UK

Robert Cluley

Department of Management, University of Birmingham, Birmingham, UK

William Green

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed equally to the research contributing to the study and manuscript drafting, and all authors read and approved the final manuscript.

Corresponding author

Correspondence to William Green .

Ethics declarations

Competing interests.

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 15 kb)

Supplementary file2 (docx 47 kb), rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cluley, R., Green, W. Market research ethics: New practices but no new ideas. AMS Rev (2024). https://doi.org/10.1007/s13162-024-00276-8

Download citation

Received : 12 September 2023

Accepted : 26 March 2024

Published : 07 June 2024

DOI : https://doi.org/10.1007/s13162-024-00276-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Marketing research ethics
• Market research
• Scoping review
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 06 June 2024

Tools for measuring curriculum integration in health professions’ education: a systematic review

• Soumaya Allouch 1   na1 ,
• Raja Mahamade Ali 1   na1 ,
• Noor Al-Wattary 2 ,
• Michail Nomikos 1 &
• Marwan F. Abu-Hijleh 1  

BMC Medical Education volume  24 , Article number:  635 ( 2024 ) Cite this article

Metrics details

Curriculum integration is an important educational concept widely implemented by various educational institutions, particularly within the healthcare field. Its significance lies in enhancing the preparation of future healthcare professionals. The assessment of these integrated curricula is imperative to guarantee their effectiveness. Consequently, the aim of this systematic review is to delve into existing literature, with the goal of identifying instruments designed to assess the extent of curriculum integration in health professions’ education.

A comprehensive search was conducted to identify peer-reviewed papers and grey literature describing the development, validation, or use of instruments measuring the degree of integration in a curriculum. Eight databases were searched: PubMed, Scopus, Google Scholar, CINAHL Ultimate, Web of Science, Cochrane, ProQuest Central and EMBASE. Grey literature was also included. Titles, abstracts, and full text screening was conducted. Data extraction was done using a data extraction tool developed by our research team.

The search resulted in the identification of 2094 references. After the removal of duplicates and title and abstract screening, 16 articles were deemed suitable for inclusion in this systematic review. Twenty-two instruments were extracted from these articles. The identified instruments assessed either integration attributes, perceptions about the integrated curriculum characteristics, process and outcomes, or curriculum integration level. Two of the instruments were focused on assessing horizontal integration ( Basic Science Curriculum Assessment Instrument and the integration characteristic tool). In addition, one instrument was developed to assess integration within a single session only, while other instruments assessed curriculum integration level. Two of the integration instruments ( The Session Integration Tool and Integration Ladder Questionnaire) provided scales for calculating integration levels. Validation of the integration assessment instruments was infrequent, with only 9 of 22 instruments validated for their psychometric properties.

Our findings reveal the existence of diverse instruments designed to assess the extent of curriculum integration within health professions’ curricula. The majority of identified instruments were focused on participants’ perceptions towards the attributes of the integrated curriculum, and a significant number of these tools lacked validation.

Peer Review reports

Curriculum integration is a concept which has been frequently discussed in educational literature for the past 3 decades [ 1 ]. Introduction of this concept was accompanied by attempts to reform medical curriculum from the traditional discipline-based curriculum to an integrated one; with the first documented attempt in McMaster University, Canada [ 1 ]. The primary reason for this shift is the recognition that conventional discipline-based curricula fall short in equipping medical students with the necessary skills to effectively apply their acquired knowledge in clinical practice [ 2 ]. While the term ‘integration’ is commonly used throughout the literature, a clear consensus on its definition within medical curriculum remains elusive. For instance, a recent systematic review by Matinho et al. (2022) on integrated learning in health professions’ education found that only 37% of relevant studies provided a clear definition of integration [ 3 ]. These definitions belonged to one of three main categories which described integrated learning as: (1) the extent to which educators from different disciplines co-present knowledge, beliefs or skills to students; (2) the organizational approach that informs how curricular elements are structured and arranged; and (3) the description of the cognitive or metacognitive processes occurring within the learners’ mind. The first category is well aligned with Harden’s definition, which states that integration is ‘the organisation of teaching matter to interrelate or unify subjects frequently taught in separate academic courses or department’ [ 4 ]. The second category identified in the systematic review aligns with Brauer & Ferguson’s perspective on integrated curriculum, which views it as ‘ a fully synchronous, trans-disciplinary delivery of information between the foundational sciences and the applied sciences throughout all years of a medical school curriculum’ [ 1 ].

Matinho et al.’s (2022) systematic review also highlights the practical implementation of this definition through vertical, horizontal, and spiral integration [ 3 ]. Horizontal integration refers to the integration across different subject areas within a finite period of time, while vertical integration refers to the integration between basic and clinical disciplines across time [ 2 , 5 ]. In vertical integration the amount of time spent on classroom education tends to decrease gradually as more clinical practice experience is introduced [ 5 ]. Integration in its most ideal form represents a combination of both horizontal and vertical, uniting integration across time and across disciplines, which has often been termed as ‘spiral integration’ [ 1 , 6 ]. These levels of integration in medical curricula are described as a continuum, or spectrum. This ranges from traditional curriculum design, where contents are taught as separate disciplines, to a highly innovative integrated approach where disciplinary boundaries are abandoned [ 7 ].

In the integrated curricula, teaching revolves around themes upon which the content of individual units is developed [ 8 ]. This approach encourages students to see the link between different subjects and helps them understand applications of this knowledge in practice. An important model of integration, which was developed for curriculum planning and review, is the Student-Centered, Problem-Based, Integrated, Community-Based, Elective, Systematic (SPICES) model [ 4 ]. This model describes a set of six educational strategies arranged in a continuum across two extremes, ranging from the least desirable traditional curriculum to the most desirable innovative curriculum. In the new integrated curriculum students are the focus of the learning experience, and they are given liberty to determine their learning objectives, learning resources, and sequence of their learning content under instructors’ guidance [ 4 ]. Problem-based learning (PBL) is the learning tool of choice in this model. PBL is a small group learning approach (8–10 students), in which students are provided with a problem they need to solve through conducting research, reviewing relevant resources, and integrating theory within practice [ 9 ]. The process is facilitated by a tutor who supports students and provides a thorough debriefing at the end of the PBL session [ 9 ]. Another important model of integration is Harden’s integration ladder, which consists of eleven steps describing the integration degree of a curriculum on a continuum ranging from isolation (no integration) to multidisciplinary (fully integrated curriculum) [ 10 ]. A more detailed model is Fogarty’s integration model which classifies integration levels according to where integration is adopted into three broad categories; within single disciplines, across several disciplines and within and across learners [ 11 ]. This model generated ten integration levels ranging from fragmented curricula, in which integration is absent, to networked curricula, in which integration of knowledge occurs within and across the learners’ mind as they direct the integration process both internally and externally (determining needed resource, expert matter experts…etc.) [ 11 , 12 ]. All these models can be used to guide the planning, development, or evaluation of integrated curricula.

Several studies report that integrated medical curricula demonstrate greater effectiveness compared to conventional curricula [ 13 , 14 , 15 ]. Students from integrated medical programs were shown to perform better in examinations of medicine, pediatrics, obstetrics and gynecology subjects when compared to students from traditional curricula [ 14 , 16 ]. Additionally, it was reported that graduates from integrated curricula tend to make definitive career choices earlier, are more likely to be accepted at residency positions faster and are more confident in their readiness for practice [ 13 ]. Similarly, PBL was found to be superior to traditional learning methods in enhancing students’ social and communication skills, as well as advancing their problem solving and self-learning skills [ 17 ]. It was also reported that students from integrated curricula and those exposed to PBL have superior diagnostic skills compared to students from traditional curricula [ 15 ]. Learning theories suggest that the integrated approach of teaching and learning enhances students’ learning, engages adult learners’ interest in meaningful learning, and improves retention of knowledge [ 18 , 19 , 20 ]. Integrated curricula are designed to encourage students to establish connections between various subjects, thus enabling them to recognize how their knowledge can be applied to real-world patient cases [ 8 ]. In addition, integrated curricula provide students with opportunity to engage in self-directed learning and develop clinical reasoning skills. This also allows students to express their personal identities and individual qualities while learning, and as a result helps them in developing their individual attributes as future healthcare providers [ 8 ].

Despite the challenges of defining integration, there are domains and dimensions to the construct that provide guidance and boundaries for defining what constitutes integration [ 21 ]. The general assumption is that integration should promote retention of knowledge and acquisition of skills through repetitive and progressive development of concepts and their applications [ 1 ]. Different educational institutions, particularly in the health profession field, adopted integration in their curriculum to prepare their students for practice [ 3 ]. However, for these curricula to be effective, it is a paramount to evaluate them and assess the degree and extent of curriculum integration following its implementation using appropriate tools. These tools will help educators and curriculum developers in identifying gaps in their curriculum design pertaining to integration and provide suitable solutions. Therefore, the aim of this systematic review is to explore the current literature to identify tools, instruments, or surveys, which have been developed to assess the degree of curriculum integration in health professions education.

The research question

What tools, instruments, or surveys are available for measuring the degree of curriculum integration in health professions education?

Methodology

This systematic review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist for systematic reviews 2020 [ 22 ].

Search strategy

A comprehensive search was conducted in 8 large databases: PubMed, Scopus, google scholar, CINAHL Ultimate, Web of science, Cochrane, ProQuest central and EMBASE. The aim was to identify peer-reviewed papers and grey literature describing the development, validation, or use of instruments measuring the degree of integration in health professions’ curriculum. To identify relevant articles, the search was conducted in these eight databases using different combinations of the keywords listed below:

Tool, instrument, survey, questionnaire, scale, measure.

Curriculum delivery, curriculum evaluation, curriculum assessment.

Integrated curriculum, vertical integration, horizontal integration, spiral integration, basic sciences integration, clinical sciences integration, clinical and basic sciences’ integration.

Medical education, medical school, medical college, health professions education.

Problem based learning, PBL, student centered curriculum.

The full search strategy for the PubMed database can be found in Additional file 1 .

Inclusion and exclusion criteria

Peer-reviewed articles or grey literature published in English up until October 2, 2023 were included in the search. Grey literature (e.g., conference proceedings, thesis, dissertation etc.), relevant to our study identified through the search was also included. Evidence, including questionnaires or instruments assessing the degree of curriculum integration (as the construct of the instrument or one of the main constructs if the instrument consists of multiple constructs/domains) in health professions’ education, was included in this study. Studies were eligible for inclusion if they incorporated the questionnaire or questions assessing the degree of integration within the article. Studies describing such tools and their questions (whether validated or not) were also included. Articles not fulfilling our inclusion criteria were excluded from this systematic review.

In the first phase, one researcher screened titles yielded from the database search, aiming to identify relevant articles for inclusion in the study. In the second phase, three researchers independently screened abstracts and keywords of articles identified in step one. In the final step, three researchers thoroughly reviewed the full texts of papers that successfully passed the previous screening. Only papers that presented or described an instrument, tool or survey assessing curriculum integration were considered eligible for inclusion. Additionally, we conducted a manual search to identify any additional relevant papers which might have been missed. All eligible articles, once identified, underwent screening for inclusion. Any conflict or disagreement between reviewers at any stage of the review was resolved through discussion or involvement of a third researcher who was involved in the study since early stages and is aware of the inclusion and exclusion criteria. Reasons for exclusion of citations were documented and are reported in the results section of this systematic review. The search process is reported in detail per the PRISMA flow diagram [ 22 ]. Characteristics of included papers :

This study only included observational studies describing the development and/or use of an instrument evaluating curriculum integration (e.g. cross-sectional studies, longitudinal studies, cohort studies etc.). Following the search of the selected databases, all identified citations were uploaded to Mendeley citation management software 2.95/2023 (©2023 Mendeley Ltd., Elsevier), and duplicates were removed [ 23 ]. Articles were then transferred to Rayyan, which is a web application that facilitates the process of abstract and full text screening of articles included in a review by different team members [ 24 ]. The tool also identifies and highlights disagreements between different researchers, and documents decisions made regarding articles screened.

Data collection process

We extracted data from studies that met the inclusion criteria using a data extraction tool developed by our research team. The extracted data included information about the papers, such as authors’ names and publication years, and data on the instruments, including their names, objectives, main domains measured, and other details such as number of items, scale, and scoring system. If the instrument was utilized within a sample, details about this sample were extracted from the relevant paper and are reported in this review. Psychometric properties were also extracted if they were reported in the retrieved articles.

Quality assessment

The quality of the papers included in this review were assessed using the Risk of Bias Utilized for Survey Tool (ROBUST) [ 25 ]. The tool consists of 8 criteria which measures sample frames, participant recruitment, exclusion rate, sample size, measurement validity, setting and data management. If the study met the criteria a score of 1 corresponding to “yes” is given while a score of “0” corresponding to “No” are assigned for studies which fails to meet the criteria. For the purpose of this study, we modified the tool by removing the “exclusion rate” item as this element is not reported in our studies. Additionally, the criteria for sample characteristics have been modified as in our study (response “yes” represents reporting the college and academic year/s). The total score for quality ranges between 0 and 7; such that 0 represent the lowest level of confidence in the results and 7 the highest confidence level.

The search resulted in the identification of 2094 references. Six additional articles were identified through manual search. After eliminating duplicates, a total of 1905 references were retained and underwent title and abstract screening. After full text screening, 16 articles fulfilled this study’s inclusion criteria and were deemed suitable for inclusion in this systematic review. Details of the search strategy are presented in Fig.  1 . Twenty-two instruments assessing curriculum integration in health professions’ education were extracted from these articles.

PRISMA flow diagram summarizing the search strategy followed to identify instruments assessing the degree of curriculum integration in health professions’ education

The papers included in this review were published between 1980 and 2023, with the majority published after 2010 ( n  = 12). All identified instruments were questionnaires developed to assess participants’ perception of curriculum integration or integration level, except for one instrument, which was administered as an assessment rubric of integration competency during an Objective Structured Clinical Examination (OSCE) [ 26 ]. Additionally, all instruments identified focused on assessing integration on a curricular level except for one instrument; the Session Integration Tool (SIT), which was developed to measure the level of integration between different disciplines in a single session [ 27 ].

Most of the included papers were centered on capturing students’ perceptions and experiences with curriculum integration ( n  = 11) [ 26 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ]. In contrast, some were specifically designed to investigate faculty members’ opinions and evaluations of the curricula and its level of integration [ 27 , 37 , 38 , 39 ]. Notably, in seven studies, integration-assessing questionnaires were distributed to both students and faculty members to benefit from the perceptions and unique experiences of each group [ 26 , 29 , 30 , 31 , 33 , 35 , 36 ]. Only two studies focused on evaluations of academic leaders and expert evaluators [ 40 , 41 ].

Half of the integration assessment questionnaires identified in this review were developed to evaluate integration in medical curricula ( n  = 11), and targeted either students, faculty members or integrated course’s developers or directors at medical schools. The remaining questionnaires ( n  = 11) were developed to assess integration in other health professions’ curricula namely physical therapy, nursing, pharmacy, dentistry, radiography and health sciences. More details of integration instruments and the samples in which they were used is provided in Tables  1 and 2 .

In terms of the instruments’ content, the number of questions varied widely, with the lowest reported being 4 in the Integrated standardized patient examination assessment rubric [ 37 ], while the largest set of questions was found in the Integrated curriculum in nursing inventory consisting of 138 questions divided into 3 different Sect [ 26 ]. The integrated curriculum in nursing inventory is a comprehensive questionnaire that evaluates the participant understanding of curriculum integration, their perception of the current integrated curriculum, and their views on how an integrated curriculum should be. The response options for all close-ended questions in the identified instruments within this review were presented using a Likert scale. Notably, the 5-point Likert scale was the most commonly employed scale in this systematic review.

Our findings suggest that the validation of instruments’ assessing integration was uncommon. Of the 22 instruments identified in this review, only 9 underwent assessment for psychometric properties. The most frequently reported psychometric property among the included instruments was content validity ( n  = 7). The instruments analyzed in this study can be classified into three main groups based on their objectives:

Instruments assessing integration attributes through students’ performance (outcome of curriculum integration).

Instruments assessing participants’ (students or faculty members) perception about the integrated curriculum characteristics, process, and outcomes .

Instruments evaluating curriculum integration level based on participants’ experiences (e.g. reviewing integration introduction in the health professions’ curricula of a country, assessing the level of curriculum integration in an institution).

Instruments assessing integration attributes

Three instruments assessing students’ ability to integrate knowledge were identified in this review. These instruments were developed by Panzarella (2003) to assess integration attributes of physical therapy students during an OSCE exam [ 26 ]. One of these instruments is the Integrated Standardized Patient Examination (ISPE), which functions as an assessment rubric for integration. It evaluates students’ competency in integration by assessing their responses to integration-related questions posed by standardized patients. The remaining two instruments were questionnaires designed to investigate perceptions of students’ performance, specifically regarding integration competencies, during the OSCE interaction with the standardized patients (SP). These questionnaires were intended for both students and expert evaluators. The ISPE was validated for validity and reliability measures. A definition of integration was provided in the beginning of the feedback assessing instruments. Detailed findings are reported in Table  1 .

Instruments assessing perceptions about integrated curriculum characteristics, process, and outcomes

Instruments in this category evaluate integrated curricula’s characteristics, including aspects of delivery and implementation. Several of these instruments also explore participants’ perceptions about the usefulness of the curriculum in terms of achieving desired outcomes of curriculum integration. The majority of questionnaires identified in this review fall into this category ( n  = 16) [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 40 ]. Many of these instruments explore participants’ opinions on different aspects of the integrated curriculum, such as the content, delivery, time-management, and teaching methods. Examples of these instruments include The Questionnaire Assessing Students’ Perception Regarding an Integrated Curriculum at a Public Sector Medical Colleg e, The Integration Characteristic Tool and The Basic Science Curriculum Assessment Instrument [ 28 , 30 , 34 ]. Some instruments like The Integrated Curriculum Implementation Inventory and Student and Faculty Online Survey Questionnaires developed by Le BK [ 31 ]. assess the degree of integrated curriculum implementation. One unique instrument within this category is The Integrated Curriculum in Nursing Inventory , which assesses the respondent’s view and understanding of the “integrated curriculum” concept at the beginning of the questionnaire before evaluating their perception of the current curriculum [ 37 ]. Additionally, this is the only instrument which enables the participants to express their perceptions of the required changes for an ideal integrated curriculum.

Two of the identified instruments; The integration survey [ 38 ] and Faculty perception of curricular integration survey [ 40 ] were developed to assess curricular integration in pharmacy programs on a national level. Therefore, the target sample for these instruments are faculty members involved in the delivery of integrated curricula and academic leaders from institutions adopting integrated curricula. Many of the studies cited in this review assessed curriculum integration through the perceptions of students, faculty members and academic leaders.

Instruments evaluating curriculum integration levels

This category includes three instruments, The Integrated Curriculum Evaluation Instrument by Howard et al. (2009), the Integration Ladder Questionnaire , and The Session Integration Tool (SIT) [ 27 , 39 , 41 ]. These three instruments are the most specific tools among all the integration instruments identified as they assess the degree of curriculum integration and provide a detailed description of the curriculum integration level. The first two of these instruments ( The Integrated Curriculum Evaluation Instrument by Howard et al., Integration Ladder Questionnaire ) were developed based on established integration models; the Fogarty models for curriculum integration [ 11 ] and Harden’s integration ladder [ 10 ] respectively.

Unlike the other two instruments in this category, the SIT has a narrower scope as it assesses the degree of integration within one session and not across the whole curriculum [ 27 ]; however, it has been adequately validated and has a clear score calculation approach. The scores are then interpreted into one of four categories each representing a level of integration. Likewise, The integration ladder questionnaire [ 39 ] has a reported method for integration score calculation, the mean score is then used to determine the integration level on Harden’s ladder. Further details about the instruments and their psychometric properties are described in Table  2 .

The majority of the studies included in the review were found to have low risk of bias with 2 instruments scoring 7 while many of them scoring between 5 and 6 out of the 7 overall score ( n  = 9). On the other hand, five of the papers had score of 4 or less. Details of the quality assessment results are reported in Table  3 .

This systematic review successfully addresses the research question by identifying articles that report on instruments evaluating the degree of curriculum integration in health professions’ education. This review identified twenty-two instruments focused on evaluating the degree of curriculum integration in health professions’ curricula. Curriculum evaluation is a process which focuses on obtaining information about different components of the curriculum [ 42 ]. There are various sources for information regarding the curriculum; however, the majority of instruments identified in our review focus on obtaining the input of students and faculty members who are the end users of a health professions curriculum.

The studies reviewed in this analysis examined the curricula of various health professions, with a predominant emphasis on medical education programs. This outcome was anticipated, as the concept of “curriculum integration” has traditionally been closely associated with medical education. However, it’s noteworthy that the concept’s application has gradually extended beyond medical training to encompass other fields within healthcare education [ 1 ]. The instruments were classified based on the objective of their assessment into three categories; instruments assessing integration attributes, instruments assessing perceptions about integrated curriculum characteristics, processes and outcomes, and instruments assessing integration level.

The first category focuses on instruments evaluating students’ integration competency within the context of an OSCE assessment. ISPE , is a rubric that assesses integration competency through students’ responses to integration specific questions asked by the SP [ 26 ]. Integration competency fulfillment represents the ultimate objective of an integrated curriculum. Here, integration occurs within the student’s mind as they synthesize all acquired knowledge and skills from across the curriculum. The comprehensive understanding enables the students to effectively apply the integrated knowledge in making informed decisions during clinical practice. This also represents the highest level of integration in both Harden’s and Fogarty’s integration models; known as the transdisciplinary level (or fusion) and the immersed model, respectively [ 10 , 11 ]. The ISPE rubric stands out as a highly effective tool for evaluating curriculum integration due to its unique focus on assessing the outcome of integration. By objectively examining students’ integrative capacity and their proficiency in applying knowledge and skills across diverse contexts within the curriculum, including real-world scenarios, the ISPE rubric offers a comprehensive evaluation framework. Its emphasis on evaluating not just the process of integration, but also its tangible impact on students’ abilities to navigate real-world challenges, underscores the robustness and relevance of the ISPE rubric in educational assessment. This makes it a valuable resource for educators measuring the effectiveness of curriculum integration efforts and the practical readiness of students for professional practice [ 26 ].

Intriguingly, student performance in the ISPE was also evaluated using questionnaires designed to solicit feedback from students and other observers who have witnessed the interaction. Considering the importance of reflection as an effective strategy for enriching the learning experience in complex subjects and fostering a deeper understanding of professional values [ 43 ]. The student feedback questionnaire is an excellent instrument which provides students with opportunity to reflect on their performance and determine areas requiring improvement.

The second category of the integration assessing instruments is the largest; containing 16 instruments developed for students, faculty members, and other academic staff. These instruments explore participants’ views on the integrated curriculum, its characteristics, implementation, and outcomes. The characteristics of integrated curricula assessed were related to the content of the curriculum, delivery, teaching methods, and student assessment ( n  = 9). An important component of curriculum integration, which was evaluated by many of the identified instruments, was content coherence. Content coherence is a necessary pre-requisite for realizing full curriculum integration as the whole curriculum cannot be correlated and made more meaningful to the learner if individual components are not coherent [ 42 ]. Integration in education involves the seamless blending of existing knowledge with new learning, creating a cohesive and interconnected educational experience. This process emphasizes the importance of organizing curriculum content in a coherent manner, where different subjects, concepts, and skills are coordinated and presented in a unified framework. By aligning learning objectives, topics, and activities across various disciplines and modules, educators can facilitate a more holistic understanding of the subject matter. Content coherence ensures that students can recognize connections between different topics and apply their learning in diverse contexts, promoting deeper comprehension and more effective transfer of knowledge and skills. Additionally, it fosters a sense of coherence and continuity in the curriculum, enhancing the overall learning experience for students. The tools mentioned above can thus serve as a crucial framework for curriculum development and as a standard for improving curricular quality. Many of the instruments extracted in this study explored opinions on integrated curriculum delivery including teaching and learning methods, preparation for class, and interactions between students and faculty within class. These elements are important as they impact the extent to which integration is achieved within individual sessions and in the curriculum as a whole. Some of the items also assessed faculty development activities which prepare them to contribute to curriculum integration. Faculty members’ preparation for their role within an integrated program is crucial because the success of integration is impacted by the instructors’ understanding of their role, the role of others, and how they can coordinate with other faculty members to help students understand the link between different subjects and disciplines [ 42 , 44 ].

Two of the identified instruments; the integration characteristic tool [ 30 ] and the Basic Science Curriculum Assessment Instrument [ 34 ] only measure horizontal integration as they were developed to assess perceptions regarding integration in basic science curricula. These instruments could be used during initial stages of integration for institutions introducing horizontal integration within the basic science discipline. A study published by Brynhildsen J et al. (2002) has shown that both students and faculty members view horizontal and vertical integration as important components of medical curricula, with a general belief that horizontal integration might be more important [ 45 ]. Revisiting foundational knowledge from basic sciences during clinical courses and practical experiences results in deeper understanding of basic knowledge, especially when it is linked with real life applications.

The implementation of integrated curricula was assessed in two of the instruments; The Integrated Curriculum Implementation Inventory by Strandell C. (1980) [ 37 ], and the Students’ and Faculty’s Online Survey Questionnaires developed by Le B. (2018) [ 31 ]. The aforementioned tools included meticulously crafted questions assessing how integration was implemented before and during class, as well as during lectures and student assessments. This is very useful in providing valuable insights into the effectiveness and thoroughness of integration efforts within the curriculum. Additionally, these instruments also evaluated the preparedness of faculty members to facilitate curriculum integration, a crucial aspect in integrated medical curricula [ 44 ]. .

The last category of the instruments identified by this review evaluate the degree or level of curriculum integration. Two of these instruments; The Integrated Curriculum Evaluation Instrument by Howard et al. (2009) [ 41 ] and the Integration Ladder Questionnaire [ 39 ] were developed based on Fogarty’s model for curriculum integration and Harden’s integration ladder, respectively [ 10 , 11 ]. While Howard’s Integrated curriculum evaluation instrument assesses the extent of vertical and horizontal integration, its main limitation is that it provides a qualitative assessment of the curriculum which cannot be standardized and might be more susceptible to bias, and its use is restricted to course directors [ 41 ]. The integration Ladder Questionnaire [ 39 ] is a user-friendly, quantitative instrument whose target audience are educators and faculty members. This questionnaire consists of 11 close-ended questions, each of which represents a step on the integration ladder. The lowest step of the ladder represents a subject-based curriculum, with increased integration as you move you up the ladder. This instrument has been ratified for validity and reliability measures, and was found to have adequate internal consistency. The SIT is the last instrument in this category, which assesses integration within a single session. This is a quantitative instrument which has a simple and clear criterion to assess the degree of curriculum integration within a session [ 27 ]. This instrument was assessed for content and construct validity, inter-rater reliability, and factor analysis.

Our systematic review reveals that the topic of integration assessing instruments’ development is an active research area, which has become popular with increased adoption of integrated educational models. More than half of the studies included in this review were published after 2010. Although many instruments have been identified in the literature, only a few were validated to assess their psychometric properties. This is a limitation for the use of these instruments for future curriculum evaluation since their reliability and validity is still unknown. Therefore, this is an area which needs to be explored and studied further in future.

Notably, the majority of the identified assessment tools are limited in their ability to measure the degree of curriculum integration. Rather than providing quantitative evaluations of integration levels, these tools primarily focused on gathering qualitative data in the form of perceptions and feedback from stakeholders. This sheds the light on a substantial gap in the field of integrative medical curriculum assessment. The identified shortcoming necessitates the development of more robust evaluation methods capable of quantitatively assessing the degree of integration within medical curricula. Addressing this gap is essential for ensuring that educational programs effectively meet the goals of integration and adequately prepare students for the complex challenges of contemporary healthcare practice. To our knowledge, this systematic review is the first study in the literature to identify and describe the characteristics of tools assessing the degree of curriculum integration. Our search was comprehensive and included 8 major academic databases, in addition to further manual searching, although there is a possibility that relevant studies published in languages other than English might have been missed. In addition, qualitative instruments that assess curriculum integration were not included as our review focused on identifying quantitative instruments. Our inclusion criteria were very specific, excluding instruments not assessing the degree of curriculum integration such as those focused on assessing PBL [ 46 , 47 , 48 , 49 ]. Our study revealed the scarcity of validated instruments assessing curriculum integration in the literature, which highlights the need for more validation studies on currently available instruments.

Curriculum integration is a contemporary concept in the field of medical education which has been widely adopted by different health professions schools globally to optimize the students’ educational experience and prepare them for practice. To assess the extent to which integration has been fulfilled within these curricula, different instruments were developed. Our study aimed to identify these instruments and extract their psychometric properties. The results of this systematic review report on numerous instruments designed to assess the extent of curriculum integration within health professions’ educational programs. The majority of these instruments explore participants’ perceptions of the characteristics of the integrated curriculum including assessment of curricular content, delivery, and implementation. This study also identified tools which provide a broad approach for integration score calculation and determination of integration level. It is important to note that the majority of these instruments have not been validated and therefore further assessment of their psychometric properties is required. Furthermore, there is a necessity to create instruments that are both sensitive and specific, and are tailored to accurately gauge the level of curriculum integration within medical curriculum.

Data availability

Data is provided within the manuscript or supplementary information files.

Abbreviations

Case Based Learning

Integrated Learning Curriculum

The Integrated Standardized Patient Examination

Large Group Interactive Sessions

Bachelor of Medicine, Bachelor of Surger

Multiple Choice Questions

Not Reported

Objective Structured Clinical Examination

Problem based Learning

The Preferred Reporting Items for Systematic reviews and Meta-Analyses

Self- directed Learning

Small Group Discussions

Session Integration Tool

Student-centered, Problem-based, Integrated, Community-based, Elective, Systematic

Team Based Learning

Brauer DG, Ferguson KJ. The integrated curriculum in medical education: AMEE guide 96. Med Teach. 2015;37(4):312–22.

Article   Google Scholar  

Quintero GA, Vergel J, Arredondo M, Ariza MC, Gómez P, Pinzon-Barrios AM. Integrated medical curriculum: advantages and disadvantages. J Med Educ Curric Dev. 2016;3:133–7.

Google Scholar  

Matinho D, Pietrandrea M, Echeverria C, Helderman R, Masters M, Regan D, et al. A systematic review of integrated learning definitions, frameworks, and practices in recent health professions education literature. Educ Sci (Basel). 2022;12(3):165.

Harden RM, Sowden S, Dunn WR. Educational strategies in curriculum development: the SPICES model. Med Educ. 1984;18:284–97.

Wijnen-Meijer M, van den Broek S, Koens F, ten Cate O. Vertical integration in medical education: the broader perspective. BMC Med Educ. 2020;20:509.

Bandiera G, Boucher A, Neville A, Kuper A, Hodges B. Integration and timing of basic and clinical sciences education. Med Teach. 2013;35(5):381–7.

Harden RM. The integration ladder: a tool for curriculum planning and evaluation. Med Educ. 2000;34(7):551–7.

Loftus S. Understanding integration in Medical Education. Med Sci Educ. 2015;25(3):357–60.

Savery JR. The process and structure of problem-based learning. In: Walker A, Hmelo-Silve C, Heather Leary H, Ertmer PA, editors. Essential readings in problem-based learning: exploring and extending the legacy of Howard S Barrows. Indiana, United States: Purdue University; 2015.

Harden RM. The integration ladder: a tool for curriculum planning and evaluation. Med Educ. 2000;34:551–7.

Fogarty R. Ten ways to integrate curriculum. Integrating Curriculum. 1991:61–5.

Atwa HS, Gouda EM. Curriculum integration in medical education: a theoretical review. Intel Prop Rights. 2014;2(2):113.

Wijnen-Meijer M, Cate OTJ, Van Der Ten M, Borleffs JCC. Vertical integration in medical school: effect on the transition to postgraduate training. Med Educ. 2010 [cited 2024 Jan 1];44:272–9. https://asmepublications.onlinelibrary.wiley.com/doi/ https://doi.org/10.1111/j.1365-2923.2009.03571.x

Mohamed RHB, Jarrar M, Abumadini MS, Al Elq A, Abusalah MAH, Al-Bsheish M, et al. Student perspectives and academic achievement in a traditional versus an integrated curriculum: evidence from a medical school. Health Professions Educ. 2023;9(4):180–90.

Schmidt HG, Machiels-Bongaerts M, Hermans H, ten Cate T, Venekampm Ruud, Boshuizen HPA. The development of diagnostic competence: comparison of a problem-based, an integrated, and a conventional medical curriculum. Acad Med. 1996;71(6):658–64.

Abbas S, Sadiq N, Zehra T, Ullah I, Adeeb H. Comparison of performance of undergraduate medical students trained in conventional and integrated curriculums. Int J Acad Med. 2022;8(2):109–15.

Trullàs JC, Blay C, Sarri E, Pujol R. Effectiveness of problem-based learning methodology in undergraduate medical education: a scoping review. BMC Med Educ. 2022;22(1):104.

Crowell S. A new way of thinking: the challenge of the future. Educ Lead. 1989;7(1):60–3.

Kaufman D, Mann K. Teaching and learning in medical education: how theory can inform practice. In: Swansick T, editor. Understanding medical education: evidence, theory and practice. West Sussex, UK: Wiley-Blackwell; 2010. pp. 7–30.

Shoemaker B. Integrative education: a curriculum for the twenty first century. OSSC Bull. 1989;33(2).

Al-Jufairi Z, Sequeira R. Integrated curriculum and assessment in a problem based learning medical curriculum: faculty attitudes, perceived barriers and suggested strategies. J Bahrain Med Soc. 2010;22:31–7.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Mendeley Ltd. Mendeley Reference Manager. Elsevier; 2024.

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. BioMed Cent. 2016;5(1):210.

Nudelman G, Otto K. The development of a new generic risk-of-bias measure for systematic reviews of surveys. Methodology. 2020;16(4):278–98.

Panzarella KJ. Assessing clinical competency in the health sciences. [Buffalo]: The State University of New York; 2003. https://search.proquest.com/openview/b5cb561813630f47891e22c1721dcfa0/1?pq-origsite=gscholar&cbl=18750&diss=y

Heck AJ, Chase AJ. A tool for evaluating session-level integration in medical education. Med Sci Educ. 2021;31(2):647–54.

Lajber M, Mahboob U, Lajber F, Khan M, Waseem Badshah Bukhari S. Student’s perception regarding an integrated curriculum at a public sector medical college. PJMHS. 2020;14(3):1196–9. https://pjmhsonline.com/2020/july-sep/1196.pdf .

Ghayur S, Rafi S, Haroon Khan A, Ahmad RN, Iqbal M. Delivering endocrinology and reproduction in an integrated modular curriculum. J Pak Med Assoc. 2012;62(9):937–41. https://www.researchgate.net/publication/233392213

Maharjan B, Bhandary S, Upadhyay S, Ghimire S, Shrestha I, Joshi M et al. Developing tool and measuring integration characteristics of basic science curriculum to improve curriculum integration. Kathmandu Univ Med J. 2018;64(4):338–44. https://www.researchgate.net/publication/333162902

Le BK. Evaluation of the first curriculum year of the new integrated and interactive curriculum at the University of Medicine and Pharmacy at Hochiminh city (UMP), Vietnam [Degree of Master of Medical sciences in Medical Education]. [Boston]: The Harvard Medical School; 2018.

Parikh N, Risinger D, Holland JN, Molony DA, van der Hoeven D. Evaluating dental students’ perspectives on the concurrent teaching of didactic and case-based courses. J Dent Educ. 2022;86(12):1643–52.

Engel-Hills PC, Lehman B. An integrated learning curriculum for radiography in South Africa [Doctor of technology-radiography degree]. Cape peninsula university of technology; 2005.

Chen CK, Horner AM, Mcclure SC, Scott M. Evaluating the construct validity of basic science curriculum assessment instrument for critical thinking: a case-study. J Syst Cybern Inf. 2018;16(1):87–92.

Dhonde S, Belwalkar GJ, Jagtap PE, Kadam YR, Parekh M, Mudiraj N, et al. Evaluation of integrated teaching method for phase I MBBS, using Kirkpatrick’s evaluation method. Natl J Integr Res Med. 2020;11(2):83–9.

Nayak KR, Nayak V, Punja D, Badyal DK, Modi JN. Simulated patient videos to supplement integrated teaching in competency based undergraduate medical curriculum. Adv Physiol Educ. 2023;47(2):296–306.

Strandell CH. Nursing faculty perceptions of an integrated curriculum and implementation of the curriculum [Degree doctor of Philosophy Field of Education ]. [Evanston, Illinois]: Northwestern University; 1980.

Islam MA, Talukder RM, Taheri R, Blanchard N. Integration of basic and clinical science courses in US PharmD programs. Am J Pharm Educ. 2016;80(10).

Baig N, Siddiqui F, Baig MAM, Khursheed I, Meah KMS. Level of integration in current undergraduate curricula of two private-sector medical colleges in Karachi. Can Med Educ J. 2022.

Poirier TI, Fan J, Nieto MJ. Survey of pharmacy schools’ approaches and attitudes toward curricular integration. Am J Pharm Educ. 2016;80(6).

Howard KM, Stewart T, Woodall W, Kingsley K, Ditmyer M. An integrated curriculum: evolution, evaluation, and future direction. J Dent Educ. 2009;73(8):962–71.

Bandaranayake RC. The Integrated medical curriculum. 1st ed. London: Radcliffe Publishing; 2011.

Winkel AF, Yingling S, Jones AA, Nicholson J. Reflection as a learning tool in graduate medical education: a systematic review. J Grad Med Educ. 2017;9(4):430–9.

Husain M, Khan S, Badyal D. Integration in medical education. Indian Pediatr. 2020;57:842–7.

Brynhildsen J, Dahle LO, Fallsberg MB, Rundquist I, Hammar M. Attitudes among students and teachers on vertical integration between clinical medicine and basic science within a problem-based undergraduate medical curriculum. Med Teach. 2002;24(3):286–8.

Hassan S. The perceptions of second year medical students towards the problem-based curriculum as compared to the traditional curriculum. University of South Africa; 1996.

Aldayel AA, Alali AO, Altuwaim AA, Alhussain HA, Aljasser KA, Bin Abdulrahman KA, et al. Problem-based learning: medical students’ perception toward their educational environment at Al-imam Mohammad Ibn Saud Islamic University. Adv Med Educ Pract. 2019;10:95–104.

Ekelund L, Elzubeir M. Diagnostic radiology in an integrated curriculum: evaluation of student appraisal. Acad Radiol. 2000;7:965–70.

Sami Hussein K. Perceptions of an integrated curriculum among dental students in a public university in Saudi Arabia. Electron Physician. 2017;9(7):4828–34.

Download references

Acknowledgements

The Open Access publication of this article was funded by QU Health at Qatar University.

This study was funded by a collaborative grant (QUCG-CMED-22/23–470) from Qatar University.

Author information

Soumaya Allouch and Raja Mahamade Ali contributed equally to this work.

Authors and Affiliations

Basic Medical Sciences Department, College of Medicine, QU Health, Qatar University, PO Box 2713, Doha, Qatar

Soumaya Allouch, Raja Mahamade Ali, Michail Nomikos & Marwan F. Abu-Hijleh

College of Education, Qatar University, PO Box 2713, Doha, Qatar

Noor Al-Wattary

You can also search for this author in PubMed   Google Scholar

Contributions

M.A-H. and M.N. contributed to the conception and design of the work and revised the manuscript. S.A. and R.A. collected and screened the papers, extracted the relevant information, interpreted the data, and drafted the manuscript. N.A. contributed to the data collection, interpretation and revised the manuscript. All authors helped revise the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Marwan F. Abu-Hijleh .

Ethics declarations

Ethics approval and consent to participate.

Not applicable to this study.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Allouch, S., Ali, R.M., Al-Wattary, N. et al. Tools for measuring curriculum integration in health professions’ education: a systematic review. BMC Med Educ 24 , 635 (2024). https://doi.org/10.1186/s12909-024-05618-5

Download citation

Received : 29 February 2024

Accepted : 29 May 2024

Published : 06 June 2024

DOI : https://doi.org/10.1186/s12909-024-05618-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Integrated curriculum
• Questionnaire
• Health professions
• Medical education

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]