how to conduct an academic literature review

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Get science-backed answers as you write with Paperpal's Research feature

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 

How to write a good literature review 

• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• How to write a literature review faster with Paperpal? 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 

Find academic papers related to your research topic faster. Try Research on Paperpal  

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

Strengthen your literature review with factual insights. Try Research on Paperpal for free!    

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Write and Cite as you go with Paperpal Research. Start now for free.   

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!  

How to write a literature review faster with Paperpal?

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface with the option to save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.   

Here’s how to use the Research feature:  

• Ask a question: Get started with a new document on paperpal.com. Click on the “Research” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations. 
• Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access. 
• Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.  

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• How Long Should a Chapter Be?
• How to Use Paperpal to Generate Emails & Cover Letters?

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, how to structure an essay, leveraging generative ai to enhance student understanding of..., what’s the best chatgpt alternative for academic writing, how to write a good hook for essays,..., addressing peer review feedback and mastering manuscript revisions..., how paperpal can boost comprehension and foster interdisciplinary..., what is the importance of a concept paper..., how to write the first draft of a..., mla works cited page: format, template & examples, how to ace grant writing for research funding....

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

The Sheridan Libraries

• Write a Literature Review
• Sheridan Libraries
• Find This link opens in a new window
• Evaluate This link opens in a new window

What Will You Do Differently?

Please help your librarians by filling out this two-minute survey of today's class session..

Professor, this one's for you .

Introduction

Literature reviews take time. here is some general information to know before you start.  .

•  VIDEO -- This video is a great overview of the entire process.  (2020; North Carolina State University Libraries) --The transcript is included --This is for everyone; ignore the mention of "graduate students" --9.5 minutes, and every second is important  
• OVERVIEW -- Read this page from Purdue's OWL. It's not long, and gives some tips to fill in what you just learned from the video.  
• NOT A RESEARCH ARTICLE -- A literature review follows a different style, format, and structure from a research article.  

Steps to Completing a Literature Review

• Next: Find >>
• Last Updated: Sep 26, 2023 10:25 AM
• URL: https://guides.library.jhu.edu/lit-review

• Conferences
• Editorial Process
• Recent Advances
• Sustainability
• Academic Resources

How To Conduct A Literature Review For Your Thesis

Literature reviews are a vital part of academic theses. They help students develop knowledge of their chosen fields and identify boundaries, gaps, and themes in the literature. Conducting these reviews effectively is paramount to success in academia, giving students the chance to think critically about their subject matter and further the field.

Here, we look into how to conduct a successful literature review and thus write a high-quality thesis.

1. Define the scope of your review

Once you’ve decided on a specific research question, you have to decide on the comprehensiveness of your literature review. There are many factors you must decide on before you begin: What specific topics will you cover or omit? What level of detail is required for each? Is there any work from other disciplines that could be relevant?

Deciding on these parameters will provide an invaluable framework to structure your review and should hopefully lend the resulting work a high level of clarity.

2. Determine keywords and search terms

Most people’s first port of call when searching the literature is trawling through large databases of research. Without specific search terms, however, you can find yourself contending with thousands of articles only tangentially related to your work.

By ensuring your keywords are focused and specific, you can save crucial time and ensure the results contain the most relevant articles to your research. Making them too specific, however, can filter out valuable papers. So, testing out and revising your list is crucial to strike the right balance.

3. Filter your search results

After you’ve conducted your preliminary search, you can then begin refining your results. Reading the abstracts of the shortlisted papers will enable you to quickly determine their relevance to your research question. Thus, you can methodically make your way through the list and use only the most applicable as the foundation of your work.

It’s also important to make sure that you factor in the dates the papers were published. Some fields revolve around cutting-edge research, so making sure you’re only incorporating the most up-to-date information is crucial. Alternatively, for other subjects it may be more suitable to include older research.

Also, looking at the reference lists of relevant papers can also help you find related articles that you may otherwise have missed. This is a good way to identify key authors and citations and build a web of high-quality, relevant research.

4. Critically evaluate sources

Assessing the validity of the papers you have shortlisted will allow you to identify potential limitations in your own work, gaps in the field, and areas for future research. It also means you’re only using the very best research to build upon.

Additionally, making detailed notes and putting it into your own words further consolidates your knowledge and understanding of your topic.

5. Identify overarching themes

Seeing where ideas and different papers interlink is crucial. This will allow you to group and organise concepts so your work is structured logically and flows effortlessly from one point to the next. A coherent structure aids comprehension, meaning readers come away with a better understanding of your field.

Identifying trends and patterns is also crucial for forecasting areas of future research, and can help you guide future researchers to valuable results.

For more information on planning your thesis or dissertation, read our previous articles on and 5 tips to help you write your dissertation  and the  b enefits of getting your thesis proofread.

Once you’ve conducted your literature review and finished your thesis, the next step is editing your work. We offer our language editing services to students completing their theses. We provide a comprehensive language edit, correcting grammar, punctuation, and phrasing.

Our team of highly skilled English editors have edited over 60,000 papers, with a 97% author satisfaction rate. Our services are available to both MDPI authors and those publishing with other journals. Visit the link above to get a free quote today.

Related posts

Open Science , Academic Resources , Open Access

Open Access in South Korea

Open Science , Academic Resources , Journals

Why Journal Metrics are Important and How to Use Them

Open Science , Academic Resources , English Resources

Author Services Guide To Word Classes

Rising Importance of Preprints for Open Access Funders

All You Need to Know About Open Access

Open Access in France

Why Figures Are Important In Academic Research

Author Services Guide To Prepositions

Open Access in India

Author Services Guide To British And American English

Add comment cancel reply.

Save my name, email, and website in this browser for the next time I comment.

Privacy Preference Center

Privacy preferences.

From sharing the latest MDPI blog news, to showcasing our most popular articles, the newsletters will keep you in the loop with everything good going on in the science world.

• Schools & departments

Literature review

A general guide on how to conduct and write a literature review.

Please check course or programme information and materials provided by teaching staff, including your project supervisor, for subject-specific guidance.

What is a literature review?

A literature review is a piece of academic writing demonstrating knowledge and understanding of the academic literature on a specific topic placed in context.  A literature review also includes a critical evaluation of the material; this is why it is called a literature review rather than a literature report. It is a process of reviewing the literature, as well as a form of writing.

To illustrate the difference between reporting and reviewing, think about television or film review articles.  These articles include content such as a brief synopsis or the key points of the film or programme plus the critic’s own evaluation.  Similarly the two main objectives of a literature review are firstly the content covering existing research, theories and evidence, and secondly your own critical evaluation and discussion of this content. 

Usually a literature review forms a section or part of a dissertation, research project or long essay.  However, it can also be set and assessed as a standalone piece of work.

What is the purpose of a literature review?

…your task is to build an argument, not a library. Rudestam, K.E. and Newton, R.R. (1992) Surviving your dissertation: A comprehensive guide to content and process. California: Sage, p49.

In a larger piece of written work, such as a dissertation or project, a literature review is usually one of the first tasks carried out after deciding on a topic.  Reading combined with critical analysis can help to refine a topic and frame research questions.  Conducting a literature review establishes your familiarity with and understanding of current research in a particular field before carrying out a new investigation. After doing a literature review, you should know what research has already been done and be able to identify what is unknown within your topic.

When doing and writing a literature review, it is good practice to:

• summarise and analyse previous research and theories;
• identify areas of controversy and contested claims;
• highlight any gaps that may exist in research to date.

Conducting a literature review

Focusing on different aspects of your literature review can be useful to help plan, develop, refine and write it.  You can use and adapt the prompt questions in our worksheet below at different points in the process of researching and writing your review.  These are suggestions to get you thinking and writing.

Developing and refining your literature review (pdf)

Developing and refining your literature review (Word)

Developing and refining your literature review (Word rtf)

Writing a literature review has a lot in common with other assignment tasks.  There is advice on our other pages about thinking critically, reading strategies and academic writing.  Our literature review top tips suggest some specific things you can do to help you submit a successful review.

Literature review top tips (pdf)

Literature review top tips (Word rtf)

Our reading page includes strategies and advice on using books and articles and a notes record sheet grid you can use.

Reading at university

The Academic writing page suggests ways to organise and structure information from a range of sources and how you can develop your argument as you read and write.

Academic writing

The Critical thinking page has advice on how to be a more critical researcher and a form you can use to help you think and break down the stages of developing your argument.

Critical thinking

As with other forms of academic writing, your literature review needs to demonstrate good academic practice by following the Code of Student Conduct and acknowledging the work of others through citing and referencing your sources.  

Good academic practice

As with any writing task, you will need to review, edit and rewrite sections of your literature review.  The Editing and proofreading page includes tips on how to do this and strategies for standing back and thinking about your structure and checking the flow of your argument.

Editing and proofreading

Guidance on literature searching from the University Library

The Academic Support Librarians have developed LibSmart I and II, Learn courses to help you develop and enhance your digital research skills and capabilities; from getting started with the Library to managing data for your dissertation.

Searching using the library’s DiscoverEd tool: DiscoverEd

Finding resources in your subject: Subject guides

The Academic Support Librarians also provide one-to-one appointments to help you develop your research strategies.

1 to 1 support for literature searching and systematic reviews

Advice to help you optimise use of Google Scholar, Google Books and Google for your research and study: Using Google

Managing and curating your references

A referencing management tool can help you to collect and organise and your source material to produce a bibliography or reference list. 

Referencing and reference management

Information Services provide access to Cite them right online which is a guide to the main referencing systems and tells you how to reference just about any source (EASE log-in may be required).

Cite them right

Published study guides

There are a number of scholarship skills books and guides available which can help with writing a literature review.  Our Resource List of study skills guides includes sections on Referencing, Dissertation and project writing and Literature reviews.

Study skills guides

This article was published on 2024-02-26

• UWF Libraries

Literature Review: Conducting & Writing

• Steps for Conducting a Lit Review

1. Choose a topic. Define your research question.

2. decide on the scope of your review., 3. select the databases you will use to conduct your searches., 4. conduct your searches and find the literature. keep track of your searches, 5. review the literature..

• Finding "The Literature"
• Organizing/Writing
• APA Style This link opens in a new window
• Chicago: Notes Bibliography This link opens in a new window
• MLA Style This link opens in a new window
• Sample Literature Reviews

Disclaimer!!

Conducting a literature review is usually recursive, meaning that somewhere along the way, you'll find yourself repeating steps out-of-order.

That is actually a good sign.  

Reviewing the research should lead to more research questions and those questions will likely lead you to either revise your initial research question or go back and find more literature related to a more specific aspect of your research question.

Your literature review should be guided by a central research question.  Remember, it is not a collection of loosely related studies in a field but instead represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor.

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

Tip: This may depend on your assignment.  How many sources does the assignment require?

Make a list of the databases you will search.  Remember to include comprehensive databases such as WorldCat and Dissertations & Theses, if you need to.

Where to find databases:

• Find Databases by Subject UWF Databases categorized by discipline
• Find Databases via Research Guides Librarians create research guides for all of the disciplines on campus! Take advantage of their expertise and see what discipline-specific search strategies they recommend!
• Review the abstracts of research studies carefully. This will save you time.
• Write down the searches you conduct in each database so that you may duplicate them if you need to later (or avoid dead-end searches   that you'd forgotten you'd already tried).
• Use the bibliographies and references of research studies you find to locate others.
• Ask your professor or a scholar in the field if you are missing any key works in the field.
• Use RefWorks to keep track of your research citations. See the RefWorks Tutorial if you need help.

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions. Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited?; if so, how has it been analyzed?

Tips: 

• Again, review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• << Previous: Home
• Next: Finding "The Literature" >>
• Last Updated: Mar 22, 2024 9:37 AM
• URL: https://libguides.uwf.edu/litreview

News alert: UC Berkeley has announced its next university librarian

Secondary menu

• Log in to your Library account
• Hours and Maps
• Connect from Off Campus
• UC Berkeley Home

Search form

Conducting a literature review: why do a literature review, why do a literature review.

• How To Find "The Literature"
• Found it -- Now What?

Besides the obvious reason for students -- because it is assigned! -- a literature review helps you explore the research that has come before you, to see how your research question has (or has not) already been addressed.

You identify:

• core research in the field
• experts in the subject area
• methodology you may want to use (or avoid)
• gaps in knowledge -- or where your research would fit in

It Also Helps You:

• Publish and share your findings
• Justify requests for grants and other funding
• Identify best practices to inform practice
• Set wider context for a program evaluation
• Compile information to support community organizing

Great brief overview, from NCSU

Want To Know More?

• Next: How To Find "The Literature" >>
• Last Updated: Apr 25, 2024 1:10 PM
• URL: https://guides.lib.berkeley.edu/litreview

New change to library operations

All Main Library and Weaver Library doors lock 15 minutes before closing.

Conduct a literature review

What is a literature review.

A literature review is a summary of the published work in a field of study. This can be a section of a larger paper or article, or can be the focus of an entire paper. Literature reviews show that you have examined the breadth of knowledge and can justify your thesis or research questions. They are also valuable tools for other researchers who need to find a summary of that field of knowledge.

Unlike an annotated bibliography, which is a list of sources with short descriptions, a literature review synthesizes sources into a summary that has a thesis or statement of purpose—stated or implied—at its core.

How do I write a literature review?

Step 1: define your research scope.

• What is the specific research question that your literature review helps to define?
• Are there a maximum or minimum number of sources that your review should include?

Ask us if you have questions about refining your topic, search methods, writing tips, or citation management.

Step 2: Identify the literature

Start by searching broadly. Literature for your review will typically be acquired through scholarly books, journal articles, and/or dissertations. Develop an understanding of what is out there, what terms are accurate and helpful, etc., and keep track of all of it with citation management tools . If you need help figuring out key terms and where to search, ask us .

Use citation searching to track how scholars interact with, and build upon, previous research:

• Mine the references cited section of each relevant source for additional key sources
• Use Google Scholar or Scopus to find other sources that have cited a particular work

Step 3: Critically analyze the literature

Key to your literature review is a critical analysis of the literature collected around your topic. The analysis will explore relationships, major themes, and any critical gaps in the research expressed in the work. Read and summarize each source with an eye toward analyzing authority, currency, coverage, methodology, and relationship to other works. The University of Toronto's Writing Center provides a comprehensive list of questions you can use to analyze your sources.

Step 4: Categorize your resources

Divide the available resources that pertain to your research into categories reflecting their roles in addressing your research question. Possible ways to categorize resources include organization by:

• methodology
• theoretical/philosophical approach

Regardless of the division, each category should be accompanied by thorough discussions and explanations of strengths and weaknesses, value to the overall survey, and comparisons with similar sources. You may have enough resources when:

• You've used multiple databases and other resources (web portals, repositories, etc.) to get a variety of perspectives on the research topic.
• The same citations are showing up in a variety of databases.

Additional resources

Undergraduate student resources.

• Literature Review Handout (University of North Carolina at Chapel Hill)
• Learn how to write a review of literature (University of Wisconsin-Madison)

Graduate student and faculty resources

• Information Research Strategies (University of Arizona)
• Literature Reviews: An Overview for Graduate Students (NC State University)
• Oliver, P. (2012). Succeeding with Your Literature Review: A Handbook for Students [ebook]
• Machi, L. A. & McEvoy, B. T. (2016). The Literature Review: Six Steps to Success [ebook]
• Graustein, J. S. (2012). How to Write an Exceptional Thesis or Dissertation: A Step-by-Step Guide from Proposal to Successful Defense [ebook]
• Thomas, R. M. & Brubaker, D. L. (2008). Theses and Dissertations: A Guide to Planning, Research, and Writing

Literature Review

• Steps for Conducting a Lit Review
• Finding "The Literature"
• Organizing/Writing
• Sample Literature Reviews
• FAMU Writing Center

1. Choose a topic. Define your research question.

Your literature review should be guided by a central research question.  Remember, it is not a collection of loosely related studies in a field but instead represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor.

2. Decide on the scope of your review.

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

Tip: This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

• Look at the Library's research guides in your discipline to select discipline-specific databases.  Don't forget to look at books!
• Make an appointment with or contact your   subject librarian to make sure you aren't missing major databases.

4. Conduct your searches and find the literature. Keep track of your searches!

Tips: 

• Review the abstracts of research studies carefully. This will save you time.
• Write down the searches you conduct in each database so that you may duplicate them if you need to later (or avoid dead-end searches   that you'd forgotten you'd already tried).
• Use the bibliographies and references of research studies you find to locate others.
• Ask your professor or a scholar in the field if you are missing any key works in the field.
• Use RefWorks to keep track of your research citations. See the RefWorks Tutorial if you need help.

5. Review the literature.

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions. Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited?; if so, how has it been analyzed?
• Again, review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.

Composing your literature review

O nce you've settled on a general pattern of organization, you're ready to write each section. There are a few guidelines you should follow during the writing stage. Here is a sample paragraph from a literature review about sexism and language to illuminate the following discussion:

  However, other studies have shown that even gender-neutral antecedents are more likely to produce masculine images than feminine ones (Gastil, 1990). Hamilton (1988) asked students to complete sentences that required them to fill in pronouns that agreed with gender-neutral antecedents such as "writer," "pedestrian," and "persons." The students were asked to describe any image they had when writing the sentence. Hamilton found that people imagined 3.3 men to each woman in the masculine "generic" condition and 1.5 men per woman in the unbiased condition. Thus, while ambient sexism accounted for some of the masculine bias, sexist language amplified the effect. (Source: Erika Falk and Jordan Mills, "Why Sexist Language Affects Persuasion: The Role of Homophily, Intended Audience, and Offense," Women and Language19:2.

Use evidence

In the example above, the writers refer to several other sources when making their point. A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid.

Be selective

Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the review's focus, whether it is thematic, methodological, or chronological.

Use quotes sparingly

Falk and Mills do not use any direct quotes. That is because the survey nature of the literature review does not allow for in-depth discussion or detailed quotes from the text. Some short quotes here and there are okay, though if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Notice that Falk and Mills do quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. But if you find yourself wanting to put in more quotes, check with your instructor.

Summarize and synthesize

Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. The authors here recapitulate important features of Hamilton's study, but then synthesize it by rephrasing the study's significance and relating it to their own work.

Keep your own voice

While the literature review presents others' ideas, your voice (the writer's) should remain front and center. Notice that Falk and Mills weave references to other sources into their own text, but they still maintain their own voice by starting and ending the paragraph with their own ideas and their own words. The sources support what Falk and Mills are saying.

Use caution when paraphrasing

When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. In the preceding example, Falk and Mills either directly refer in the text to the author of their source, such as Hamilton, or they provide ample notation in the text when the ideas they are mentioning are not their own, for example, Gastil's. For more information, please see our handout on plagiarism .

• << Previous: Home
• Next: Finding "The Literature" >>
• Last Updated: Oct 20, 2022 11:24 AM
• URL: https://library.famu.edu/literaturereview

• Harvard Library
• Research Guides
• Faculty of Arts & Sciences Libraries

Chemistry and Chemical Biology Resources

Literature review.

• Getting Started
• Chemistry journals and databases
• Find Dissertations and Theses
• Find Conference Proceedings
• Find Technical Reports
• Managing Citations
• Research Data Management
• Managing Your Academic Identity  
• Helpful Tools

Reviewing the Literature: Why do it?

• Personal: To familiarize yourself with a new area of research, to get an overview of a topic, so you don't want to miss something important, etc.
• Required writing for a journal article, thesis or dissertation, grant application, etc.

Literature reviews vary; there are many ways to write a literature review based on discipline, material type, and other factors.

Background:

• Literature Reviews - UNC Writing Center
• Literature Reviews: An Overview for Graduate Students  - What is a literature review? What purpose does it serve in research? What should you expect when writing one? - NCSU Video

Where to get help (there are lots of websites, blogs , articles,  and books on this topic) :

• The Center for writing and Communicating Ideas (CWCI)
• (these are non-STEM examples: dissertation guidance , journal guidelines )
• How to prepare a scientific doctoral dissertation based on research articles (2012)
• Writing a graduate thesis or dissertation (2016)
• The good paper : a handbook for writing papers in higher education (2015)
• Proposals that work : a guide for planning dissertations and grant proposals (2014)
• Theses and dissertations : a guide to planning, research, and writing (2008)
• Talk to your professors, advisors, mentors, peers, etc. for advice

READ related material and pay attention to how others write their literature reviews:

• Dissertations
• Journal articles
• Grant proposals
• << Previous: Find Technical Reports
• Next: Managing Citations >>
• Last Updated: Sep 13, 2023 2:15 PM
• URL: https://guides.library.harvard.edu/CCB

Harvard University Digital Accessibility Policy

• Academic Writing Guides

How to Write a Literature Review? A Beginner’s Guide

Sooner or later in your academic path, you will be required to compose a literature review. So, it’s important to approach this task well-prepared and understand how to write a literature review inside out. 

Are you interested in how to write lit review projects correctly and cover the subject comprehensively, from all angles? This article will explore the concept of review of literature , dwell on how to write a literature review in line with your professor’s expectations, and share a universal literature review template for your usage. 

What Is the Purpose of a Literature Review? 

To understand what should be included in a literature review , you need to understand its purpose and value in a larger work. A well-researched and written lit review usually addresses the following objectives: 

• Inform . The fundamental purpose of any review of literature is to provide the foundation of knowledge on a specific topic or phenomenon. You explore what people have learned about it from prior studies and summarize those findings to inform your readers. 
• Give credit . Another purpose of a lit review is to identify researchers who have contributed to the advancement of research on your chosen literature review topic and have produced the most valuable findings. This way, you pay tribute to those researchers and showcase your knowledge of the most considerable influencers. 
• Identify gaps . By performing a thorough review of literature , you may not only discover what is known about your topic but also find out what it yet to be learned about it. As a result of reviewing the available evidence, you may identify gaps for addressing through your academic inquiry. 
• Identify patterns . Those who know how to write a literature review can also effectively embrace data trends and patterns in the collected dataset. As a result, they can present a more nuanced analysis of the existing knowledge in your literature review and uncover dependencies that inform people’s understanding of certain phenomena and processes.  
• Contextualize research . When you perform lit review writing, you can also create a spot for your own study within the broad field of your academic research interest. This way, you show to your readers that you can effectively navigate the landscape of your academic area. 

These purposes lay the foundation for understanding how to write a literature review that will attain all academic goals. You simply need to use this list as your checklist for structuring an impactful lit review and including all vital data in it. 

How to Write a Literature Review? 

Now, we come to the main topic of this article – how to write a good literature review for dissertation projects, research papers, and other works. Follow the steps we’ve covered below to arrive at a consistent, logical piece of lit review . 

Identify Relevant Sources 

Any literature review writing starts with academic research. You should look for sources that explore your topic from various angles and provide valuable literature review findings to expand your knowledge on the subject. It’s best to look for subject-specific books first and then go through academic databases that publish journal articles. This way, you will start with the evidence of the highest reliability level and move on to expand your literature review dataset conveniently. 

Screen Sources for Quality 

The best solution on how to write a literature review without challenges is to rely on high-quality evidence. Your task is to research extensively in reliable academic databases to find peer-reviewed academic journals and books written by experts in your field. Don’t over-rely on online sources in your literature review, like blogs or opinion pieces, because they rarely possess the needed degree of credibility for an academic review. By choosing only industry-approved sources from qualified professionals, you can build a solid foundation for your writing and impress the audience. 

Determine Data Patterns and Gaps 

How to write a literature review of value for your readers? One of the best approaches is to go beyond mere summarization of what other researchers have found on the subject and to apply critical thinking and data categorization. This way, you will manage to uncover existing patterns and trends and examine those dependencies in your literature review. A systematic, critical approach is always evaluated much higher than a simple outline of what people say on your subject. 

Draft an Outline 

Now, it’s time to compose an outline for a literature review . The outline should include the main concepts you’re planning to cover in the literature review text and should structure the narrative consistently. By means of composing an outline before the actual writing process, you give yourself a hands-on roadmap for composing a logically flowing piece. As a result of using an outline, you will write the literature review faster and will avoid the risk of going off-topic. 

Compose the Review 

With a good and detailed outline, you should have no more problems or concerns about how to write a literature review . The writing process should go quickly and smoothly when you have all your evidence at your fingertips, categorized by themes and requiring only proper summarization in the text. 

We recommend starting with a broad introduction to the topic and concepts related to it. You should give definitions and explain the topic’s features and components that require attention in the research process. After that, you may briefly outline the main sections of your review and then proceed to the exploration of each section in depth. 

At times, your professor will give you a specific structure for review writing – such as the general introduction, coverage of theories, and then coverage of empirical evidence. At times, it may be a review of the data search strategy and a report on the identified resources that follow. In any case, you should follow the tutor’s prompt closely to ensure compliance with the task. 

Make Use of This Generic Literature Review Template 

Looking for a universal, ready-to-use literature review template ? Here is an effective literature review template that everyone can apply with minor tweaks to produce a high-quality review of literature . 

LITERATURE REVIEW TEMPLATE 

Introduction 

• Introduce the topic of your literature review 
• Examine its significance for your academic area 
• Determine the scope of your literature review inquiry 
• Give a brief outline of subtopics and sections included in your literature review 

Body of the literature review

• Describe the subtopic and indicate how it relates to your literature review’s main idea
• Summarize the evidence available about it 
• Compare the available data and voice your opinion 

Conclusion 

• Summarize the main points and findings from your literature review 
• State the main contribution you have managed to achieve 
• Identify the research gaps your literature review has revealed 

Use this literature review template to pump your writing muscle and get ready for new literature review challenges. 

More Pro Tips for Writing a Literature Review 

If you’re still unsure about how to do a literature review with excellence, these pro tips may improve your understanding of this task type. 

• Mind the audience . Understanding how to do a literature review for a research paper often has little to do with how to write literature review for thesis . This difference is explained by the fact that these types of academic work are of different lengths and pursue different scholarly goals. This way, you may need to cover only some basic seminal research in the review of literature for a research paper but will need to dig deeper into theoretical and applied research with deeper analysis and more critical thinking when dealing with a thesis.
• Mind the length . How long should a literature review be ? This is a vital question that you should answer before starting the outlining and writing process. Ask your professor if you’re not sure or apply the rule of thumb, where this section usually takes from 15% to 25% of the entire paper. 
• Mind the structure . It’s important to cover all lit review aspects that your professor wants to see in the paper; otherwise, you risk getting a low grade even if your literature review is comprehensive and interesting. What should a literature review include ? In most cases, you will be required to cover some seminal research works in your literature review to show that you understand who the pioneers in the field are, and what contribution they have made to the topic’s exploration. Next, you should examine relevant theories that inform studies in your subject. At the end of the literature review, you should typically cite a variety of studies of applied nature, thus showing what empirical research is conducted in your academic field.  

With these recommendations at your disposal, you’re sure to become much more proficient in how to do a lit review . If you need more help with a literature review project, welcome to use our professional and quick literature review writing service . Our experts know everything about how to write a literature review , so they will handle your literature review task with ease within the timeframe you set for them.

• Essay Writing Guides

Systematic Literature Review: Easy Guide

WRONG. It turns out that typing “what is a systematic literature review” into Google will only overwhelm a new researcher! I came across plenty of journal articles that claimed to be explaining what an SLR was (and how that somehow differed from another term I was learning – a scoping review), but for the life of me I could not find a clear-cut set of instructions. All of the information seemed to be pitched at a level far above the one I was operating at, and I began to feel frustrated that I could not find a source that was putting this methodology into terms that the average person could understand. But I knew I needed to figure it out, so over the course of the next few weeks I read what felt like dozens of explainers and guides.

Eventually, my reading and furious note-taking paid off, because by the end of 2023 I had successfully completed my research, entitled “How are language barriers bridged in hospitals?: a systematic review” . But in the process, I had spoken to so many academics who also voiced their frustration that they couldn’t find explanations on how to conduct an SLR in clear lay terms, and so I knew I hadn’t been alone.

Something I feel VERY passionate about is that, as academics, we must be able to talk to people outside of academia, and that means that we need to be able to communicate complex ideas in easily digestible ways. Higher knowledge shouldn’t be reserved for people who have weeks to teach themselves a new research methodology, and I wanted to be able to explain an SLR to everyone, not just other researchers.

And so, I created this “ SLR: Easy Guide ” explainer for anyone and everyone who would like to conduct an SLR but has no idea where to start. If that’s you, please feel free to use this resource – and know that you aren’t alone as an early researcher who is learning things for the first time. We’ve all got to start somewhere, and we can make it easier on others by sharing what we’ve figured out the hard way!

What exactly is a systematic literature review (SLR)?

Ok, so you know how you need to do a literature review before you write a research paper? In that literature review, you are basically summarising what other researchers have said about your research topic so that you can show how your research is building on prior knowledge.

An SLR is different to that. An SLR is your research (your “experiment”, if you will). In an SLR, you read and analyse lots of different published journal articles in order to see patterns in already-published data. There’s an actual methodology that you have to use (which I detail in SLR: An Easy Guide ) in order to select these journal articles.

I haven’t heard of an SLR, but I’ve heard of a meta-analysis. What’s the difference?

Literally nothing. They mean the same thing! Surprise! Academia is fun and not at all confusing.

I’ve also heard of a scoping review. Is that the same as a systematic literature review?

In this case, there actually is a difference, albeit a relatively small one. The methodology for both types of reviews will be the same (whew!), but the reason for conducting one versus the other will be a bit different. Let me give you an example based on my own research. When I began looking into how hospitals manage linguistic diversity between patients and staff, I knew that there was already a lot of literature out there about the subject (generally having to do with the work of professional interpreters). I had four very specific research questions that I wanted to answer based on that literature. This is why I conducted a systematic review – because I already knew that I would be able to find existing research that could answer my questions.

HOWEVER, you might not know how much literature already exists on a given topic. Maybe your topic is fairly niche, so you haven’t seen much about it in publications. This is where a scoping review comes in. In conducting a scoping review, you’ll find out exactly how much literature on the topic already exists. In doing so, you’ll be able to make an argument for why a particular area of research should be looked into more.

If this still sounds confusing (totally understandable!), be sure to talk to a fabulous university librarian. They are really good at knowing the difference between the two!

Is there any kind of SLR “authority” that I should know about?

There sure is! There is an organisation called PRISMA (which stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses). You can go to their website for two very crucial items that you will need for your SLR: a checklist and a flow chart.

The PRISMA checklist is great because it tells you exactly what you need to include in your SLR. The PRISMA flow chart is what you include in your SLR to show why/how you included and excluded studies during your screening process (which you can see in steps 3 and 4 of my SLR: An Easy Guide  resource). But don’t worry, you don’t need to create the flow chart from scratch. If you use Covidence, the platform will create it for you. And speaking of Covidence…

This feels overwhelming! Is there one place I can go to manage all my SLR data easily?

Absolutely. I used Covidence , an online platform that essentially walks you through the SLR process. I would HIGHLY recommend using Covidence or a similar service to help you manage all your data in one place. Covidence will also automatically create your flow chart for you as you go through your screening process. What I especially liked about Covidence was that I was able to custom-create my data collection template based on my specific research questions. This made my data analysis much easier than it would have been without it!

What do I do if I’m still confused or feel like I don’t know how to do this?

Remember that every single one of us who goes on to do higher degree research feels like this. We don’t know what we don’t know! I’ve now completed two Masters degrees and am currently working on my PhD, and let me tell you, the learning curve is steep! But you know what? You can do it. Don’t be afraid to ask questions. Tell your supervisors and colleagues when you feel lost. Remind yourself that learning these research skills is just as important as the research itself. And when you get super stressed, grab a cup of coffee, stand in the sunshine and take a 10-minute break. You’ve got this!

Download and cite my free “SLR: An Easy Guide” resource

“ SLR: An Easy Guide ” is a free cheat sheet for your systematic literature review. You can download it here .

If you find it useful, please cite as:

Quick, B. (2024). Systematic Literature Review: An Easy Guide. Language on the Move . Retrieved from https://www.languageonthemove.com/systematic-literature-review-easy-guide

Next Post Life in a New Language, Part 2: Work

Related posts, because internet.

Is it ok for linguists to hate new words?

Community Languages Schools Transforming Education

Author Brynn Quick

Brynn Quick holds a Master of Applied Linguistics and a Master of Research from Macquarie University. For her PhD, also at Macquarie University, she is investigating how language barriers are bridged between patients and staff in Australian hospitals. Her linguistic interests are many and varied, and include sociolinguistics, anthropological linguistics, sociophonetics, and historical linguistics, particularly the history of English.

Join the discussion 3 Comments

This is really helpful! Thanks a lot!

Wow, Brynn! What a creative and relatable way to navigate an otherwise intimidating research method! Thanks for sharing your guide. Will definitely use it!

Thanks so much, Pia! I’m so glad you find it helpful!

Leave a Reply Cancel Reply

Save my name, email, and website in this browser for the next time I comment.

Notify me of follow-up comments by email.

Notify me of new posts by email.

UNTERSTÜTZT VON / SUPPORTED BY

NEWSLETTER SUBSCRIPTION

Be the first to know. Sign up now to stay updated on our news and latest posts.

Email Address

• Life in a New Language
• COVID-19 Crisis Communication
• Intercultural Communication
• Japanese on the Move
• Next Generation Literacies
• PhD Hall of Fame
• Research blog
• Language on the Move 2023
• Language on the Move 2022
• Language on the Move 2021
• Language on the Move 2020
• Language on the Move 2019
• Language on the Move 2018
• Language on the Move 2017
• Language on the Move 2016
• Language on the Move 2015
• Language on the Move 2014
• Language on the Move 2013
• Language on the Move 2012
• Language on the Move 2011
• Language on the Move 2010
• Language on the Move 2009
• Translators
• Web developer

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

A Systematic Review of Personal Information Sharing in Smart Cities: Risks, Impacts, and Controls

• Open access
• Published: 24 June 2024

Cite this article

You have full access to this open access article

• Maha Ibrahim Alabsi   ORCID: orcid.org/0000-0003-1791-6907 1 , 2 &
• Asif Qumar Gill   ORCID: orcid.org/0000-0001-6239-6280 1  

85 Accesses

Explore all metrics

Smart cities aim to deliver smart services that rely on emerging technologies to their users. In order for users to get the provided services, they need to share their personal information with different parties. However, sharing personal information in smart cities may impact the privacy of that information. Thus, there is a need to address privacy risks relevant to sharing personal information in smart cities. This study aims to address this issue by conducting a systematic literature review (SLR) to identify and extract privacy risks, impacts, and existing controls associated with sharing personal information, considering elements involved and interacting during the sharing activity in smart cities. A set of 83 selected studies in both academic and industry fields were reviewed, and the results were categorised into three main groups: privacy risks, impacts, and controls. Moreover, the implications and future research directions were also reported. The proposed privacy risk taxonomy will provide a much-needed foundation for the industry and research community, intending to research and evaluate privacy risk frameworks and design solutions for sharing personal information in smart cities.

Avoid common mistakes on your manuscript.

Introduction

More recently, the concept of smart cities has been getting significant attention from research and practice perspectives (Ahmad Mohamad et al., 2019 ). Several countries across the globe (e.g. Asia, Africa, America, and Europe) aim to consider their cities “smart” by developing and delivering smart services to their citizens and residents by using emerging ICT (information and communication technologies) (Ahmad Mohamad et al., 2019 ; Albino et al., 2015 ; Hsiao et al., 2021 ). The definitions of smart cities focus on the quality of citizens’ performance and activities, along with enhancing economic competitiveness by managing city resources and improving information and communications technology (ICT) infrastructure (Giffinger et al., 2007 , Caragliu et al. 2009 , Vu & Hartley, 2018 ). Also, smart city is defined as a 4th industrial model where emerging technologies, such as the Internet of Things, cloud computing, and big data, are used to optimise the cities (Safiullin et al., 2019 ). Accordingly, smart cities are proposed in particular areas or sectors such as governments, health, energy, buildings, airports, and businesses/organisations (Khatoun & Zeadally, 2017 ).

Due to the strong relationship between ICT and smart services within the overarching concept of smart cities, a vast amount of personal information is collected from users, devices, and applications (Martinez-Balleste et al., 2013 ). Furthermore, sharing and exchanging information among parties, including individuals and organisations, is possible using different sharing platforms that play a vital role in smart cities (Kong et al., 2018 , Kusumastuti et al., 2022 ). Internet of Things (IoT), Cloud, fog computing, and blockchain technology are examples of such platforms (Qian et al., 2018 , Imine et al., 2020 ; Gill, 2021 ). However, the flow of personal information in smart cities may result in individuals suffering from serious privacy risks that may impact their information (Martinez-Balleste et al., 2013 , Sharma et al., 2020 ).

According to NIST (Stoneburner et al., 2002 ), the risk is the possibility of a threat source exploiting a specific information system vulnerability and the resultant consequence. Assessing information privacy risks in smart cities is challenging due to information complexity and uncertain impact levels (Bogoda et al., 2019 ). In addition, privacy risks need to be assessed to minimise the risk impact by using appropriate controls (Hong et al., 2004 ). Thus, there is a need to assess privacy risks when sharing personal information in smart cities. This includes identifying and addressing privacy threats and vulnerabilities, their impacts, and appropriate privacy risk mitigation controls.

To the best of our knowledge, there is a lack of consolidated literature on this important topic of privacy assessments that cover privacy risks, impacts, and current controls for sharing personal information, considering the interaction among elements involved in sharing activity in smart cities. A consolidated view of the current work is needed to provide a foundation for further development in this important area of research.

Thus, this paper addresses this need by conducting a SLR and synthesising published research with a view to identify and extract privacy risks, impacts, existing controls, and elements involved and interacting to share personal information in smart cities, along with relevant regulation, to influence this activity. Thus, this paper focuses on the following key research questions:

RQ1: What are the privacy risks associated with sharing personal information in the context of smart cities considering the elements involved and interacting while sharing personal information?

RQ2: What are the impacts of those personal information privacy risks?

RQ3: What current privacy controls are in place to mitigate the identified risks?

This work builds on the earlier research on identifying privacy risks in smart airports (Alabsi & Gill, 2021 ). This paper extended this work to provide broader coverage of smart cities. This will help extract and define more comprehensive views of privacy risks, which will be used to design a holistic solution for assessing the privacy risks that may impact passengers’ personal information in their interaction journey in smart airports within the border context of smart cities. This will ensure that important privacy concerns are not overlooked when dealing with information privacy in smart airports. The main motivation behind this paper is the future development of the privacy framework in a smart airport context. The development of the proposed framework is beyond this paper’s scope and is subject to further research.

Contribution

The key contributions of this research are outlined below:

This paper provides an updated knowledge base covering various articles published in academic and industrial databases between 2017 and 2021, including smart cities, sharing information, privacy risk, impact, and existing control.

This paper provides both a theoretical and practical view of the review results by using the Adaptive EA and Concerns for Information Privacy framework (CFIP) as a theoretical lens and the NIST 800–30 framework as a practical lens. These lenses help identify the risk assessment components: privacy risk, the resulting impact, and current privacy control.

This paper contributes to enhancing the understanding of the review results by proposing a privacy risk taxonomy using the Concerns for Information Privacy framework (CFIP) as a theoretical lens. Based on CFIP, the proposed taxonomy categorises threats and vulnerabilities into the following: collection, error, unauthorised use, and improper access types.

This paper provides novel knowledge by mapping the privacy risks associated with sharing personal information with elements involved and interacting during the sharing activity by adopting the Adaptive EA framework as a theoretical lens. The mapping links the privacy risks dimensions under CFIP with the layers of Adaptive EA, including human, technology, facility, and environmental.

This paper provides a set of actionable knowledge by providing a clear understanding and mapping of the identified privacy threats to the requirements and available existing controls.

This paper provides future research directions regarding the privacy risks of sharing personal information in smart cities.

In a nutshell, this research provides a knowledge foundation, which can be casted into developing theoretical and practical frameworks and solutions for studying and enhancing personal information privacy in the contemporary context of smart cities.

This paper is organised as follows: the “Background and Related Work” section provides the research background and related works. The “Research Method” section explains the research method. Then, data extraction and synthesis are discussed in the “Data Extraction and Synthesis” section, followed by the SLR results in the “ Results ” section. The discussion of implications, study validity and limitations, and work directions is elaborated in the "Discussion" section. The last section encompasses the conclusion.

Background and Related Work

The meaning of privacy varies from one researcher to another. However, core components are common to most definitions of privacy. The most historical definition of privacy was “the right to be let alone” (Warren & Brandeis, 1890 ). Information privacy is defined as the relationship between an individual’s right to privacy and the ability to access and control the information held by organisations (Cranor, 2012 ; Hoffman, 1977 ; Hough, 2009 ; Martinez-Balleste et al., 2013 ). At present, many definitions of privacy have been proposed, and through the years, these definitions have evolved based on societal changes and technological development (Hiller & Russell, 2017 ; Li & Palanisamy, 2018 ; Peppet, 2014 ).

The smart city context has recently risen, and technology has gradually developed. A smart city is identified as an urban area that uses information and communication technology (ICT) to improve its services and enhance its residents’ quality of life (Giffinger et al., 2007 ; Kusumastuti et al., 2022 ). As a result, the individual shares their personal information with service providers, who share it with other organisations either explicitly—implying that the user is involved—or implicitly without the user’s knowledge (Spiekermann & Cranor, 2008 ). Personal information can be used to identify an individual, either directly or indirectly, such as name, email, or biometric information email (Wolford, 2020 ).

Accordingly, information privacy and security concerns have been significantly increased because cities are digitally connected, and individuals’ personal information has become more accessible and available (Hiller & Russell, 2017 ; Solove, 2011 ). This sometimes obstructs society’s adoption of smart cities (Pal et al., 2021 ). For that, personal information privacy risks that arise when sharing personal information in smart cities should be considered carefully to seize new threats and find reasonable solutions. This section briefs privacy risks, regulations, and privacy-enhancing technologies.

Privacy Risks

Privacy risk is defined as the expected losses related to personal information disclosure (Xu et al., 2011 ). Pervasive literature attempts to identify the privacy risks of personal information. For example, Nissenbaum ( 2004 ) proposed a privacy taxonomy based on the contextual integrity (CI) theory, which considers human factors, including their norms and attitudes, as part of privacy risk arising in public surveillance. Henriksen-Bulmer et al. ( 2019 )proposed a taxonomy using the same theoretical lens, IC, to address privacy risks in open data publishing. The privacy taxonomy developed by Solove ( 2006 ) aimed to improve the understanding of information privacy in the legal system. This taxonomy classified privacy risk into four elements: collection, processing, dissemination, and invasion (Solove, 2006 ). Avancha et al. ( 2012 ) developed a privacy taxonomy that classified privacy threats into identity threats, access threats, and disclosure threats in the health system. The framework designed by Deng et al. ( 2011 ) provides a comprehensive analysis of privacy threats to help analysts cover key issues in designing software. In the smart airport, unauthorised access, information leakage, and second use were discussed as privacy threats that affect passenger information (Choudhury & Rabbani, 2019 ; Khi, 2020 ; Tedeschi & Sciancalepore, 2019 ; Zhang, 2019 ). The review conducted by Ismagilova et al. ( 2020 ) focused on security, privacy, and risk in smart cities and how they impact the operational process of smart cities. In addition, a systematic literature review is conducted to identify privacy risks and current solutions relevant to passengers’ information (Alabsi & Gill, 2021 ). In this work, the privacy risks were classified based on the CFIP theory into four types: collection, error, unauthorised use, and improper access.

This review of the literature shows that despite attempts to analyse privacy risks, they only focused on addressing threats without considering vulnerabilities as an essential factor in privacy risk analysis. Furthermore, there is a lack of addressing privacy risks relevant to personal information in other smart city themes, such as smart airport.

Privacy Regulations

The General Data Protection Regulation (GDPR) is a significant regulation that regulates information privacy. The EU adopted the GDPR in 2018 and incorporated principles for personal information processing (Wolford, 2020 ). The GDPR explains principles that help in protecting individual privacy (EUGDPR, 2018 ). Consent, breach announcement, and privacy by design are examples of GDPR principles (EUGDPR, 2018 ).

In the USA, the Fair Information Practices (FIPs) regulation was developed in 1973 to discuss the importance of protecting individual privacy, and it was adopted by the U.S. Privacy Act (Gellman, 2017 ; Li & Palanisamy, 2018 ). Following that time, different sectors in the USA, such as the health and business sectors, developed their privacy regulations called the Health Insurance Portability and Accountability Act (HIPAA) (Silva et al., 2021 ).

In Australia, the Privacy Act 1988 (Act) developed the Australian Privacy Principles (APPs) to protect and guide the use of personal information (Office of the Australian Information Commissioner n.d. ). The APPs consist of principles governing the collection, handling, accessing of personal information, and ensuring the accuracy and integrity of personal information (Office of the Australian Information Commissioner n.d. ).

Based on the above review, it is clear that countries share a common objective in protecting the privacy of personal information and governing how to use it despite their differing regulations.

Privacy-Enhancing Technologies

The interest in privacy protection has been increasing since the 1990s. Thus, there has been a continuous flux of efforts to develop and use Privacy-Enhancing Technologies (PETs) (Hiller & Blanke, 2016 ). PETs are well-designed (ICT) systems for securing and protecting the privacy of information through the reduction, deletion, or avoidance of improper and unnecessary processing of personal data without decreasing the value of the individual information (Chun, 2015 ). The goal of using PET in smart cities is to enable the personal and sensitive information embedded in the collected data to be hidden and not be discovered by any third party or service provider (Curzon et al., 2019 ). Recently, many PETs have been proposed to protect the privacy of information. For example, Van Blarkom et al. ( 2003 ) described PETs techniques such as encryption, anonymisation, pseud-identity, biometric, identification, authorisation, and authentication. Heurix et al. ( 2015 ) provided PETs taxonomy that covered privacy aspects such as user privacy and data privacy across domains not covered in security classifications. Curzon et al. ( 2019 ) provided a detailed review of privacy-enhancing technologies, commonly classified as anonymisation (such as masking and disruption of sensitive data) and security techniques (such as hashing and cryptographic techniques), as the broad types of techniques used mostly for personal information privacy protection. The PETs classification proposed by Kang et al. ( 2007 ) includes three types based on the privacy information life-cycle, including operation technology, common-based technology, and administrative technologies.

It is clear from previous and related research that the study of privacy-enhancing technology has been actively addressed, reflecting its importance in protecting the privacy of personal information.

In summary, protecting the privacy of personal information in smart cities is critical for its effective adoption by citizens or users. Studies have attempted to cover this topic by investigating many solutions and approaches. However, lack of systematic reviews effectively address and assess privacy risks, including threats, vulnerabilities, impacts, and exciting controls relevant to sharing personal information in smart cities, considering who and what is involved and interacted during the sharing activity. This study aims to address this critical need by employing the well-known SLR approach detailed in the following section.

Research Method

This section presents the SLR method applied to conduct this systematic literature review (Kitchenham & Charters, 2007 ). This section includes the following SLR stages: (A) study inclusion and exclusion criteria, (B) data sources and search strategies, (C) study selection process, and (D) quality assessment.

Study Inclusion and Exclusion Criteria

In this study, a set of inclusion and exclusion criteria based on the research questions was used to select the relevant studies from well-known academic and industrial sources. It is important to note here that industry sources have been used to complement the academic sources. Academic studies must be peer-reviewed, including journal articles, conference papers, and book chapters. The studies must satisfy the following criteria: written in the English language, published between 2017 and 2021, include the specified search terms (see Table  1 ), and provide information to address the research questions listed in “ Introduction ” section. Studies that did not meet the inclusion criteria were excluded. This ensures that recent literature relevant to the scope of this study has been adequately covered.

Data Source and Search Strategy

The following well-known electronic databases were used to answer the identified research questions: IEEE Xplore ( www.ieexplore.ieee.org/Xplore/ ), ScienceDirect ( www.sciencedirect.com ), ProQuest( www.proquest.com ), Willy (onlinelibrary.wiley.com/), Gartner ( www.gartner.com/ ).

The selected databases collectively cover a wide range of disciplines relevant to the topic at hand. Furthermore, this SLR includes academic and industrial studies, which distinguishes it from traditional SLR. However, the industrial sources were analysed separately to avoid mixing the non-peer-reviewed studies with academic sources. In the initial research stage, we used the selected search categories and terms presented in Table  1 to find the relevant studies that address the identified research questions. Each search term in the “privacy-preserving” category was combined with each term under the “information sharing” and “smart cities” categories with the operator “AND”. Furthermore, the operator “OR” is used to combine similar terms in each category to ensure maximum coverage.

Study Selection Process

The study selection process assesses the inclusion and exclusion criteria through the following stages. In stage 1, all identified search terms and keywords (see Table  1 ) were searched in the selected databases (as explained earlier), and studies not relevant to inclusion and exclusion criteria were excluded. This stage resulted in 1089 industrial and academic studies. In stage 2, a set of 372 industrial and academic studies were selected after the titles and keywords assessment. In stage 3, further assessments were conducted for the abstract and conclusion, and 127 from both academic and industrial sources were included. A full-text assessment was applied in the final stage to obtain the final set of 83 studies. Further, the quality assessment has been performed on the final selected studies based on pre-identified assessment criteria (Table 3 ) (Kitchenham & Charters, 2007 ). The relevant studies from each stage were stored and managed using EndNote and then exported to Excel sheets to recode inclusion/exclusion decisions. A flowchart of the study selection process, including stages and the number of included studies in each stage, is shown in Fig.  1 . Table 2 also presents the number of selected studies from each selected database in each stage.

Selection process stages and number of included studies

Quality Assessment

The quality assessment was performed based on the checklist made by Kitchenham and Charters ( 2007 ) to ensure the quality of this SLR. The quality assessment criteria items are presented in Table  3 .

The questions of quality criteria were applied to identify the study’s context, aim, and credibility. The selected studies were scored between 1 and 5 based on criteria items. The total score of the study reflects its quality. Each criterion got a score of “1” or “0”. The selected studies from academic sources scored 1 in the research column. Four selected studies scored “0” in the aim column due to a lack of clarity about the study’s aim, while a set of 3 selected studies scored “0” in the column of context because they did not include clear research context details. The majority of studies scored “1” in the finding column. A set of 12 selected studies scored “0” in the future column because of the lack of clarity about the future research directions. To sum up, as indicated in the last column of Table  4 , the quality of selected studies is considered acceptable if the score is 3 or more out of 5 (60% or above).

Data Extraction and Synthesis

We systematically analysed and synthesised the selected studies using the Adaptive Enterprise Architecture (AEA) and Concerns for Information Privacy framework (CFIP) as a theoretical lens, besides the NIST 800–30 framework as a practical lens. We used the CFIP because it helps extract the privacy risk elements (threats and vulnerability) of sharing personal information, which was configured into a proposed privacy risk taxonomy (Fig.  2 ). Our proposed taxonomy consists of four categories based on CFIP: collection, error, unauthorised use, and improper access. CFIP seems to be an appropriate lens (Smith et al., 1996 ) to assess and analyse individual concerns regarding the privacy of organisational information practices. It is a multidimensional framework used as one of the most reliable tools for addressing individual information privacy concerns in many areas, such as e-commerce (Van Slyke et al., 2006 ). The extracted privacy risks under CFIP dimensions are mapped with the AEA framework’s human, technology, facility, and environmental layers (Fig.  3 ). We also used Adaptive EA because it provides systematic layers to extract and map elements involved and interact while sharing personal information, besides relevant regulation as a governmental element that influences this activity. It is important to note here that sharing activity is considered the main element under the interaction layer. Adaptive EA (Gill, 2015 ) is a framework that guides the interaction in the digital ecosystems among five main layers: human, technology, facility, environment, and security. Further, we used NIST SP 800–30, the well-known standard, as a practical lens to identify and extract essential elements to assess privacy risks (Stoneburner et al. 2002 ). NIST was used to complement the theoretical lenses used in this study.

Proposed privacy risk taxonomy based on CFIP framework

Mapping CFIP with Adaptive EA

This was done to ensure that important points from practice were not overlooked. Thus, this study provides rich information incorporating both theoretical and practical perspectives. These elements include privacy threats, vulnerabilities, requirements, and privacy controls (see Fig. 4 ). The identified privacy controls include technical and non-technical controls (Fig. 4 ). The NIST 800–30 is used to carry out risk assessments according to the NIST guidelines (Peacock, 2021 ). The dimensions of CIFP cover different types of privacy risk components (threats and vulnerabilities) related to sharing personal information. Further, NIST 800–30 also offers a structured process that is used to assess privacy risks. Thus, we use CFIP and NIST 800–30 to report the results of this study, which are presented in the following section.

Assessing information privacy risk based on NIST 800–30

To answer the indicated research questions, we analysed the final selected papers in Table 14 in the Appendix. We reviewed and analysed the selected studies using CFIP and NIST 800–30 frameworks to address the research questions to identify privacy risks (privacy threats, vulnerability), privacy risk impacts, and existing privacy controls. It is worth mentioning that the majority of the papers (86%) were taken from academic sources, whereas only 14% of selected studies were found relevant from the well-known industry Gartner data.

It is widely accepted that information risk is composed of threats and relevant vulnerabilities that may impact information assets (Norta et al., 2019 ). In this context, privacy controls are placed to mitigate the risk.

To answer RQ1, we use the CFIP and Adaptive EA as theoretical lenses. Firstly, we identify and categorise the privacy risk components, including privacy threats and vulnerabilities, related to the privacy risk of sharing personal information in smart cities by adopting the CFIP framework dimensions: collection, error, unauthorised use, and improper access (Smith et al., 1996 ). Then, we mapped the identified risks with the layers of Adaptive EA to present the elements involved and interacted in sharing personal information associated with the identified risks and relevant regulation as a governmental element that influences this sharing activity. Adaptive EA consists of the following layers: human, technology, facility, and environmental (Gill, 2015 ).

Privacy Threats

NIST defines threats as undesired and potential harm to the organisational assets such as information, operation and service, or individuals (National Institute of Standards and Technology 2013 ). We reviewed the selected studies to identify privacy threats that affect the sharing of personal information in smart cities in general and several smart city sectors such as smart healthcare, smart grid, smart governments, smart business/organisation, and smart transportation. Based on the CFIP framework, we identified seven types of privacy threats: collection, unauthorised use, improper access, and error from 41% of selected studies. Table 5 presents the identified threats, categories, and selected studies.

As shown in Table  5 , the majority of selected studies (31%) discussed privacy threats under the unauthorised use category. This category includes the following threats: secondary use (T2), information modification (T3), information leakage (T4), and identity theft (T5). Seventeen percent of the reviewed studies highlighted unauthorised access (T1) as a privacy threat under the improper access category. The remaining studies discussed policy and regulation non-compliance privacy threat (T7) under the collection category (6%), with a few studies (2%) focused on information misuse (T6) privacy threats under the error category (3).

As shown in Table  5 , the privacy threats related to patient information sharing in smart health have been widely discussed in the reviewed studies (N3, S4, S5, S6, S7, S8, S12, S17, S3, S27). For example, unauthorised access (T1), information misuse (T6), and modification (T3) threats have been identified as the most common threats that affect the privacy of patient information (Iwaya et al., 2019 ). Patient biometric data are collected and shared with many parties in the smart health sector, which leads to secondary use (T2) and ID theft (T5) threats (Romanou, 2018 ). Regulators and ethics committees are relevant to the health sector classified information leakage (T4) as a privacy threat that affects the collection, use, and sharing of personal information in smart health (Thapa & Camtepe, 2020 ).

As for smart grid, reviewed studies (S9, S16, S18, S19) highlighted that threats included information modification (T3), information leaking (T4), and unauthorised access (T1) are the most common threats that impact consumers’ privacy information shared with different parties. On the other hand, unauthorised access (T1), secondary use (T2), and information leakage (T4) are discussed in the reviewed studies (S11, S20, S21, S13, S10, S22, N2, N5) as privacy threats that affect personal information sharing in smart cities.

As shown in Table  5 , 6 % of reviewed studies identified non-compliance with privacy policies and regulations (T7) as a privacy threat. Several countries and organisations have taken considerable steps toward data privacy policies and regulations in order to protect personal information. According to Wall et al. ( 2015 ), privacy compliance refers to an organisation’s adherence to regulatory privacy requirements to protect personal information. Studies have discussed the increasing information privacy issues in organisations due to non-compliance with privacy policies and regulations in different sectors, including smart cities. For example, healthcare industries handle patients’ information in the USA without explicit patient consent, which is at odds with granular consent under the Health Insurance Portability and Accountability Act (HIPAA) (Runyon, 2020 ).

Vulnerability

According to NIST (National Institute of Standards and Technology 2013 ), vulnerability is the weakness of an asset (e.g. information and system) plausibly exploited by threats. This section reviewed the selected studies based on this definition to extract the perceived vulnerabilities that identified threats might exploit.

As shown in Table  6 , we identified three types of vulnerabilities relevant to the identified threats. Based on our review, 5% of selected studies mentioned that lack and un-transparent policies lead to several privacy threats (Chua et al., 2017 ; Hou et al., 2018 ; Taplin, 2021 ). Examples of these policies include consent, ethics, and privacy policies. Furthermore, the lack of privacy regulation related to handling and sharing personal information, including biometric data, could make this information vulnerable to several privacy threats (S30) (Khi, 2020 ). Insecure/unprotected storage systems and insecure/unprotected sharing mechanisms were identified as vulnerabilities in 3% of selected studies. Insecure storage refers to storing sensitive data without appropriately controlling access. Sharing information in unsecured or unprotected environments leads to privacy risks in smart cities (Agrawal et al., 2021 ; Romanou, 2018 ).

Mapping CFIP Dimensions with Adaptive EA Layers

Our review focused on the threats that affect personal information shared in smart cities in general and different smart city sectors such as smart health, smart grid, smart government, and smart business/organisation. Furthermore, we considered who and what are involved and interacted in the sharing activity, besides relevant regulation as a governmental element that influences this activity (based on Adaptive EA). Tables 7 , 8 , 9 , 10 , and 11 present the elements relevant to Adaptive EA layers: human, technology, facility, and environment, in smart cities. Figures  5 , 6 , 7 , and 8 represent the map of CFIP dimensions with Adaptive EA layers.

As illustrated in Fig.  5 , in the smart health context, elements under human layers are identified from 11% of selected studies that discussed the unauthorised use privacy risk associated with sharing patients’ information in smart health. In contrast, with improper access and error risks, the studies’ percentages dropped to 7% and 1%. On the other hand, elements under technology layers are discussed in 6% of selected studies that investigated improper access and unauthorised use privacy risks, with 0% of studies in error and collection risks. However, the environmental layer is considered in selected studies (4%) when addressing privacy risks categorised under unauthorised use more than in improper access (1%) and collection dimensions (2%). We identified patients, service providers, and doctors as the main actors under human layers from 13% of selected studies. At the same time, infrastructure such as IoT and data storage, such as centralised databases, are identified under technology layers in 11% of selected studies. Facility layers are discussed in 6% of selected studies. The facility layer presents different smart health buildings, such as hospitals, medical centres, laboratories, and clinics. Privacy regulations are mainly discussed under the environmental layer in 6% of selected studies, which can be used to define or inform a separate layer of privacy. This seems to suggest the extension of the Adaptive EA framework through the introduction of the privacy layer. Table 7 presents elements under each layer of Adaptive AE in smart health context.

Mapping CFIP dimensions with AEA layers in smart health

In the smart grid, Fig.  6 shows that more selected studies mentioned human, technology, and facility layers when addressing improper access and unauthorised use privacy risks associated with sharing users’ information, while no studies discussed theses layers with error and collection privacy risks.

Mapping CFIP dimensions with AEA layers in the smart grid

In Table 8 , 4% of selected studies identified different actors under the human layer in the smart grid context, including users and customer service providers. Based on our review, 6% of selected studies discuss the usage of the cloud as the main data storage in the smart grid, while IoT applications and smart metres are the main infrastructures discussed in the smart grid system. Elements under facilities layers are found in 6% of selected studies that discuss privacy risks associated with sharing personal information in the smart grid. Examples of facility layer elements are control centres, power sources, and home gateways.

As presented in Fig.  7 , almost a few percent of studies only mentioned human and technology layers with improper access risk compared with studies that addressed unauthorised use privacy risks associated with sharing users’ information in the smart city context.

Mapping CFIP dimensions with AEA layers in smart city

Based on Table  9 , from 5% of selected studies, we identified two main actors under human layers who are involved in sharing personal information in smart cities. The main actors include individuals, such as citizens and users, and organisations, including service providers and data holders. Moreover, IoT devices, Cloud systems, and smart city applications are identified in 6% of selected studies as elements under technology layers used in sharing personal information in smart cities.

As illustrated in Fig.  8 , most selected studies in the smart business/organisation context explain elements in human, technology, and facilities layers when addressing unauthorised privacy risks associated with sharing personal information, whereas this percentage decreased with improper access privacy risk. On the other hand, the environmental layer is mentioned in 2% of studies that addressed privacy risks under improper access and unauthorised risks, with 1% with collection privacy risks.

Mapping CFIP dimensions with AEA layers in smart business/organisation

Based on Table  10 , we identified several actors, such as employees, customers, and experts, under the human layer from 4% of selected studies. The facility layer includes buildings, such as organisations, public workplaces, and industry, discussed in 7%. On the other hand, technical layer elements, such as infrastructure and data storage, and environmental elements, such as privacy regulation, are discussed in 5% of selected studies.

As shown in Table  11 , human, technology, and facility layers have been mentioned in 2% of selected studies that discussed improper access and unauthorised use privacy risks in smart government, with 1% of studies addressing unauthorised use in the smart transportation context.

Privacy Risks Impacts

To answer RQ2, we reviewed the selected studies to identify and extract privacy requirements impacted by the identified privacy risks. The proper privacy requirements should be considered when personal information is shared in smart cities. Thus, we reviewed the selected studies to extract the privacy requirements that the identified threats might impact (Table  12 maps the requirements with relevant threats). As shown in Table  12 , we identified eight classified requirements. The classifications include the CIA triad (confidentiality, integrity, availability) and IAAA (identification, authentication, authorization, accounting). In addition, we extracted the privacy requirements based on the classification proposed by Pfitzmann and Hansen ( 2010 ), which is very common in the privacy domain. The classification consists of anonymity and pseudonymity, unlinkability, undetectability, and unobservability. Table 12 includes a list of privacy requirements that need to be satisfied when sharing personal information in smart cities.

Concerning the CIA classification, 20% of selected studies discussed confidentiality and integrity as essential requirements to achieve privacy (Table  12 ). In contrast, availability is discussed in 10% of selected studies to achieve security besides privacy. In smart health, Health Information Exchange (HIE) has been adopted to enable the electronic sharing of patient information between several parties (Mutanu et al., 2022 ). Thus, confidentiality, integrity, and availability are essential to preserve patient information privacy and security (Yi et al., 2013 ). In addition, the CIA triad should be satisfied with a smart grid and smart transportation to protect privacy as the information is shared between relevant parties to provide various services to the users (Yang et al. 2014 ).

As for the IAAA classification, 13% of selected studies discussed authentication as a requirement for privacy (Table  12 ). However, authorization was discussed in 5% of selected studies, whereas identification was discussed in 2% of selected studies. In the smart grid, identification and authentication requirements need to be satisfied to secure access to the information or system component (Ferrag et al., 2018 ; Sadhukhan et al., 2021 ). In smart health, authentication, authorization, and identification requirements should be satisfied when sharing patient information to ensure that privacy is not compromised (Shamshad et al., 2020 ; Wang et al., 2019 ).

We reviewed the selected studies to extract the requirements classified based on the terminology proposed by Pfitzmann and Hansen ( 2010 ). As shown in Table 12 , 12% of selected studies discussed anonymity as an essential requirement to ensure the privacy of information, whereas only 1% mentioned unlinkability requirements. These requirements are addressed in both smart health and smart transportation to achieve the privacy of personal information (Yang et al., 2018 , Chenthara et al., 2019 ).

Existing Privacy Control

To answer the RQ3, we reviewed the privacy-preserving schemes for sharing personal information in smart cities. We also extracted the existing privacy controls proposed to mitigate the identified risks from the selected studies (Table  13 maps the privacy controls with identified threats). Further, we classified the identified control under technical and non-technical, as shown in Table  13 . Figure  9 represents the percentage of the identified privacy controls from the selected studies. Technical control methods include security-based solutions, such as encryption, access control, etc., whereas non-technical methods refer to policies, procedures and standards (National Institute of Standards and Technology, 2013 ).

Existing privacy control

Considering the technical solution, we identified ten technical controls categorised into four groups: anonymisation, cryptographic techniques, access control techniques, blockchain, and machine learning (Table  13 ). In this study, the classification of technical solutions is based on the classification of PETs proposed by Van Blarkom et al. ( 2003 ) and Curzon et al. ( 2019 ). In addition, we reviewed technical controls developed on blockchain and machine learning.

Data Anonymization

As sown in Table  13 , 7% of reviewed studies discussed anonymization techniques as technical privacy controls. This includes K-anonymity, differential privacy, and pseudonym. Data anonymization is the method used to protect personal information by preventing linking their identities (Curzon et al., 2019 ; Iyengar, 2002 ; Silva et al., 2021 ). K-anonymity and differential privacy are the most common methods of anonymization technique (Iyengar, 2002 ). As for smart health, the reviewed study (S12) discussed the popularity of using anonymity to preserve the privacy of transmitted personal information between parties. On the other hand, the pseudonym is discussed in (S49) as an anonymous technique that is proposed to preserve the privacy of sharing information in smart transportation.

Cryptographic Technique

Table 13 includes cryptographic techniques used in privacy-preserving schemes for sharing personal information in smart cities. The techniques were extracted from 8% of selected studies. Cryptographic technology entails ways of totally hiding data equivalent to the intensity of the cryptographic key and algorithm employed. Encrypting transmitted or stored personal information in smart cities is a broadly used technology that protects from leakage and achieves privacy requirements (Curzon et al., 2019 ; Gaire et al., 2019 ). For example, attribute-based encryption (ABE) is proposed to preserve patient information sharing in smart health (S7, S57). Cryptographic technique for processing biometric data is presented in (S12); in this method, the digital key is securely linked by a biometric sample that is used to encrypt and decrypt the key. Elliptic curve cryptography to secure and authenticate the communication between the consumer and the service provider in the smart grid is discussed in (S36, S28).

Access Control Mechanism

Access control is defined as security methods to control the access and use of information by applying access policies (Sandhu & Samarati, 1994 ). In Table  13 , 6% of reviewed studies discussed privacy-preserving schemes developed based on the access control mechanism. For example, schemes presented in selected studies proposed several access control mechanisms, such as fine-grained access control and multi-layer access control (MLAC), to preserve the privacy of patient information shared between different parties in a cloud-based environment.

Machine Learning

Table 13 shows that privacy-preserving schemes for sharing information in smart cities using machine learning techniques are discussed in 2% of selected studies. A self-organising map (SOM) is a machine learning technique used to share information about electricity usage between parties in the smart grid (S65). The machine learning technique, federated learning, is used to share and analyse medical cases in smart health without compromising patient privacy (S58).

As shown in Table  13 , 42% of selected studies proposed privacy-preserving schemes for sharing information using Blockchain technology. Blockchain is a decentralized cryptographic scheme employed to privatise and safeguard transactions in the confines of a network (Curzon et al., 2019 ). It has been noticed that the privacy-preserving schemes in selected studies integrated blockchain with other PETs to share personal information without compromising their privacy. For example, access control mechanisms and blockchain are proposed in studies (S4, S6, S20, S41, S48, S50, S6, S8, S26, S27, S33, S34) mainly for two purposes. The first one is to allow individuals to monitor and regulate their information sharing between parties in smart cities. The second purpose is to authenticate the identity while sharing and accessing the information in smart cities. The selected studies (S9, S39, S14, S63, S21, S45, S31) proposed privacy-preserving schemes that use several cryptographic techniques, including signature, identity-based proxy, proxy re-encryption, zero-knowledge, and attribute-based encryption, with blockchain to protect the privacy of individual information in smart grid and smart health.

Non-technical Control

Among the selected studies, a total of 35% discussed non-technical privacy control to mitigate the identified threats (Table  13 ). For example, the importance of privacy by design (PbD) as a principle of GDPR is discussed in an attempt to protect the privacy of personal information in smart health and biometric applications (S12). Several policy-based schemes are discussed to capture the imposed requirements and restrictions that enhance the privacy of shared information in smart cities (S5, S66). On the other hand, privacy management is discussed in the selected studies as a type of non-technical privacy controls (S42, S13, S68, S67). As shown in Table  13 , the non-technical privacy controls are discussed widely in the industrial reports (N1, N6, N7, N8, N9, N10, N11, N12, N4). Organisations need to reduce information disclosure as it leads to privacy and financial risks (Brian Lowans & Meunier, 2019 ). Effective privacy management programs should address privacy risk prevention and incorporate privacy-by-design principles into all business activities (Bart Willemsen, 2017 ). In this context, many risk management approaches, such as integrated risk management (IRM), data security governance (DSG) framework, privacy impact assessment(PIA), and continuous adaptive risk and trust assessment (CARTA), are discussed to help businesses dealing with risks and their consequences and also to ensure the sustainability of the protection of any project (N6, N7, N1, N11). Furthermore, the importance of designing a privacy-aware risk programme to define and assess the risk of using blockchain technology for sharing personal information is discussed in industry publications (N8, N9).

This research provided a consolidated view of the selected studies from academic and industrial sources and reported on the privacy risks, impacts, and controls related to personal information sharing in smart cities. This was done to thoroughly identify the privacy risks that affect the sharing of personal information in smart cities. Since sharing personal information in smart cities results from the interaction among different elements, this study also aims to identify these elements, including actors, technologies, facilities, and privacy laws, that are involved in sharing activity. Identifying privacy risks, including threats and vulnerabilities, the risk impacts, and existing controls, taking into account the elements involved in sharing activity, will assist organisations in determining the appropriate controls to mitigate the risks when sharing personal information in smart cities. This section describes the implications based on our review and analysis of selected studies. It also includes the limitations of this work.

Implications

Privacy risk.

Many studies have proposed threat taxonomies that organise threats into different categories (Deng et al., 2011 ; Xiong & Lagerström, 2019 ). However, to the best of our knowledge, there is a lack of systematic and theoretical understanding, which is filled by this study using the CFIP as a theoretical lens. This study proposed a taxonomy of privacy risks of sharing personal information in smart cities, including threats and vulnerabilities, based on the CFIP theoretical lens. Based on Table  5 , our findings show that the selected studies do not properly investigate policies and consent non-compliance, misuse, and ID theft as serious threats that widely affect the privacy of sharing personal information in smart cities. Furthermore, we found that selected studies did not clearly distinguish between threats’ events and their sources, making it hard to identify the relevant privacy threats to the scope of this study. Thus, there is still a great deal of work to be done in this area in both academic and industrial research.

On the other hand, based on Table  5 , we found that most selected studies discussed privacy threats associated with sharing personal information in smart cities in general and in the smart health system. In contrast, studies that discussed the same topic under the smart grid, smart government, smart business, and smart transportation systems were limited. One immediate impact of this finding on the digital economy is the reinforcement of the importance of investing in robust technological solutions and infrastructures, as well as developing risk management frameworks to mitigate the privacy and security risks associated with personal information in smart cities (Ahmed, 2021 , Jnr et al., 2023 , Jin, 2024 ).

The digital economy is the deep integration of digital technology and production factors in smart cities to manage the transformation cost, improve cities’ capabilities and implement innovative solutions (Sotirelis et al., 2022 ; Vinod Kumar & Dahiya, 2017 ; Wang et al., 2021 ; Zhiyong et al., 2024 ).

The emphasis on privacy risks of sharing personal information in smart cities highlights the need for innovative solutions that simultaneously advance their capabilities while rigorously safeguarding individual privacy. This could increase investment in implementing privacy controls to protect individual information handled within smart city sectors (Jin, 2024 ).

As smart city sectors heavily rely on sharing individual information by integrating smart technologies, there is a pressing need to address privacy risks associated with personnel. This could spur investment in privacy-enhancing technologies, regulatory frameworks, and public awareness campaigns tailored to these specific domains. This draws our attention to the need for more studies in order to cover this gap.

On the other hand, selected studies from industry sources discussed the identified privacy threats relevant to personal information without mentioning their relationship with smart cities or any other smart system.

On the other hand, it is well-accepted that any risk analysis should be done based on identified threats and relevant vulnerabilities (Stoneburner et al. 2002 , Norta et al., 2019 ). The identification of vulnerabilities is an essential factor that plays a role in identifying privacy risks. Based on Table  6 , we found that selected studies do not investigate vulnerabilities as a significant factor in addressing privacy risks relevant to sharing personal information in smart cities. As a result, the knowledge about the identified privacy risks was limited. Thus, there is a need to understand the threats and vulnerabilities to identify and mitigate privacy risks.

Based on our review, very limited studies currently explain who and what elements are involved when addressing privacy risks associated with sharing personal information in smart cities. Furthermore, to the best of our knowledge, no previous studies have demonstrated the interaction among the elements involved when addressing the topic mentioned above. To overcome the shortcomings of previous studies outlined above, we adopted Adaptive EA as a theoretical lens to map the identified privacy risks relevant to sharing personal information in smart cities, with elements involved and interacting in sharing activity. This study mapped the identified privacy risks based on CFIP dimensions, including improper access, unauthorised use, error, and collection, with Adaptive EA layers that include human, technology, facility, and environmental. Based on Figs.  5 , 6 , 7 , and 8 , we found that out of all the studies that addressed privacy risks associated with sharing personal information, most studies discussed human and technical layers, followed by the facility layer in all smart city sectors. However, few studies discussed the environmental layer, including privacy regulation and policies, only when addressing improper access and unauthorised use of privacy risks relevant to sharing personal information in smart health and smart business/organisation contexts.

Furthermore, according to Tables 7 , 8 , 9 , 10 , and 11 , we found that most studies that defined elements under human and technology layers are relevant to smart health, with few studies in other smart city sectors. Additionally, although applying policies and regulations is vital to mitigate privacy risks associated with personal information in any smart city, we noticed that these elements, mainly categorised under the environmental layer, have not been investigated enough in the selected studies. Based on the above, there is a need to cover these gaps in future work.

Undoubtedly, defining privacy requirements helps to study the consequences of privacy risks relevant to personal information. Moreover, it helps to choose the proper treatment for the identified risks. In this regard, we reviewed the selected studies to identify the privacy requirements based on well-known classifications such as CIA, IAAA, and the privacy requirement terminology (Pfitzmann & Hansen, 2010 ). Based on Table  12 , our findings reveal that current studies investigate CIA triad and identification, authorization, authentication, and anonymity requirements for privacy risk in smart cities. However, addressing the impact of privacy risk on accounting, undetectability, unobservability, and pseudonymity is still largely unclear. This draws our attention to the need for more studies defining those requirements when discussing the privacy risks of sharing personal information in smart cities. Another finding shows that most selected studies link the requirements with the proposed technical controls. They test proposed solutions against those requirements to explain how they should satisfy them. However, there is a lack of studies that discuss the link between these requirements and privacy risks. For example, to the best of our knowledge, secondary use, ID theft, and policy and consent non-compliance threats are not linked with any one of the identified requirements; thus, more studies need to cover this gap to address the consequences and impacts of these risks.

Existing Control

We reviewed the selected studies to extract the existing privacy controls to preserve the privacy of sharing personal information in smart cities. We categorised privacy controls based on the well-known practical framework NIST 800–30 into technical and non-technical controls. Based on Table  13 , our findings show that technical privacy controls, such as cryptography, anonymity, access control, blockchain, and machine learning, are frequently discussed in the selected studies. However, those controls are insufficient to preserve personal information privacy in smart cities because they are poorly developed due to technical and cost restrictions. Another finding shows that a set of 23 selected studies proposed technical solutions without implicitly explaining what kind of privacy threats could be mitigated by the proposed solution. This means they proposed the solution to preserve privacy issues in smart cities. Thus, linking the technical solution with specific privacy threats needs more investigation in the literature. Table 13 also finds that blockchain is widely used in privacy-preserving schemes proposed in academic literature. This indicates the importance and effectiveness of using it to share personal information in smart cities without compromising privacy when integrating it with different PETs. On the other hand, our findings show that risk management has fewer research activities in academic fields; thus, this area requires further investigation.

Finally, the current research investigates risks, impact, and existing controls in different areas of focus (e.g. information security/privacy), and  across various domains (e.g. smart health, smart grid, smart airport, and smart organisations). However, based on the analysis results, these studies seem to lack a systematic and common understanding of information privacy risks in smart cities. To address this challenge, there is a need to develop an ontology-based privacy risk assessment framework for a systematic and common understanding of privacy risks associated with sharing personal information in smart cities. Thus, this study is the first step to systematically synthesis and conceptualise the knowledge dispersed across different papers. It will provide a knowledge base and foundation for developing the personal information privacy risk ontology. The ontology will help enhance understanding the complex concepts and their relationships. Furthermore, it will help establish a common understanding for assessing and mitigating privacy risks in an informed manner. The development and evaluation of such ontology are beyond this paper’s scope and subject to further research. However, this paper provided a strong foundation for this much-needed ontology work.

Validity and Limitations

This work has some limitations like any other SLR. Given this study’s scope, we used well-known academic and industry databases to ensure sufficient coverage of the research topic. This provided a combination of academic and industrial studies explicitly emphasised in the analysis.

Given our emphasis on rigorously identifying and selecting relevant publications through systematic search strategies, the research methodology used in this study was suitable because it provided a multistage process. The process includes applying predefined inclusion and exclusion criteria and synthesising findings to derive meaningful insights to ensure that the process is unbiased.

One potential methodological limitation of the employed methodology in this study is the reliance on predefined databases, which may limit the comprehensiveness of the literature search. However, the identified databases encompass academic and industry sources, totalling six. This ensures that the selected databases cover a wide range of studies relevant to the topic at hand.

To ensure the validity and rigour of the adopted research methodology, we tested the search terms and keywords based on the identified research questions across the pre-selected databases. Furthermore, the process was reviewed to confirm the research’s quality and coverage prior to the documentation stage. In addition, the quality assessment criteria were used to avoid researcher bias and ensure the selected studies’ relevance and quality. Human error might lead to inconsistencies when conducting such research. Thus, regular meetings between the senior researcher and this study’s author were held to minimise the possibility of human error and ensure the quality of the research process and results. This also includes reviewing and learning from the SLRs published in different domains in quality academic outlets. Integrating the employed approach with an additional one to enhance the rigour and comprehensiveness of reviews is suggested as a future research direction.

The term “smart city” has become the focus of several countries striving to improve their population quality, enhance their economies, and ensure sustainability. To achieve their objectives, cities have adopted innovative technologies and applications and developed their ICT infrastructure to support smart city initiatives in many sectors. These sectors include health, government, transportation, business, and organisation. However, due to the strong relationship between ICT and smart cities, personal information is easily shared among relevant parties, leading to serious privacy risks that may affect individuals and organisations. These risks need to be addressed, as highlighted in this SLR. This study analysed and synthesised published research to identify and extract privacy risks, impacts, and existing controls related to sharing personal information in different sectors in smart cities. It also considers elements involved and interacting in the sharing activity based on the well-known CFIP framework and Adaptive EA as theoretical lenses and NIST 800–30 as a practical lens. Based on NIST 800–30, we identified seven privacy threats, three vulnerabilities, and eight requirements that might be impacted by the identified threats, along with seven privacy controls classified into technical and non-technical types. Furthermore, we used CFIP as a theoretical lens to identify and categorise privacy threats and vulnerabilities relevant to the scope of this study. Based on CFIP, we categorised the identified privacy risks (threats and vulnerabilities) into four main groups: collection, unauthorised access, improper use, and errors.

Furthermore, we mapped the identified risks to identified requirements and current controls. The Adaptive EA is used to map the identified risks under CFIP dimensions with layers that interact while sharing personal information in smart cities. Our findings show the need for contemporary solutions to improve the privacy level of sharing personal information in smart cities. Furthermore, there is a need to represent privacy risk assessment components and their relationship and the relation among elements involved in sharing personal information using ontology to facilitate common understanding and sharing of the relevant concepts between different parties involved in connected smart cities. This SLR can benefit both academia and industry by helping them better understand the privacy of sharing personal information in smart cities and providing a synthesised foundation for further work in this important area of research.

Data Availability

Not applicable.

Agrawal, T. K., Kumar, V., Pal, R., Wang, L., & Chen, Y. (2021). Blockchain-based framework for supply chain traceability: A case example of textile and clothing industry. Computers & Industrial Engineering, 154 , 107130.

Article   Google Scholar  

Ahmad Mohamad, A.-S., Alsmadi, M. K., Abdel Karim, B., Ibrahim, A., Abouelmagd, H., & Osman SaadShidwan, A. (2019). Emergent situations for smart cities: A survey. International Journal of Electrical and Computer Engineering, 9 (6), 4777–4787.

Google Scholar  

Ahmed, E. M. (2021). Modelling Information and communications technology cyber security externalities spillover effects on sustainable economic growth. Journal of the Knowledge Economy, 12 (1), 412–430.

Alabsi, M. I., & Gill, A. Q. (2021). A review of passenger digital information privacy concerns in smart airports. IEEE Access, 9 , 33769-33781.

Albino, V., Berardi, U., & Dangelico, R. M. (2015). Smart cities: Definitions, dimensions, performance, and initiatives. Journal of Urban Technology, 22 (1), 3–21.

Avancha, S., Baxi, A., & Kotz, D. (2012). Privacy in mobile technology for personal healthcare. ACM Computing Surveys (CSUR), 45 (1), 1–54.

Bart Willemsen, P. B. (2017). The four do’s and don’ts of implementing your privacy program . Gartner.

Bogoda, L., Mo, J., Bil, C., & Ieee, (2019). A systems engineering approach to appraise cybersecurity risks of cns/atm and avionics systems. In 2019 Integrated Communications, Navigation and Surveillance Conference .

BrianLowans, B. W., & Meunier, M.-A. (2019). Use the data security governance framework to balance business needs and risks . Gartner.

Caragliu, A., Del Bo, C., & Nijkamp, P. (2009). Smart cities in Europe. Journal of Urban Technology, 18 , 65–82.

Chenthara, S., Khandakar, A., & Whittaker, F. (2019). Privacy-preserving data sharing using multi-layer access control model in electronic health environment. EAI Endorsed Transactions on Scalable Information Systems , 6, 22. https://doi.org/10.4108/eai.13-7-2018.159356

Choudhury, Z. H., & Rabbani, M. M. A. (2019). Biometric passport for national security using multibiometrics and encrypted biometric data encoded in the QR code. Journal of Applied Security Research, 15 , 1–31.

Chua, H. N., Herbland, A., Wong, S. F., & Chang, Y. (2017). Compliance to personal data protection principles: A study of how organisations frame privacy policy notices. Telematics and Informatics, 34 (4), 157–170.

Chun, S.-H. (2015). Privacy enhancing technologies (PETs) and investment strategies for a data market. Procedia-Social and Behavioral Sciences, 185 , 271–275.

Cranor, L. F. (2012). Necessary but not sufficient: Standardized mechanisms for privacy notice and choice. J on Telecomm & High Tech L, 10 , 273.

Curzon, J., Almehmadi, A., & El-Khatib, K. (2019). A survey of privacy enhancing technologies for smart cities. Pervasive and Mobile Computing, 55 , 76–95.

Deng, M., Wuyts, K., Scandariato, R., Preneel, B., & Joosen, W. (2011). A privacy threat analysis framework: Supporting the elicitation and fulfillment of privacy requirements. Requirements Engineering, 16 (1), 3–32.

EUGDPR. (2018). “GDPR key changes.” Retrieved 20 Sep, 2019, from http://www.eugdpr.org/key-changes.html .

Ferrag, M. A., Maglaras, L. A., Janicke, H., Jiang, J., & Shu, L. (2018). A systematic review of data protection and privacy preservation schemes for smart grid communications. Sustainable Cities and Society, 38 , 806–835.

Gaire, R., Ghosh, R. K., Kim, J., Krumpholz, A., Ranjan, R., Shyamasundar, R., & Nepal, S. (2019). Crowdsensing and privacy in smart city applications (pp. 57–73). Elsevier.

Gellman, R. (2017). Fair information practices: A basic history. Available at SSRN 2415020.

Giffinger, R., Fertner, C., Kramar, H., Kalasek, R., Milanović, N., & Meijers, E. (2007). Smart cities - Ranking of European medium-sized cities . Centre of Regional Science, Vienna University of Technology.

Gill, A. Q. (2015). Adaptive cloud enterprise architecture . World Scientific.

Book   Google Scholar  

Gill, A. Q. (2021). A theory of information trilogy: Digital ecosystem information exchange architecture. Information, 12 (7), 283.

Henriksen-Bulmer, J., Faily, S., & Jeary, S. (2019). Privacy risk assessment in context: A meta-model based on contextual integrity. Computers & Security, 82 , 270–283.

Heurix, J., Zimmermann, P., Neubauer, T., & Fenz, S. (2015). A taxonomy for privacy enhancing technologies. Computers & Security, 53 , 1–17.

Hiller, J. S., & Blanke, J. M. (2016). Smart cities, big data, and the resilience of privacy. Hastings LJ, 68 , 309.

Hiller, J. S., & Russell, R. S. (2017). Privacy in crises: The NIST privacy framework. Journal of Contingencies and Crisis Management, 25 (1), 31–38.

Hoffman, L. (1977). Modern methods for computer security and privacy . Englewood Cliffs: Prentice-Hall.

Hong, J. I., Ng, J. D., Lederer, S., & Landay, J. A. (2004). Privacy risk models for designing privacy-sensitive ubiquitous computing systems. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques .

Hou, Y., Gao, P., & Nicholson, B. (2018). Understanding organisational responses to regulative pressures in information security management: The case of a Chinese hospital. Technological Forecasting and Social Change, 126 , 64–75.

Hough, M. G. (2009). Keeping it to ourselves: Technology, privacy, and the loss of reserve. Technology in Society, 31 (4), 406–413.

Hsiao, Y.-C., Wu, M.-H., & Li, S. C. (2021). Elevated performance of the smart city-A case study of the IoT by innovation mode. IEEE Transactions on Engineering Management, 68 (5), 1461–1475.

Imine, Y., Lounis, A., & Bouabdallah, A. (2020). An accountable privacy-preserving scheme for public information sharing systems. Computers & Security, 93 , 101786.

Ismagilova, E., Hughes, L., Rana, N. P., & Dwivedi, Y. K. (2020). Security, privacy and risks within smart cities: Literature review and development of a smart city interaction framework. Information Systems Frontiers.

Iwaya, L. H., Fischer-Hübner, S., Åhlfeldt, R.-M., & Martucci, L. A. (2019). Mobile health systems for community-based primary care: Identifying controls and mitigating privacy threats. JMIR mHealth and uHealth, 7 (3), e11642.

Iyengar, V. S. (2002). Transforming data to satisfy privacy constraints. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining .

Jin, W. (2024). Security and privacy of digital economic risk assessment system based on cloud computing and blockchain. Soft Computing, 28 (3), 2753–2768.

Jnr, B. A., Sylva, W., Watat, J. K., & Misra, S. (2023). A framework for standardization of distributed ledger technologies for interoperable data integration and alignment in sustainable smart cities. Journal of the Knowledge Economy .

Kalloniatis, C., Kavakli, E., & Gritzalis, S. (2008). Addressing privacy requirements in system design: The PriS method. Requirements Engineering, 13 (3), 241–255.

Kang, Y., Lee, H., Chun, K., & Song, J. (2007). Classification of privacy enhancing technologies on life-cycle of information. The International Conference on Emerging Security Information, Systems, and Technologies (SECUREWARE 2007), IEEE.

Khatoun, R., & Zeadally, S. (2017). Cybersecurity and privacy solutions in smart cities. IEEE Communications Magazine, 55 (3), 51–59.

Khi, I. A. (2020). Ready for take-off: How biometrics and blockchain can beat aviation’s quality issues. Biometric Technology Today, 2020 (1), 8–10.

Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering, 2 (3).

Kong, Y., Zhao, J., Yuan,, L., Dong, N., Lin, Y. & Yang, B. (2018). Research on data sharing analysis and key technology of smart city. In 2018 26th International Conference on Geoinformatics .

Kusumastuti, R. D., Nurmala, N., Rouli, J., & Herdiansyah, H. (2022). Analyzing the factors that influence the seeking and sharing of information on the smart city digital platform: Empirical evidence from Indonesia. Technology in Society, 68 , 101876.

Li, C., & Palanisamy, B. (2018). Privacy in Internet of Things: From principles to technologies. IEEE Internet of Things Journal, 6 (1), 488–505.

Martinez-Balleste, A., Perez-Martinez, P. A., & Solanas, A. (2013). The pursuit of citizens’ privacy: A privacy-aware smart city is possible. IEEE Communications Magazine, 6 , 136.

Mutanu, L., Gupta, K., & Gohil, J. (2022). Leveraging IoT solutions for enhanced health information exchange. Technology in Society, 68 , 101882. https://doi.org/10.1016/j.techsoc.2022.101882

National Institute of Standards and Technology, (2013). Guide for conducting risk assessments. https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-30r1.pdf

Nissenbaum, H. (2004). Privacy as contextual integrity. Wash. l. Rev., 79 , 119.

Norta, A., Matulevičius, R., & Leiding, B. (2019). Safeguarding a formalized Blockchain-enabled identity-authentication protocol by applying security risk-oriented patterns. Computers & Security, 86 , 253–269.

Office of the Australian Information Commissioner (n.d.). “Australian privacy principles.” Retrieved 2 Jun, 2020, from https://www.oaic.gov.au/privacy/australian-privacy-principles

Pal, D., Zhang, X., & Siyal, S. (2021). Prohibitive factors to the acceptance of Internet of Things (IoT) technology in society: A smart-home context using a resistive modelling approach. Technology in Society, 66 , 101683.

Peacock, J. (2021). What is NIST SP 800 30. Retrieved 9 September 2021, from https://www.cybersaint.io/blog/what-is-nist-sp-800-30

Peppet, S. R. (2014). Regulating the internet of things: First steps toward managing discrimination, privacy, security and consent. Tex. l. Rev., 93 , 85.

Pfitzmann, A., & Hansen, M. (2010). A terminology for talking about privacy by data minimization: Anonymity, unlinkability, undetectability, unobservability, pseudonymity, and identity management . Dresden.

Qian, Y., Liu, Z., Yang, J. & Wang, Q. (2018). A method of exchanging data in smart city by blockchain. In 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS) .

Romanou, A. (2018). The necessity of the implementation of Privacy by Design in sectors where data protection concerns arise. Computer Law & Security Review, 34 (1), 99–110.

Runyon, B. (2020). Healthcare CIOs: Prepare for granular patient consent . Gartner.

Sadhukhan, D., Ray, S., Obaidat, M. S., & Dasgupta, M. (2021). A secure and privacy preserving lightweight authentication scheme for smart-grid communication using elliptic curve cryptography. Journal of Systems Architecture, 114 , 101938.

Safiullin, A., Krasnyuk, L., & Kapelyuk, Z. (2019). Integration of Industry 4.0 technologies for “smart cities” development. IOP conference series: materials science and engineering, IOP Publishing.

Sandhu, R. S., & Samarati, P. (1994). Access control: Principle and practice. IEEE Communications Magazine, 32 (9), 40–48.

Shamshad, S., Mahmood, K., Kumari, S., & Chen, C.-M. (2020). A secure blockchain-based e-health records storage and sharing scheme. Journal of Information Security and Applications, 55 , 102590.

Sharma, S., Singh, G., Sharma, R., Jones, P., Kraus, S., & Dwivedi, Y. K. (2020). Digital health innovation: exploring adoption of COVID-19 digital contact tracing apps. In IEEE transactions on engineering management , 1–17.

Silva, P., Monteiro, E., & Simões, P. (2021). Privacy in the Cloud: A survey of existing solutions and research challenges. IEEE Access, 9 , 10473–10497.

Smith, H., Milberg, S., & Burke, S. J. (1996). Information privacy: Measuring individuals’ concerns about organisational practices. MIS Quarterly, 20 , 167–196.

Solove, D. J. (2006). A taxonomy of privacy. University of Pennsylvania Law Review, 154 (3), 477–564.

Solove, D. J. (2011). “Nothing to hide: The false tradeoff between privacy and security (Introduction).”

Sotirelis, P., Nakopoulos, P., Valvi, T., Grigoroudis, E., & Carayannis, E. (2022). Measuring smart city performance: A multiple criteria decision analysis approach. Journal of the Knowledge Economy, 13 (4), 2957–2985.

Spiekermann, S., & Cranor, L. F. (2008). Engineering privacy. IEEE Transactions on Software Engineering, 35 (1), 67–82.

Stoneburner, G., Goguen, A., & Feringa, A. (2002). Risk management guide for information technology systems, special publication (NIST SP), National Institude of Standard and Technology.

Taplin, K. (2021). South Africa’s PNR regime: Privacy and data protection. Computer Law & Security Review, 40 , 105524.

Tedeschi, P., & Sciancalepore, S. (2019). Edge and fog computing in critical infrastructures: Analysis, security threats, and research challenges. In 2019 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) .

Thapa, C., & Camtepe, S. (2020). Precision health data: Requirements, challenges and existing techniques for data security and privacy. Computers in Biology and Medicine , 104130.

Van Blarkom, G., Borking, J. J., & Olk, J. E. (2003). Handbook of privacy and privacy-enhancing technologies. Privacy Incorporated Software Agent (PISA) Consortium, The Hague 198.

Van Slyke, C., Shim, J., Johnson, R., & Jiang, J. (2006). Concern for information privacy and online consumer purchasing. Journal of the Association for Information Systems , 7 (6). https://doi.org/10.17705/1jais.00092

Vinod Kumar, T., & Dahiya, B. (2017). “Smart economy in smart cities. In Smart economy in smart cities: International collaborative research: Ottawa, St. Louis, Stuttgart, Bologna, Cape Town, Nairobi, Dakar, Lagos, New Delhi, Varanasi, Vijayawada, Kozhikode, Hong Kong, 3–76.

Vu, K., & Hartley, K. (2018). Promoting smart cities in developing countries: Policy insights from Vietnam. Telecommunications Policy, 42 (10), 845–859.

Wall, J., Lowry, P. B., & Barlow, J. B. (2015). Organisational violations of externally governed privacy and security rules: Explaining and predicting selective violations under conditions of strain and excess. Journal of the Association for Information Systems, 17 (1), 39–76.

Wang, Y., Zhang, A., Zhang, P., & Wang, H. (2019). Cloud-assisted EHR sharing with security and privacy preservation via consortium blockchain. IEEE Access, 7 , 136704–136719.

Wang, C., Zhang, N., & Wang, C. (2021). Managing privacy in the digital economy. Fundamental Research, 1 (5), 543–551.

Warren, S. D., & Brandeis, L. D. (1890). Right to privacy. Harvard Law Review, 4 , 193.

Wolford, B. (2020). “What is GDPR, the EU’s new data protection law?” Retrieved 2 Feb, 2020, from https://gdpr.eu/what-is-gdpr/ .

Xiong, W., & Lagerström, R. (2019). Threat modeling – A systematic literature review. Computers & Security, 84 , 53–69.

Xu, H., Dinev, T., Smith, J., & Hart, P. (2011). Information privacy concerns: Linking individual perceptions with institutional privacy assurances. Journal of the Association for Information Systems, 12 (12), 1.

Yang, Y., Zheng, X., Guo, W., Liu, X., & Chang, V. (2018). Privacy-preserving fusion of IoT and big data for e-health. Future Generation Computer Systems, 86 , 1437–1455.

Yang, L., Xue, H., & Li, F. (2014). Privacy-preserving data sharing in smart grid systems. In 2014 IEEE International Conference on Smart Grid Communications (SmartGridComm) , IEEE.

Yi, X., Miao, Y., Bertino, E. & Willemson, J. (2013). Multiparty privacy protection for electronic health records. In 2013 IEEE Global Communications Conference (GLOBECOM) , IEEE.

Zhang, Z. (2019). Technologies raise the effectiveness of airport security control. In 2019 IEEE 1st International Conference on Civil Aviation Safety and Information Technology (ICCASIT) .

Zhiyong, Z., Yongbin, X., & Jiaying, C. (2024). Digital economy, industrial structure upgrading and green innovation efficiency of family enterprises. International Entrepreneurship and Management Journal, 20 (1), 479–503.

Download references

Open Access funding enabled and organized by CAUL and its Member Institutions. This work was supported by Taibah University, Saudi Arabia, which provided a Ph.D. scholarship that covered funding for this work. This work was done at the University of Technology Sydney, Australia.

Author information

Authors and affiliations.

Faculty of Engineering and Information Technology, School of Computer Science, University of Technology Sydney, Ultimo, Sydney, NSW, 2007, Australia

Maha Ibrahim Alabsi & Asif Qumar Gill

Applied College, Taibah University, Al-Madinah Al-Munawarrah, Madina, Saudi Arabia

Maha Ibrahim Alabsi

You can also search for this author in PubMed   Google Scholar

Contributions

All authors have contributed to this manuscript and approved the published version. The first author participated in all the work, including drafting, reviewing, and updating the article. The second author contributed to the research approach, review, and revisions of the article.

Corresponding author

Correspondence to Maha Ibrahim Alabsi .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Alabsi, M.I., Gill, A.Q. A Systematic Review of Personal Information Sharing in Smart Cities: Risks, Impacts, and Controls. J Knowl Econ (2024). https://doi.org/10.1007/s13132-024-02126-1

Download citation

Received : 19 October 2023

Accepted : 02 June 2024

Published : 24 June 2024

DOI : https://doi.org/10.1007/s13132-024-02126-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Information privacy
• Privacy risk
• Privacy requirements
• Privacy control
• Privacy risk assessment
• Find a journal
• Publish with us
• Track your research

This paper is in the following e-collection/theme issue:

Published on 25.6.2024 in Vol 26 (2024)

Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses

Authors of this article:

• Xufei Luo 1, 2, 3, 4, 5   ; 
• Fengxian Chen 6 , PhD   ; 
• Di Zhu 7 , MPH   ; 
• Ling Wang 7 , MPH   ; 
• Zijun Wang 1, 2, 3, 4, 5   ; 
• Hui Liu 1, 2, 3, 4, 5   ; 
• Meng Lyu 7 , MPH   ; 
• Ye Wang 7 , MPH   ; 
• Qi Wang 8, 9 , PhD   ; 
• Yaolong Chen 1, 2, 3, 4, 5 , MD, PhD  

1 Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China

2 World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China

3 Institute of Health Data Science, Lanzhou University, Lanzhou, China

4 Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China

5 Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China

6 School of Information Science & Engineering, Lanzhou University, Lanzhou, China

7 School of Public Health, Lanzhou University, Lanzhou, China

8 Department of Health Research Methods, Evidence and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada

9 McMaster Health Forum, McMaster University, Hamilton, ON, Canada

Corresponding Author:

Yaolong Chen, MD, PhD

Evidence-Based Medicine Center

School of Basic Medical Sciences

Lanzhou University

No 199 Donggang West Road

Chengguan District

Lanzhou, 730000

Phone: 86 13893104140

Email: [email protected]

Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses.

Introduction

A systematic review is the result of a systematic and rigorous evaluation of evidence, which may or may not include a meta-analysis [ 1 ]. Owing to the strict methodology and comprehensive summary of evidence, high-quality systematic reviews are considered the highest level of evidence, positioned at the top of the evidence pyramid [ 2 ]. Additionally, high-quality systematic reviews and meta-analyses are often used to support the development of clinical practice guidelines, aid clinical decision-making, and inform health care policy formulation [ 3 ]. Currently, the methods of systematic reviews and meta-analyses are applied in various disciplines in medicine and beyond such as law [ 4 ], management [ 5 ], and economics [ 6 ], and have yielded positive results, contributing to the continuous advancement of these fields [ 7 ].

The process of conducting systematic reviews demands a substantial investment in terms of time, resources, human effort, and financial capital [ 8 ]. To expedite the development of systematic reviews and meta-analyses, various automated or semiautomated tools such as Covidence have been developed [ 9 , 10 ]. However, the emergence of large language models (LLMs), particularly chatbots such as GPT, presents a set of both challenges and opportunities in the realm of systematic reviews and meta-analyses [ 11 ]. Based on the emerging literature in this field, we here provide our perspectives on the potential for harnessing the capabilities of LLMs to accelerate the production of systematic reviews and meta-analyses, while also scrutinizing the potential impacts and delineating the crucial steps involved in this process.

The Process and Challenges of Performing a Systematic Review and Meta-Analysis

The procedures and workflows for conducting systematic reviews and meta-analyses are well-established. Currently, researchers often refer to the Cochrane Handbooks recommended by the Cochrane Library for intervention or diagnostic reviews [ 12 , 13 ]. In addition, some scholars and institutions have developed detailed guidelines on the steps and methodology for performing systematic reviews and meta-analyses [ 14 - 17 ]. Generally speaking, researchers should take the following steps to produce a high-quality systematic review and meta-analysis: determine the clinical question, register and draft a protocol, set inclusion and exclusion criteria, develop and implement a search strategy, screen the literature, extract data from included studies, assess the quality and risk of bias of included studies, analyze and processed data, write up the full text of the manuscript, and submit the manuscript for publication, as illustrated in Figure 1 . These different steps contain many subtasks; therefore, conducting a complete systematic review and meta-analysis requires fairly complex and time-consuming work.

Although systematic reviews and meta-analyses have been widely applied and play an important role in developing guidelines and informing clinical decision-making, their production process faces many challenges. One of these challenges is the long production time and large resource requirements. The average estimated time to complete and publish a systematic review is 67.3 weeks, requiring 5 researchers and costing approximately US $140,000 [ 18 , 19 ]. More recently, the development of automated and semiautomated tools using natural language processing and machine learning have accelerated systematic review and meta-analysis production to some extent [ 20 ], with studies showing that such tools can help to produce a systematic review and meta-analysis within 2 weeks [ 21 ]. However, these tools also have some limitations. First, no single tool can fully accelerate the entire production process of systematic reviews and meta-analyses. Second, these tools cannot process and analyze literature written in different languages. Finally, the reliability of the results generated by these automated and semiautomated tools needs further validation as they are not yet widely adopted for this purpose.

Applications of LLMs in Medical Research

Chatbots based on LLMs such as ChatGPT, Google Gemini, and Claude have become widely applied in medical research. These chatbots have proven to be valuable in tasks ranging from knowledge retrieval, language refinement, content generation, and medical exam preparation to literature assessment. ChatGPT has been shown to excel in accuracy, completeness, nuance, and speed when generating responses to clinical inquiries in psychiatry [ 22 ]. Moreover, LLMs such as ChatGPT play a pivotal role in automating the evaluation of medical literature, facilitating the identification of accurately reported research findings [ 23 ]. Despite their significant contributions, these chatbots are not without limitations. Challenges such as the potential for generating misleading content and susceptibility to academic deception necessitate further scholarly discourse on effective mitigation strategies. Standardized reporting practices may contribute to delineating the applications of ChatGPT and mitigating research biases [ 24 ].

ChatGPT has also demonstrated significant application potential and promise in the process of conducting systematic reviews and meta-analyses. Various studies [ 11 , 25 - 32 ] indicate that ChatGPT can play a pivotal role in formulating clinical questions, determining inclusion and exclusion criteria, screening literature, assessing publications, generating meta-analysis code, and assisting the full-text composition, among other relevant tasks. The details of these capabilities are summarized in Table 1 .

Potential Roles of LLMs in Producing Systematic Reviews and Meta-Analyses

Determine the research topic/question.

Determining the clinical question of interest represents the initial and paramount step in the process of conducting systematic reviews and meta-analyses. At this juncture, it is crucial to ascertain whether comparable systematic reviews and meta-analyses have already been published and to delineate the scope of the forthcoming review and meta-analysis. Generally, for interventional systematic reviews, the Patient, Intervention, Comparison, and Outcome (PICO) framework is considered for defining the scope and objectives of the research question [ 60 ]. In this context, ChatGPT serves a dual role. On the one hand, it expeditiously aids in searching for published systematic reviews and meta-analyses related to the relevant topics (see Multimedia Appendix 1 and Multimedia Appendix 2 ) [ 34 ]. On the other hand, ChatGPT assists in refining the clinical question that needs to be addressed (see Multimedia Appendix 3 ), facilitating prompt determination of the feasibility of undertaking the proposed study. However, it is important to be cautious of the retrieval of false literature [ 35 ].

Register and Write a Research Proposal

The registration and proposal writing process constitutes a pivotal preparatory phase for conducting systematic reviews and meta-analyses. Registration enhances research transparency, fosters collaboration among investigators, and mitigates the redundancy of research endeavors. Drafting a proposal helps in elucidating the research objectives and methods, providing robust support for the smooth execution of the study. For LLMs, generating preliminary registration information and initial proposal content is remarkably convenient and facile (see Multimedia Appendix 4 and Multimedia Appendix 5 ). For example, ChatGPT can assist researchers in generating the statistical methods for a research proposal [ 37 ]. However, considering that LLMs often generate fictitious literature, the content they produce may be inaccurate; thus, discernment and validation of the generated content remain essential considerations.

Define Inclusion and Exclusion Criteria

The inclusion and exclusion criteria for systematic reviews and meta-analyses are instrumental in determining the screening standards for studies. Therefore, strict and detailed inclusion and exclusion criteria contribute to the smooth and high-quality conduct of preparing systematic reviews and meta-analyses. The use of a chatbot based on LLMs can help in establishing the inclusion and exclusion criteria (see Multimedia Appendix 6 ) [ 38 ]; however, the inclusion criteria need to be optimized and adjusted according to the specific research objectives and the exclusion criteria should be based on the foundation of the inclusion criteria. Therefore, manual adjustments and optimizations are also necessary.

Develop a Search Strategy and Conduct Searches

ChatGPT can assist in formulating search strategies, using PubMed as an example [ 40 ]. Researchers can simply list their questions using the PICO framework and a search strategy can be quickly generated ( Multimedia Appendix 1 and Multimedia Appendix 2 ). Based on the generated search strategy, one method is to copy the strategy from ChatGPT and paste it into the PubMed search box for direct retrieval [ 40 , 41 ]. Another approach involves using the OpenAI application programming interfaces (APIs) to invoke PubMed APIs with the search strategy generated by ChatGPT. This facilitates searching the PubMed database, obtaining search results, and applying predetermined inclusion and exclusion criteria. Subsequently, ChatGPT can be used to filter the search results, exporting and recording the filtered results in JSON format. This integrated process encompasses search strategy formulation, retrieval, and filtering. However, the direct use of LLMs to generate search strategies and complete the one-stop process of searching and screening may not yet be mature, and this poses a significant challenge for generating the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) flowchart. Therefore, we suggest using LLMs to generate search strategies, which should then be optimized and modified by librarians and computer experts (specializing in LLMs) before manually searching the databases. Additionally, to use search strategies transparently and reproducibly, the detailed prompts used should be reported [ 40 , 42 ].

Screen the Literature

Literature screening is one of the most time-consuming steps in the creation of systematic reviews and meta-analyses. Prior to the advent of ChatGPT, there were already many automated and semiautomated tools available for literature screening, such as Covidence, EPPI-Reviewer, DistillerSR, and others [ 39 ]. With the emergence of ChatGPT, researchers can now train the model based on predefined inclusion criteria. Subsequently, ChatGPT can be used to automatically screen records retrieved from databases and obtain the filtered results. Previous studies suggested that using ChatGPT in the literature selection process for a meta-analysis substantially diminishes the workload while preserving a recall rate on par with that of manual curation [ 28 , 44 - 47 ].

Extract the Data

Data extraction involves obtaining information from primary studies and serves as a primary source for systematic reviews and meta-analyses. Generally, when conducting systematic reviews and meta-analyses, basic information must be extracted from the original studies, such as publication date, country of conduct, and the journal of publication. Additionally, characteristics of the population, such as patient samples, age, gender/sex, and outcome data, are also extracted, including event occurrences, mean change values, and total sample size. Currently, tools based on natural language processing and LLMs, such as ChatGPT and Claude, demonstrate high accuracy in extracting information from PDF documents (see Multimedia Appendix 7 for an example) [ 47 - 50 ]. However, it is important to note that despite their promising capabilities, manual verification remains a necessary step in the data extraction process when using these artificial intelligence (AI) tools [ 61 ]. Using LLMs to extract data can help avoid random errors; however, caution is still required when extracting data from figures or tables [ 47 - 50 ].

Assess the Risk of Bias

Assessing the bias of risk involves evaluating the internal validity of studies included in research. For randomized controlled trials, tools such as Risk of Bias (RoB) [ 62 ] or its updated version RoB 2 [ 63 ] are typically used, with an estimated review time of 10-15 minutes per trial. However, automated tools such as RobotReviewer can streamline the extraction and evaluation process in batches [ 51 - 53 ], thereby improving efficiency, although manual verification is still necessary. Additionally, chatbots based on LLMs can aid in risk of bias assessment (see Multimedia Appendix 8 ), and their accuracy appears to be comparable to that of human evaluations [ 23 ].

Analyze the Data/Meta-Analysis

Data analysis serves as the source of systematic review results, typically encompassing basic information and outcome findings. The meta-analysis may be one outcome, along with potential components such as subgroup analysis, sensitivity analysis, meta-regression, and detection of publication bias. Numerous software options are available to facilitate these data analyses, including Stata, RevMan, Rstudio, and others [ 43 ]. Currently, it appears that chatbots based on LLMs may not fully execute data analysis independently, although they can extract the relevant information. Subsequently, one can employ corresponding software for comprehensive data analysis. Alternatively, after extracting information with chatbots, the ChatGPT Code Interpreter can assist in analysis and generating graphical results, although this requires a subscription to ChatGPT Plus. Moreover, an LLM markedly accelerates the data analysis process, empowering researchers to handle larger data sets with greater efficacy [ 54 ].

Draft the Full Manuscript

The complete drafting of systematic reviews and meta-analyses should adhere to the PRISMA reporting guidelines [ 64 ]. It is not advisable to use chatbots such as ChatGPT for article composition. On the one hand, the accuracy and integrity of content generated by ChatGPT require human verification. On the other hand, various research types and journals have different requirements for full-length articles, making it challenging to achieve uniformity in the generated content. However, using tools such as GPT for language refinement and adjusting the content logic can be considered to enhance the quality and readability of the article [ 33 , 55 ]. It is important to declare the use of GPT-related tools in the methods, acknowledgments, or appendices of the article to ensure transparency [ 24 , 65 ].

Submit and Publish

Submission and publication represent the final steps in the process of conducting systematic reviews and meta-analyses, aside from subsequent updates. At this stage, the potential role of LLM-based tools is to assist authors in recommending suitable journals (see Multimedia Appendix 9 ). These tools might also aid in crafting components required along with submission of the manuscript such as cover letters and highlights [ 59 ]. However, it is imperative to emphasize that the content generated by these tools requires manual verification to ensure accuracy, and all authors should be accountable for the content generated by LLMs.

Benefits and Drawbacks of Using LLMs

Systematic reviews and meta-analyses are crucial evidence types that support the development of guidelines [ 3 ]. The benefits of employing LLM-based chatbots in the production of systematic reviews and meta-analyses include increased speed, such as in the stages of evidence searching, data extraction, and assessment of bias risk; these tools can also enhance accuracy by reducing human errors such as those made while extracting essential information and pooling data. However, there are also drawbacks of these applications of LLMs, such as the potential for generating hallucinations, the requirement for human verification owing to the poor reliability of the models, and that the entire systematic review process is not replicable. Moreover, when interacting with LLM chatbots, it is important to manage data privacy. In particular, when using LLMs to analyze data, especially when including personal patient information, ethical approval and management must be properly addressed.

Challenges and Solutions

While LLMs can assist in accelerating the production of systematic reviews and meta-analyses in some steps, enhancing accuracy and transparency, and saving resources, they also face several challenges. For instance, LLMs cannot promptly update their versions and information. For example, ChatGPT 3.5 has been trained on data available in 2021. Thus, limitations such as the length of prompts and token constraints, as well as restrictions related to context associations, may potentially impact the overall results and user experience [ 25 ]. Although LLM-based autonomous agents have made strides in tasks related to systematic reviews and meta-analyses, their applications are still associated with various issues related to personalization, updating knowledge, strategic planning, and complex problem-solving. The development of LLM-driven autonomous agents adept at systematic reviews and meta-analyses warrants further exploration [ 66 ]. The use of LLMs as centrally controlled intelligent agents encompasses the ability to handle precise literature screening, extract and analyze complex data, and assist in manuscript composition, as highlighted by proof-of-concept demonstrations such as MetaGPT [ 67 ]. Moreover, the continuous growth of the use of LLMs can pose a significant challenge in ensuring the accuracy of information provided in systematic reviews, particularly if LLMs are indiscriminately overused.

To better facilitate the use of tools such as ChatGPT in systematic reviews and meta-analyses, we believe that, first and foremost, authors should understand the scope and scenarios for applying ChatGPT, clearly defining which steps can benefit from these tools. Second, for researchers, collaboration with computer scientists and AI engineers is crucial to optimize the prompts and develop integrated tools based on LLMs, such as web applications. These tools can assist in seamless transitions between different tasks in the systematic review process. Lastly, for journal editors, collaboration with authors and reviewers is essential to adhere to reporting and ethical principles associated with the use of GPT and similar tools [ 24 , 68 ]. This collaboration aims to promote transparency and integrity, while preventing indiscriminate overuse in the application of LLMs in systematic reviews and meta-analyses.

Future Perspectives and Conclusion

The emergence of LLMs could have a significant impact on the production of systematic reviews and meta-analyses. In this process, the application of chatbots such as ChatGPT has the potential to speed up certain steps such as literature screening, data extraction, and risk of bias assessment, which are processes that typically consume a considerable amount of time. However, it is important to note that if AI methods such as GPT are employed in performing systematic reviews, disclosure and declaration of the use of these tools are essential. This includes specifying the AI tools used, their roles, and the areas of application within the review process, among other relevant information for full disclosure [ 24 ]. In this context, developing a reporting guideline is warranted to guide the application of LLM tools in systematic reviews and meta-analyses. Although the PRISMA 2020 guideline briefly addresses the use of automation technologies, its coverage is limited to steps such as screening, and there is a lack of comprehensive guidance on the broader spectrum of applications [ 64 ].

Acknowledgments

ChatGPT 3.5 designed by OpenAI was used to help with language editing. The authors take the ultimate responsibility for the content of this publication.

Authors' Contributions

XL and YC were responsible for conceptualization of the article. XL, FC, DZ, and LW generated the examples with the large language models and wrote the first draft of the article. XL, ZW, HL, ML, YW, QW, and YC reviewed and edited the manuscript. YC supervised the study, takes full responsibility for the work and conduct of the study, has access to the data, and controlled the decision to publish. All authors read the final manuscript and approved the submission.

Conflicts of Interest

None declared.

Using ChatGPT 4.0 to assist in generating PubMed search strategies for assessing systematic reviews.

The results obtained after searching the PubMed database based on the search strategy generated by ChatGPT.

Using ChatGPT 4.0 to assist in optimizing the clinical question for conducting a systematic review and meta-analysis.

Using ChatGPT 4 to generate PROSPERO (International Prospective Register of Systematic Reviews) registration information.

Proposal of a systematic review and meta-analysis related to exercises for osteoarthritis generated by Claude 3 based on the provided prompts.

The inclusion and exclusion criteria for a systematic review and meta-analysis on exercise therapy for osteoarthritis based on GPT-4.

Using Claude 3 for data extraction from PDF documents: an example with three randomized controlled trials.

Using Claude 3 for risk of bias assessment: an example with two randomized controlled trials.

Using GPT-4 to assist in selecting target journals for submission of a systematic review and meta-analysis.

• Jahan N, Naveed S, Zeshan M, Tahir MA. How to conduct a systematic review: a narrative literature review. Cureus. Nov 04, 2016;8(11):e864. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wallace SS, Barak G, Truong G, Parker MW. Hierarchy of evidence within the medical literature. Hosp Pediatr. Aug 01, 2022;12(8):745-750. [ CrossRef ] [ Medline ]
• Institute of Medicine, Board on Health Care Services, Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Graham R, Mancher M, Miller Wolman D, et al. Clinical Practice Guidelines We Can Trust. Washington, DC. National Academies Press; 2011.
• Bystranowski P, Janik B, Próchnicki M, Skórska P. Anchoring effect in legal decision-making: a meta-analysis. Law Hum Behav. Feb 2021;45(1):1-23. [ CrossRef ] [ Medline ]
• Geyskens I, Krishnan R, Steenkamp JEM, Cunha PV. A review and evaluation of meta-analysis practices in management research. J Management. Feb 05, 2008;35(2):393-419. [ CrossRef ]
• Bagepally BS, Chaikledkaew U, Chaiyakunapruk N, Attia J, Thakkinstian A. Meta-analysis of economic evaluation studies: data harmonisation and methodological issues. BMC Health Serv Res. Feb 15, 2022;22(1):202. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. Mar 07, 2018;555(7695):175-182. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tsertsvadze A, Chen Y, Moher D, Sutcliffe P, McCarthy N. How to conduct systematic reviews more expeditiously? Syst Rev. Nov 12, 2015;4:160. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Scott AM, Forbes C, Clark J, Carter M, Glasziou P, Munn Z. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol. Oct 2021;138:80-94. [ CrossRef ] [ Medline ]
• Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. J Clin Epidemiol. Apr 2022;144:22-42. [ CrossRef ] [ Medline ]
• Qureshi R, Shaughnessy D, Gill KAR, Robinson KA, Li T, Agai E. Are ChatGPT and large language models "the answer" to bringing us closer to systematic review automation? Syst Rev. Apr 29, 2023;12(1):72. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.4. Aug 2023. URL: https://training.cochrane.org/handbook [accessed 2024-06-11]
• Deeks J, Bossuyt P, Leeflang M, Takwoingi Y. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Jul 2023. URL: https://training.cochrane.org/handbook-diagnostic-test-accuracy/current [accessed 2024-06-11]
• Xiao Y, Watson M. Guidance on conducting a systematic literature review. J Plan Educ Res. Aug 28, 2017;39(1):93-112. [ CrossRef ]
• Muka T, Glisic M, Milic J, Verhoog S, Bohlius J, Bramer W, et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol. Jan 2020;35(1):49-60. [ CrossRef ] [ Medline ]
• Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol. Jan 04, 2019;70:747-770. [ CrossRef ] [ Medline ]
• Tawfik GM, Dila KAS, Mohamed MYF, Tam DNH, Kien ND, Ahmed AM, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47:46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. Feb 27, 2017;7(2):e012545. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Michelson M, Reuter K. The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun. Dec 2019;16:100443. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev. Jul 11, 2019;8(1):163. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM. A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol. May 2020;121:81-90. [ CrossRef ] [ Medline ]
• Luykx JJ, Gerritse F, Habets PC, Vinkers CH. The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two-layer assessment. World Psychiatry. Oct 15, 2023;22(3):479-480. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Roberts RH, Ali SR, Hutchings HA, Dobbs TD, Whitaker IS. Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards. BMJ Health Care Inform. Oct 2023;30(1):e100830. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Luo X, Estill J, Chen Y. The use of ChatGPT in medical research: do we need a reporting guideline? Int J Surg. Dec 01, 2023;109(12):3750-3751. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alshami A, Elsayed M, Ali E, Eltoukhy A, Zayed T. Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions. Systems. Jul 09, 2023;11(7):351. [ CrossRef ]
• Mahuli SA, Rai A, Mahuli AV, Kumar A. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J. Jul 2023;235(2):90-92. [ CrossRef ] [ Medline ]
• van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, van der Palen J, Doggen CJM, Lenferink A. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open. Jul 07, 2023;13(7):e072254. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods. Mar 14, 2024:online ahead of print. [ CrossRef ] [ Medline ]
• Gwon YN, Kim JH, Chung HS, Jung EJ, Chun J, Lee S, et al. The use of generative AI for scientific literature searches for systematic reviews: ChatGPT and Microsoft Bing AI performance evaluation. JMIR Med Inform. May 14, 2024;12:e51187-e51187. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hossain MM. Using ChatGPT and other forms of generative AI in systematic reviews: challenges and opportunities. J Med Imaging Radiat Sci. Mar 2024;55(1):11-12. [ CrossRef ] [ Medline ]
• Giunti G, Doherty CP. Cocreating an automated mHealth apps systematic review process with generative AI: design science research approach. JMIR Med Educ. Feb 12, 2024;10:e48949. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nashwan AJ, Jaradat JH. Streamlining systematic reviews: harnessing large language models for quality assessment and risk-of-bias evaluation. Cureus. Aug 2023;15(8):e43023. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huang J, Tan M. The role of ChatGPT in scientific communication: writing better scientific review articles. Am J Cancer Res. 2023;13(4):1148-1154. [ FREE Full text ] [ Medline ]
• Issaiy M, Ghanaati H, Kolahi S, Shakiba M, Jalali AH, Zarei D, et al. Methodological insights into ChatGPT's screening performance in systematic reviews. BMC Med Res Methodol. Mar 27, 2024;24(1):78. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Branum C, Schiavenato M. Can ChatGPT accurately answer a PICOT question? Assessing AI response to a clinical question. Nurse Educ. 2023;48(5):231-233. [ CrossRef ] [ Medline ]
• Macdonald C, Adeloye D, Sheikh A, Rudan I. Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis. J Glob Health. Feb 17, 2023;13:01003. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Richard E, Pozzi A. Using ChatGPT to develop the statistical analysis plan for a randomized controlled trial: a case report. Zurich Open Repository and Archive. Preprint posted online on October 17, 2023. [ CrossRef ]
• Hutson M. How AI is being used to accelerate clinical trials. Nature. Mar 2024;627(8003):S2-S5. [ CrossRef ] [ Medline ]
• Van der Mierden S, Tsaioun K, Bleich A, Leenaars CHC. Software tools for literature screening in systematic reviews in biomedical research. Altex. 2019;36(3):508-517. [ CrossRef ] [ Medline ]
• Wang S, Scells H, Koopman B, Zuccon G. Can ChatGPT write a good Boolean query for systematic review literature search? arXiv. Preprint posted online on February 9, 2023. [ FREE Full text ] [ CrossRef ]
• Alaniz L, Vu C, Pfaff MJ. The utility of artificial intelligence for systematic reviews and Boolean query formulation and translation. Plast Reconstr Surg Glob Open. Oct 2023;11(10):e5339. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guimarães NS, Joviano-Santos JV, Reis MG, Chaves RRM, Observatory of Epidemiology‚ Nutrition‚ Health Research (OPENS). Development of search strategies for systematic reviews in health using ChatGPT: a critical analysis. J Transl Med. Jan 02, 2024;22(1):1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tantry TP, Karanth H, Shetty PK, Kadam D. Self-learning software tools for data analysis in meta-analysis. Korean J Anesthesiol. Oct 2021;74(5):459-461. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cai X, Geng Y, Du Y, Westerman B, Wang D, Ma C, et al. Utilizing ChatGPT to select literature for meta-analysis shows workload reduction while maintaining a similar recall level as manual curation. medRxiv. Preprint posted online on September 7, 2023. [ CrossRef ]
• Eugene S, Istvan D, Gauransh K. Assessing the ability of ChatGPT to screen articles for systematic reviews. arXiv. Preprint posted online on July 12, 2023. [ CrossRef ]
• Kohandel Gargari O, Mahmoudi MH, Hajisafarali M, Samiee R. Enhancing title and abstract screening for systematic reviews with GPT-3.5 turbo. BMJ Evid Based Med. Jan 19, 2024;29(1):69-70. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guo E, Gupta M, Deng J, Park Y, Paget M, Naugler C. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. Jan 12, 2024;26:e48996. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Polak MP, Morgan D. Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nat Commun. Feb 21, 2024;15(1):1569. [ CrossRef ] [ Medline ]
• Mahmoudi H, Chang D, Lee H, Ghaffarzadegan N, Jalali MS. A critical assessment of large language models for systematic reviews: utilizing ChatGPT for complex data extraction. SSRN. Preprint posted online on April 19, 2024. [ CrossRef ]
• Sun Z, Zhang R, Doi SA, Furuya-Kanamori L, Yu T, Lin L, et al. How good are large language models for automated data extraction from randomized trials? medRXiv. Preprint posted online on February 21, 2024. [ CrossRef ]
• Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. Jan 2016;23(1):193-201. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lai H, Ge L, Sun M, Pan B, Huang J, Hou L, et al. Assessing the risk of bias in randomized clinical trials with large language models. JAMA Netw Open. May 01, 2024;7(5):e2412687. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pitre T, Jassal T, Talukdar JR, Shahab M, Ling M, Zeraatkar D. ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: a methods study. medRXiv. Preprint posted online on November 22, 2023. [ CrossRef ]
• Rasheed Z, Waseem M, Ahmad A, Kemell KK, Xiaofeng W, Nguyen Duc A, et al. Can large language models serve as data analysts? A multi-agent assisted approach for qualitative data analysis. arXiv. Preprint posted online on February 2, 2024.
• Kim S. Using ChatGPT for language editing in scientific articles. Maxillofac Plast Reconstr Surg. Mar 08, 2023;45(1):13. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gao CA, Howard FM, Markov NS, Dyer EC, Ramesh S, Luo Y, et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digit Med. Apr 26, 2023;6(1):75. [ CrossRef ] [ Medline ]
• Kim JK, Chua M, Rickard M, Lorenzo A. ChatGPT and large language model (LLM) chatbots: the current state of acceptability and a proposal for guidelines on utilization in academic medicine. J Pediatr Urol. Oct 2023;19(5):598-604. [ CrossRef ] [ Medline ]
• Mugaanyi J, Cai L, Cheng S, Lu C, Huang J. Evaluation of large language model performance and reliability for citations and references in scholarly writing: cross-disciplinary study. J Med Internet Res. Apr 05, 2024;26:e52935. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Firdausa Nuzula I, Miftahul Amri M. Will ChatGPT bring a new paradigm to HR World? A critical opinion article. J Manag Stud Devel. Apr 24, 2023;2(02):142-161. [ CrossRef ]
• Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. J Med Libr Assoc. Oct 2018;106(4):420-431. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Roberts R. I tested how well ChatGPT can pull data out of messy PDFs. Source. Mar 01, 2023. URL: https://source.opennews.org/articles/testing-pdf-data-extraction-chatgpt/ [accessed 2024-06-11]
• Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Cochrane Bias Methods Group, et al. Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. Oct 18, 2011;343:d5928. [ CrossRef ] [ Medline ]
• Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. Aug 28, 2019;366:l4898. [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gaggioli A. Ethics: disclose use of AI in scientific manuscripts. Nature. Feb 14, 2023;614(7948):413-413. [ CrossRef ] [ Medline ]
• Wang L, Ma C, Feng X, Zhang Z, Yang H, Zhang J, et al. A survey on large language model based autonomous agents. Front Comput Sci. Mar 22, 2024;18(6):186345. [ FREE Full text ] [ CrossRef ]
• Hong S, Zhuge M, Chen J, Zheng X, Cheng Y, Zhang C, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv. Preprint posted online on November 6, 2023.
• Flanagin A, Pirracchio R, Khera R, Berkwits M, Hswen Y, Bibbins-Domingo K. Reporting use of AI in research and scholarly publication-JAMA Network Guidance. JAMA. Apr 02, 2024;331(13):1096-1098. [ CrossRef ] [ Medline ]

Abbreviations

Edited by G Eysenbach; submitted 20.02.24; peer-reviewed by A Jafarizadeh, M Chatzimina, AS Van Epps; comments to author 03.05.24; revised version received 21.05.24; accepted 29.05.24; published 25.06.24.

©Xufei Luo, Fengxian Chen, Di Zhu, Ling Wang, Zijun Wang, Hui Liu, Meng Lyu, Ye Wang, Qi Wang, Yaolong Chen. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 25.06.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.