What are Systematic Reviews

A systematic Review is a specific methodology for research, developed for thesynthesis and appraisal of the available evidence pertaining and relevant toa particular research question, focused topic area, or phenomenon of interest[1, 2]. It aims to present a comprehensive evaluation of a research topic byusing a trustworthy, rigorous, and auditable methodology [2]. Synonyms of this methodology include overview, research review, research synthesis, research integration, systematic overview, systematic research synthesis, integrative re-search review, and integrative review[1]. Systematic Reviews are supported by a well-defined process and that makethem different from traditional or conventional literature reviews in the followingrespects [3, 4]):

• Definition and documentation of a systematic review protocol in advance of conducting the review, to specify the research question(s) and the procedures to be used to perform the review;
• Definition of a search strategy as part of the protocol, to identify as much of the relevant literature related to the research question(s);
• Documentation of a search strategy, such that it can be followed rigorously;
• Description of the explicit inclusion and exclusion criteria as part of theprotocol, to be used to appraise each potential primary study;
• Description of quality assessment mechanisms as part of the protocol, to evaluate each primary study; and,
• Description of review and cross-checking processes as part of the protocol, and involving multiple independent researchers, in order to control researcher bias.

How to perform Systematic Reviews

Although there are slight differences in the ways different systematic reviewprocesses have been documented, they all involve planning, execution, analysis,and the reporting of results (as shown in figure 1):

Planning Stage aims to specifically define a Protocol, which contains all the necessary information and procedures for the following stages of Execution and Summarization. Examples of the information and procedures required include the research question(s), the keywords for search strings, the searchengines/online libraries, as well as the inclusion and exclusion criteria.

Execution Stage consists of three steps: a) Identifying the primary studies in the search engines/online libraries (which were defined in the protocol); b)Selecting the identified studies, based on the inclusion and exclusion criteria; c)Extracting data from the selected studies.

Summarization Stage occurs when the extracted data are summarized and/or analysed to answer the research question(s) defined in the protocol.

After these three stages have been completed, the results should be reported through technical reports or scientific papers to show the current state of research in the field in question, and to identify current challenges and/or gaps in existing research [5].

Planning

1. Research Question

A systematic review is based on a pre-defined specific research question. A clear research question will help clarify the eligibility criteria for inclusion of relevant studies and exclusion of irrelevant studies) [6]. Specifying the research questions is the most important part of any systematic review [2].

• The review questions drive the entire systematic review methodology.
• The search process must identify primary studies that address the research questions.
• The data extraction process must extract the data items needed to answer the questions.
• The data analysis process must synthesise the data in such a way that the questions can be answered.

The critical issue in any systematic review is to ask the *right* question.

Pose research question(s) in a structured and explicit way will also contributeto a more systematic, well-analysed and efficient systematic review.

In this tutorial, we introduce two frameworks, the PIO (Population, Interventions, and Outcomes) and the PICOC (Population, Interventions, Comparison, Outcomes and Context):

PIO Framework [7] Considers questions that consist of three elements:

• Population, i.e. people affected by the intervention.
• Interventions, i.e. comparison of groups with and without the intervention or exposure of interest.
• Outcomes, i.e. changes due to interventions.

PICOC Framework [8] Frames a question from five viewpoints:

• Who?– The Population in which the evidence is collected, i.e. whichgroup of people, programs or businesses are of interest for the review?
• What or How?– The Intervention applied in the empirical study, i.e.which technology, tool or procedure is under study?
• Compared to what?– The Comparison, i.e. what is the comparison or alternative to the intervention?
• Outcomes?, i.e. what are the possible Outcomes, such as the measures of how effective the interventions have been.
• In what kind of circumstances?– The Context of the study, i.e. what isthe context in which the intervention is delivered?

2. Review Protocol

A systematic review protocol is a formal and rather concrete plan for the execution of the systematic review. Kitchenham [2] notes that a pre-defined protocolis necessary to reduce the possibility of researcher bias.

The contents of a systematic review protocol in many ways foreshadows the structure of the final report. It describes the background context for the research, the specific research questions, the planned search strategy, the study selection criteria, the study selection procedures, the study quality assessment checklists and procedures, the data extraction strategy, the data synthesis strategy, the dissemination strategy and the project timetable [2].

3. Search Strategy

The search queries were formulated in three stages [9]: (a) creating primarykeywords of search strings based on PICOC; (b) identifying synonyms for thekeywords, and (c) constructing a search string.

Search strings were created to be used in the search process, applying Boolean operators (OR, AND) [10]. Without changing the search terms, the searchstrings were personalized to be adapted to the syntaxes of each of the databases.

Useful Tips

• Previous review studies conducted in the same/related research topic can help determine an appropriate strategy.
• Before executing the systematic review, it is necessary to evaluate theplanned review. A way to perform such evaluation is to ask experts toreview the protocol. Another way to evaluate the planning is to test theprotocol execution. The review is executed in a reduced set of selected sources. If the obtained results are not suitable, the protocol must bereviewed and a new version must be created [1].

Execution

During the Execution stage, the search in the defined sources will be conducted, the studies obtained will be evaluated according to the inclusion andexclusion criteria. Finally, the relevant information to the research question willbe extracted from the selected studies [1].

1. Selection Crtiteria

In order to reduce the likelihood of bias, selection criteria should be decided during the protocol definition, but they might be refined during the Execution process.

Examples of study selection criteria[2] for considering:

• Language
• Journal
• Authors
• Setting
• Participants or subjects
• Research Design
• Sampling method
• Date of publication

Useful Reference

M. Gusenbauer and N. R. Haddaway, “Which academic search systems aresuit-able for systematic reviews or meta-analyses? evaluating retrieval qual-ities ofgoogle scholar, pubmed, and 26 other resources,”Research synthesismethods,vol. 11, no. 2, pp. 181–217, 2020

2. Data Extraction

Data extraction should be performed followed by the pre-defined systematic review protocol in the Planning stage which has described data extraction forms, and procedures for data extraction.

Kitchenham [3] suggests performing data extraction by two or more researchers and settling disagreements by consensus or by use of additional re-searchers.

Tips: Data monitoring could also be performed in this phase. It is suggested that multiple reports of the same study can be identified, and missingor unpublished data can be sought from the publications’ authors.

When data has been extracted, it should be grouped and summarised so asto shed light on the research questions for the systematic review. As with otherstages, the procedures to be followed should be pre-defined in the protocol.

Kitchenham [3] discusses options for combing data from different types ofstudies, and combining different types of data. Where some studies are of much higher quality, it is possible to perform sensitivity analyses to determine the effects on the synthesis results of ignoring low quality publications [4].

Useful Tools/Resources

Despite the advantages of a systematic review, as good coverage, replicability and reliability, its process is more laborious than the one related to informal literature research [3]. Thus, considering that there are several stages to be executed and several documents to be managed, the support of computational tools is essential to facilitate the work and enable higher quality in the execution process [5].

1. PRISMA Flow diagram

The flow diagram depicts the flow of information through the different phasesof a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions (http://prisma-statement.org/PRISMAStatement/FlowDiagram).

2. Mendeley

Mendeley is a free software which could help in organising research as a reference manager and also collaborate with other researchers online with the latest research. There is a possibility to import papers and read from anywhere online[11].

3. Excel

Use Excel to keep track of all resources (https://www.insidehighered.com/blogs/gradhacker/organizing-your-literature-spreadsheet-style).

Advantages:

• It doesn’t require buying any software
• It doesn’t take hours to learn

4. Other Useful Tools/References

• Systematic Review Toolbox http://systematicreviewtools.com/
• C. Marshall and P. Brereton, “Systematic review toolbox: a catalogueoftools to support systematic reviews,” inProceedings of the 19th Interna-tional Conference on Evaluation and Assessment in Software Engineer-ing,pp. 1–6, 2015
• H. Harrison, S. J. Griffin, I. Kuhn, and J. A. Usher-Smith, “Software tool-sto support title and abstract screening for systematic reviews in health-care:an evaluation,”BMC medical research methodology, vol. 20, no. 1,p. 7,2020
• C. Kohl, E. J. McIntosh, S. Unger, N. R. Haddaway, S. Kecke, J. Schie-mann, and R. Wilhelm, “Online tools supporting the conduct and re-portingof systematic reviews and systematic maps: a case study on cadimaandreview of existing tools,”Environmental evidence, vol. 7, no. 1, p. 8,2018

Summarization

After the Execution stage, the extracted data will be summarized and/or be critically analyzed based on the review question(s)/the data synthesis strategy pre-defined during the Planning phase.

Data synthesis consists of tabulation of the extracted information about thestudies (i.e. intervention, population, context, sample sizes, outcomes, study quality), and they should be tabulated in a manner consistent with the review question(s). Importantly, tables should be structured to highlight similaritiesand differences between study outcomes [3].

It is suggested to identify whether results from studies are consistent withone another (i.e. homogeneous) or inconsistent (e.g. heterogeneous) in thesystematic review. Results may be tabulated to display the impact of potentialsources of heterogeneity, e.g. study type, study quality, and sample size [3].

Reporting the review is a single-stage phase. Usually, systematic reviews are reported using two formats: in a technical report and in a journal or conference papers [3]. The structure and contents of reports are guided by the pre-defined systematic review protocol.

Must-Read References

• B. Kitchenham and S. Charters, “Guidelines for performing sys- tematicliterature reviews in software engineering,” Eng., vol. 2, 2007, Art. no.1051.
• J. Biolchini, P. G. Mian, A. Candida, and C. Natali, “Systematic reviewin software engineering,” Eng., vol. 679, pp. 165–176, May 2005.
• B. A. Kitchenham, “Procedures for Performing Systematic Reviews”, Soft-ware Engineering Group , Keele University, Keele, UK, Tech. Rep. TR/SE0401, Jul. 2004.

References

[1] J. Biolchini, P. G. Mian, A. C. C. Natali, and G. H. Travassos, “Systematicreview in software engineering,”System Engineering and Computer ScienceDepartment COPPE/UFRJ, Technical Report ES, vol. 679, no. 05, p. 45,2005.

[2] B. Kitchenham and S. Charters, “Guidelines for performing systematicliterature reviews in software engineering,” 2007.

[3] B. Kitchenham, “Procedures for performing systematic reviews,”Keele,UK, Keele University, vol. 33, no. 2004, pp. 1–26, 2004.

[4] M. Staples and M. Niazi, “Experiences using systematic review guidelines,”Journal of Systems and Software, vol. 80, no. 9, pp. 1425–1437, 2007.

[5] E. Hernandes, A. Zamboni, S. Fabbri, and A. D. Thommazo, “Using gqmand tam to evaluate start-a tool that supports systematic review,”CLEIElectronic Journal, vol. 15, no. 1, pp. 3–3, 2012.

[6] A. Pollock and E. Berge, “How to do a systematic review,”InternationalJournal of Stroke, vol. 13, no. 2, pp. 138–156, 2018.

[7] K. S. Khan, R. Kunz, J. Kleijnen, and G. Antes, “Five steps to conductinga systematic review,”Journal of the royal society of medicine, vol. 96, no. 3,pp. 118–121, 2003.

[8] C. Wohlin, P. Runeson, M. H ̈ost, M. C. Ohlsson, B. Regnell, andA. Wessl ́en,Experimentation in software engineering. Springer Science &Business Media, 2012.

[9] C. K. Pereira, S. W. M. Siqueira, B. P. Nunes, and S. Dietze, “Linkeddata in education: a survey and a synthesis of actual research and futurechallenges,”IEEE Transactions on Learning Technologies, vol. 11, no. 3,pp. 400–412, 2017.