Conceptualising the process and an approach to websearching in complex systematic reviews: an initial conceptual model and description of user interaction for secondary searches.

Websearching – that is, searching the internet directly through a search engine as opposed to searching named websites specifically – has an unclear role in the process of searching for studies for systematic reviews (1-5). Given the vastness of the internet, it seems useful, but how best to do this?

We have been thinking about this in the context of undertaking complex reviews (defined in Figure 1).

Figure 1: What is a ‘complex systematic review’? Mahtani et al. (2018)

Our hypothesis is that websearching should be used as a secondary search, shortly after data extraction has been completed. In this blog post, we illustrate a first-draft of a conceptual process model that we have developed and we briefly illustrate a user’s interaction with the process model to explain the idea. The model has been developed based on our experience of conducting complex reviews. We are currently testing the model.

Figure 2: An initial conceptual process model to illustrate the possibility of secondary searches in complex systematic reviews

A user’s interaction with the process model (Figure 2)

For the purpose of illustration, let us assume that the review in question is a systematic review of intervention effectiveness.

The model (set out in Figure 2) is defined by a series of stages. These are enumerated in the model and described below using this enumeration.

1. Protocol is agreed/published

The work of undertaking the systematic review starts after the protocol is agreed/published. The protocol should indicate that secondary searches are planned and list the methods and approach used following best practice guidance.

2. Primary searches

We assume that A Conventional Approach to searching would be used (2). That is, a search process led by bibliographic database searching followed by non-database search methods, as appropriate, for example: bibliographic database searches followed by searches of trials registers, searching conference proceedings, and searching specific websites which have been defined a priori in the protocol (7). 

3. Study selection

Study selection (aka screening) of all items identified at Stage 2 above. This stage represents title/abstract and full-text screening in the model.

4. Data extraction and quality appraisal

Studies are the unit of interest but we anticipate dealing with multiple reports of the same study. Multiple reports are compiled for data extraction and study quality is appraised. When this stage is completed, the searcher (librarian)/researcher undertaking the searching needs to know the following for the secondary searches (Stage 2.1):

·       Study name: that is the name given to the study or the name by which the study is known;

·       Intervention names: if not already known and searched for, or new names, which have been identified from the primary searches, but which were omitted in the primary searches. For reviews of medicines, extracting trial names or trial registry numbers would also be useful;

·       Study location: place and/or country;

·       Names of principal study authors or  sponsors; and

·       Titles of study reports: that is, the title of each eligible published or unpublished item fulfilling inclusion in the review.

2.1 Secondary searches

Secondary searches are positioned after Stage 5 but before Stage 6 because they require data from the primary searches (Stage 2) to operationalise the search method. Data needed are defined above.

We hypothesise that it is desirable to begin secondary searching after Stage 5 to ensure that only confirmed eligible study reports/studies are searched for. We find that, in complex reviews, it is still possible that seemingly eligible studies might be excluded as late as the data extraction stage. We want to avoid wasted effort in the secondary search process as well as any confusion in reporting in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) flow-chart where ineligible studies are searched for and later excluded.

The purpose of secondary searches is to identify any new studies/study reports, or hints to studies, which might have been overlooked by the original searches at Stage 2. The mechanism of action for secondary searches is largely different to the search methods used at Stage 2. It is the difference between keyword searching using a search strategy of pre-defined search terms and identifying links between study reports (citation searching) or identification of unpublished or unlinked study reports/data through new search terms, such as the name of the studies fulfilling inclusion criteria, which have emerged through the primary search (websearching) (1).

Examples of secondary searches may include:

2.1.1 Citation searching: any study reports fulfilling inclusion at full text would be backwards and forwards citation searched. This is a well-established ‘secondary search’ and there is evidence that citation searching can identify study reports missed by bibliographic database searching (7).  

2.1.2 Re-searching trials registers with new search terms such as trial names or numbers identified in the primary search (Stage 2) but unknown at Stage 1 of this model. This is likely to significantly duplicate the primary searches so would be a ‘safety net’ search only used if a need for precision was absolute.

2.1.3 Websearching: here we mean searches of the internet by a search engine as opposed to searches of specific websites.

The websearches would be developed using the study name/possibly the intervention name as the search terms (depending on context), with the purpose being to identify published/unpublished study reports, evaluation studies, unpublished data sets, extended qualitative reports, or sibling studies. These are items which are less likely to be indexed in bibliographic databases and which may have been overlooked by other search methods at Stage 2. We assume that the focus on the study name will generate very low returns from the searches.

Whilst we envision websearching using a search engine generally, we wonder if it might be valuable to focus these websearches in the country where the study/studies were conducted. In a previous study we found that search results varied depending on the geographic location of the searcher (4). Based on this finding, we hypothesise that, by localising a websearch to the country where a study was conducted, a researcher might be more likely to identity unpublished study reports or data of relevance to the study. For instance, if you identified a study conducted in Australia, and you are not located in Australia, it might be desirable to use a Virtual Private Network and select a server in Australia to conduct these searches. We acknowledge that it might be costly in terms of the time needed to search and that – in common with websearching generally – that it may impact the reproducibility of websearches but our previous work suggests results may vary by location.

We suggest that these secondary searches are conducted whilst data from the primary searches are prepared for analysis. We also suggest that the search strategies are reviewed in the same way as the primary searches, ideally using the Peer Review of Electronic Search Strategies (PRESS) checklist (8).

5. Analysis

Developed per protocol.

6. Writing up

The writing up of the secondary searches should be to the same standard as the primary search and follow best practice guidance. It might be desirable to report the secondary searches separately from (that is, in a defined paragraph after) the report of the primary search methods in the final review. This to capture the difference between, and report the conduct of searches developed, a priori, and those searches which have emerged a posteriori.

In so far as the secondary searches are based on studies identified by the primary searches, we do not immediately perceive a bias here, since the aim is to increase sensitivity of the searches and further minimise publication bias. Where a problem could emerge is if the secondary searches identify new studies or data which change the original understanding of the research question or scope of review. The research team would need to consider the implications of such an event.

The results of the search should be reported under ‘identification of studies via other methods’ in the PRISMA flow chart and the search strategies used reporting in full in an annex available to the readers (9-11).

7. Next Steps

We are testing the ideas set out above using the illustrated process model. Our specific interest is secondary websearching.

The outcomes we are looking to evaluate relate to effectiveness and resource-use, namely:

1. Effectiveness

·       Identification of unique items through secondary websearches: this will be measured by identification of any study report or data which is novel to search returns when compared to the original searches;

·       Do study authors reply to e-mails: This is the second comparison that we are making. In each case, we are contacting the study author(s) to request any unpublished data and lists of study reports relating to the study. Author contact, where authors reply, is likely to be more efficient than the process described above. It does, however, require the author to reply, so we are comparing where an author reply offers us the study reports or data we might have got from a websearch. We will still undertake the websearch;

·       Value of any unique items identified through secondary websearches: where possible, we are seeking to measure any effect relating to the identification of any new study reports or data and how this might impact the findings of a synthesis, be that quantitatively or qualitatively. 

2. Resource-use

·       Time: how much time was spent undertaking these secondary websearches; and

·       Cost: how much did secondary searches cost to undertake.

Please cite this blog as: Chris Cooper. Conceptualising the process and approach to websearching in complex systematic reviews: an initial conceptual model and description of user interaction. 2021. http://doi.org/10.5281/zenodo.4940058

Please drop us a line too: thesearcheruk@outlook.com

References

1.         Centre for Reviews and Dissemination. Systematic reviews – CRD’s guidance for undertaking reviews in healthcare [Internet]. In: 3rd ed. York: Centre for Reviews and Dissemination, University of York; 2009: https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf. Accessed 9 May 2021.

2.         Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol. 2018;18(1):85. https://doi.org/10.1186/s12874-018-0545-3

3.         Cooper C. Does websearching provide 'hints' to studies overlooked by bibliographic database searching? 2021. http://doi.org/10.5281/zenodo.4889399

4.         Cooper C, Lorenc T, Schauberger U. What you see depends on where you sit: The effect of geographical location on web-searching for systematic reviews: A case study. Res Synth Methods. 2021. https://doi.org/10.1002/jrsm.1485

5.         Eysenbach G, Tuische J, Diepgen TL. Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews. Med Inform Internet Med. 2001;26(3):203-218.

6.         Mahtani KR, Jefferson T, Heneghan C, Nunan D, Aronson JK. What is a ‘complex systematic review’? Criteria, definition, and examples. BMJ Evid Based Med. 2018;23(4):127-130. http://dx.doi.org/10.1136/bmjebm-2018-110965

7.         Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017;6(1):234. https://doi.org/10.1186/s13643-017-0625-1

8.         McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40-46. https://doi.org/10.1016/j.jclinepi.2016.01.021

9.         Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372.

10.       Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. https://doi.org/10.1136/bmj.n160

11.       Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10(1):39. https://doi.org/10.1186/s13643-020-01542-z

Thanks to: Theo Lorenc, Daniela Gonçalves Bradley, and Ute Schauberger.