Types of studies

We will include randomised controlled studies, cross‐sectional diagnostic studies and cohort studies. These may be either prospective or retrospective. Patients should be included consecutively. We will exclude case‐control studies. We will include studies with a direct comparison of two or three imaging modalities in the same patient population. We will also include studies in which only one of the three index tests and studies in which multiple index tests are evaluated, but not in the same patient population. We realise that ideally all three index tests are used in the same patient population. However, a first exploration of the literature has led us to believe there are insufficient studies to limit inclusion to studies with such a design.

Furthermore, studies must report sufficient data in order to extract the following:

• the number of true positives: patients categorised as diseased by both the index and reference test;

• the number of false negatives: patients categorised as diseased by the reference test, but as non‐diseased by the index test;

• the number of true negatives: patients categorised as non‐diseased by both the index and reference test;

• the number of false positives: patients categorised as non‐diseased by the reference test, but as diseased by the index test.

We will not use any restrictions based on a minimal quality standard, minimal sample size or number of patients with the target condition (i.e. lymph node metastases).

Participants

We will include studies that include patients with histopathologically proven head and neck squamous cell carcinoma. We will exclude studies that predominantly include patients with nasopharyngeal, sinonasal, thyroid, salivary gland, skin and unknown primary cancer. These tumours are characterised by a different pathophysiology. We will not exclude studies with a mixed study population, including up to 10% of patients with any of these cancer types. We will only include studies including patients without previous neck dissection, to ensure that the reference standard (definitive histology) can be obtained. We will not exclude studies including patients that have undergone previous (chemo)radiation. Study inclusion will not be limited by age, gender or T‐classification. All included patients should have received an MRI, CT and/or PET‐CT for lymph node staging purposes. Study inclusion will not be limited by clinical N‐classification; we will include studies including both cN0 and cN+ patients.

Index tests

The index tests under review are MRI, CT and PET‐CT. These tests were chosen because they are currently used to detect lymph node metastases in head and neck cancer patients. Different criteria can be used to diagnose lymph node metastases, including morphological criteria and size‐related criteria for MRI and CT, and quantitative metrics for PET. We will not limit study inclusion based on the diagnostic criteria that were used. It needs to be noted though that differences in diagnostic criteria between MRI, CT and PET‐CT may introduce variation in the percentage of cN0 and cN+ patients.

We will limit study inclusion based on the quality of the MRI, CT and/or PET‐CT. Regarding CT, we will exclude studies using a CT slice thickness of more than 3 mm. Intravenous contrast should have been used in the studies in order to be included. Regarding MRI, we will exclude studies using a MRI slice thickness of more than 4 mm and/or an interslice gap of more than 0.4 mm. At least T1‐ and T2‐weighted, turbo spin echo (TSE) or short tau inversion recovery (STIR) sequences should have been used, as well as a post‐Gadolinium contrast series. Either a 1.5 or 3.0 Tesla magnet should have been used. Regarding PET‐CT, at least 18F‐FDG should have been used. Even though we will limit exclusion based on the quality of the MRI, CT and/or PET‐CT, small variations in these imaging techniques may introduce bias.

Target conditions

Lymph node metastases in patients with histologically proven head and neck squamous cell carcinoma.

Reference standards

The primary reference standard is definitive histopathology following surgery, preferably in combination with one‐year clinical follow‐up during which patients will not receive any further treatment. Since the former is invasive, it is not usually performed on patients with negative index test results, which can lead to verification bias. Long‐term clinical follow‐up of at least one year is the primary differential reference standard in patients that will not undergo neck dissection. If no obvious lymph node metastases are present at one‐year clinical follow‐up, these patients are considered negative. However, studies often only include patients with an indication for surgery and exclude patients that will not undergo treatment or radio(chemo)therapy. As it seems likely that in these study populations the reported prevalence of lymph node metastases is higher than it is in the general population of patients with head and neck squamous cell carcinoma, this could lead to overestimation of the positive predictive value and underestimation of the negative predictive value. We will not pool the results for these two reference standards (histopathology and clinical follow‐up). We will perform a subgroup analysis instead.

It is important to note that different histopathological techniques can be used to assess lymph nodes. Routine histopathological examination can miss minimal lymph node metastases. Step‐serial sectioning and immunohistochemistry can increase the yield of diseased lymph nodes by up to 15% (Rinaldo 2004). The use of routine histopathological examination likely leads to an overestimation of the number of true negatives. Furthermore, the reported pre‐test probability or prevalence of disease is influenced by the amount of lymph nodes harvested. Increased lymph node harvesting lowers the risk of missing micrometastases (van den Brekel 1996). The lymph node ratio (ratio of the number of positive lymph nodes to total number of lymph nodes in the surgical specimen) is not only a strong prognostic factor in head and neck cancer, it is also influenced by the amount of harvested lymph nodes (Marres 2014). We will be using the total number of harvested lymph nodes to assess the quality of the neck dissections performed.

Electronic searches

Published, unpublished and ongoing studies will be identified by searching the following databases from their inception:

• Ovid MEDLINE (1946 to date);

• Ovid MEDLINE (In‐Process & Other Non‐Indexed Citations);

• PubMed (as a top up to searches in Ovid MEDLINE);

• Ovid EMBASE (1974 to date);

• EBSCO CINAHL (1982 to date);

• Web of Knowledge, Web of Science (1945 to date);

• ClinicalTrials.gov, www.clinicaltrials.gov (search via the Cochrane Register of Studies to date);

• World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP) (search to date);

• ISRCTN, www.isrctn.com (search to date);

• Google Scholar (search to date);

• Google (search to date).

The subject strategies for databases will be modelled on the search strategy designed for MEDLINE (Appendix 1).

Searching other resources

We will scan the reference lists of identified publications for additional studies and contact study authors if necessary. In addition, the Information Specialist will search PubMed, TRIPdatabase and Google to retrieve existing systematic reviews relevant to this systematic review, so that we can scan their reference lists for additional studies. We will search for conference abstracts using the Cochrane ENT Trials Register and EMBASE.

Selection of studies

We will import all references identified by the electronic searches into RefWorks and remove duplicates. Two review authors (IW and IS) will independently screen the retrieved records. Discrepancies will be resolved through a consensus discussion between the two review authors. If disagreements persist, a third review author will resolve the discrepancies (LH, JBR, FAP, OSH or RdB). The third review author will be chosen based on his or her expertise. LH and JBR will resolve discrepancies regarding epidemiology/methodology, FAP and OSH regarding radiology, and RdB regarding clinical questions. We will review titles and abstracts and exclude those articles that are obviously ineligible. We will exclude systematic reviews, opinion papers, editorials, conference abstracts and poster presentations at this stage. We will exclude studies that clearly do not evaluate the diagnostic accuracy of MRI, CT or PET‐CT, but other imaging modalities such as diffusion‐weighted MRI. We will exclude studies reviewing the use of MRI, CT and/or PET‐CT following surgical treatment, as well as studies evaluating treatment response.

We will proceed by retrieving and reviewing the full‐text copies. We will only include those studies evaluating the diagnostic accuracy of MRI, CT and/or PET‐CT in comparison with definitive histology or clinical follow‐up in patients with head and neck squamous cell carcinoma for detecting lymph node metastases. We will exclude studies evaluating the diagnostic accuracy of these imaging modalities for establishing tumour characteristics other than lymph node metastases. We will only include studies if they meet the MRI, CT and/or PET‐CT criteria listed under Index tests. Studies will not be limited by language, location or setting at any stage of the selection procedure.

Data extraction and management

Two review authors (IW and IS) will independently extract key data regarding the study population, the index test, the reference test and the diagnostic test results from the included studies using data extraction sheets. A complete list of characteristics and results that we will be extracting is enclosed in Appendix 2. Discrepancies will be resolved through a consensus discussion between the two review authors or a third review author will resolve the discrepancies (LH, JBR, FAP, OSH or RdB).

The two review authors (IW and IS) are capable of assessing and evaluating articles written in English, Dutch, German and French. Data from articles written in other languages will be extracted by radiologists or ENT surgeons with knowledge of the language, correspondence with the study authors, or by a translator working in conjunction with two review authors (IW and IS).

Assessment of methodological quality

Two review authors (IW and IS) will independently assess the methodological quality of the included studies using predefined criteria. Discrepancies will be resolved through a consensus discussion between the two review authors or a third review author will resolve discrepancies (LH, JBR, FAP, OSH or RdB). We will use the QUADAS‐2 tool to assess the methodological quality (Whiting 2011). A full list of the criteria and their operational definitions is enclosed in Appendix 3. We will include a filled out QUADAS‐2 form for each of the included studies, as well as a methodological quality summary figure and a methodological quality graph. The methodological quality summary figure presents for each included study the 'yes', 'no' and 'unclear' judgements for each quality assessment item in graphical form. The methodological quality graph presents for each quality assessment item the percentage of the included studies that rate the item 'yes', 'no' and 'unclear' in a stacked bar chart.

We will include studies regardless of their quality of evidence. Instead, we will explore the impact of (poor) methodological quality using meta‐regression analyses. The methods are described in the section Investigations of heterogeneity.

Statistical analysis and data synthesis

Disease status and index test results are both binary, leading to the extraction of true‐positive, false‐positive, true‐negative and false‐negative test results for MRI, CT and/or PET‐CT for each included study. We will construct forest plots showing estimates of sensitivity and specificity and their corresponding 95% confidence intervals for each study. We will sort forest plots by imaging technique (MRI, CT and PET‐CT). We will calculate confidence intervals around these proportions according to the method of Wilson (Newcombe 1998). We will also plot sensitivities and specificities in receiver operating characteristic (ROC) space, using different symbols for MRI, CT and PET‐CT estimates. For all direct comparisons, we will link MRI, CT and/or PET‐CT points in ROC space derived from the same study. We will use random‐effects bivariate regression models to meta‐analyse the logit transformed sensitivity and specificity of MRI, CT and PET‐CT to obtain pooled estimates and 95% confidence intervals of these parameters (Macaskill 2010). We will use the exact binomial distribution to model the within‐study variance. Therefore, we will use non‐linear mixed models to estimate the parameters of interest. We will compare the accuracy between MRI, CT and PET‐CT by adding a covariate for the type of index test to the model to investigate whether sensitivity, specificity, or both, are different between the three imaging techniques. If sufficient studies providing direct evidence are available (n = 5 or higher), we will perform a sensitivity analysis focusing on these studies alone, taking the paired nature into account (Trikalinos 2013). In case of paired diagnostic studies (study participants receive more than one index test under investigation) there is additional information available if such studies report the fully cross‐classified data. It means a two‐by‐two table of one index test against the other index test among the patients with the target condition and another two‐by‐two table among patients without target condition. Unfortunately, paired diagnostic accuracy studies hardly report their data in such format, and this is also our experience within this field. Therefore, we will perform a sensitivity analysis of the paired studies in which we will focus on the difference in logit sensitivity between the index tests in each study and then summarise these differences across studies. We will do the same for differences in logit specificity. When the number of studies is either small or the correlation is low, we will perform separate meta‐analysis, otherwise we will extend the model by incorporating the correlation between logit difference sensitivity and logit difference specificity over studies. We will compare the results of the paired studies with the results of the analysis including all studies.

We will present the results of the bivariate regression models in ROC space, showing pooled estimates of sensitivity and specificity together with 95% confidence intervals around these pooled estimates (Reitsma 2005).

We will perform analyses using SAS and R software.

Investigations of heterogeneity

We will add covariates to the bivariate regression models to assess whether heterogeneity in sensitivity and/or specificity can be related to the following factors:

• methodological quality rating for the four QUADAS‐2 domains (low risk of bias versus high risk of bias versus unclear risk of bias);

• previous (chemo)radiotherapeutic treatment of the neck region (yes versus no);

• subsite distribution (oral versus oropharyngeal versus hypopharyngeal versus laryngeal versus other);

• N‐classification distribution (cN0 versus cN+);

• magnetic field strength used (1.5 Tesla versus 3.0 Tesla);

• slice thickness imaging;

• slice thickness pathology;

• reference test (histopathology with clinical follow‐up versus histopathology versus clinical follow‐up);

• use of immunohistochemistry;

• use of step serial sectioning;

• number of harvested lymph nodes.

We will include one covariate in the analyses at a time.

Sensitivity analyses

We will perform two sensitivity analyses:

• a separate analysis of studies with an overall low risk of bias, characterised by a low risk of bias in three out of four QUADAS‐2 domains (patient selection, index test, reference standard, and flow and timing) compared to the analysis of all studies combined regardless of their methodological quality;

• a separate analysis of studies with direct comparisons compared to the analysis of studies with both direct and indirect comparisons.

Assessment of reporting bias

There is a lack of sensitive tests for use in diagnostic test accuracy reviews and the determinants of publication bias are unknown. Therefore we elect not to investigate reporting bias.