Types of studies

We will include test accuracy studies that allow comparison of results of the index test with that of a reference standard, including:

• prospective and retrospective studies;

• studies where all participants receive a single index test and a reference standard;

• studies where all participants receive more than one index test(s) (concurrently) and a reference standard;

• studies where participants are allocated (by any method) to receive different index tests or combinations of index tests and all receive a reference standard (between‐person comparative studies);

• studies that recruit a series of participants unselected by true disease status; and

• diagnostic case‐control studies that separately recruit diseased and nondiseased groups (Rutjes 2005).

We will exclude follow‐up or surveillance studies using repeated imaging tests to detect disease recurrence, as defining the most appropriate follow‐up schedule for melanoma patients is not the primary objective of these reviews. We will exclude studies if it is not possible to derive the number of true positives, false positives, false negatives and true negatives from data provided in the paper, or small studies with less than five disease‐positive or less than five disease‐negative participants. Although the size threshold of five is arbitrary, such small studies are likely to give unreliable estimates of sensitivity or specificity, and may be biased in a similar way to small randomised controlled trials (RCTs) of treatment effects.

We will include studies reporting either lesion‐based or patient‐based analyses; however, we will only include those reporting data on a per patient basis in the primary analysis. This is particularly pertinent for the reviews of imaging tests where multiple metastatic sites may be detected in an individual patient.

Participants

We will include studies in adults with cutaneous melanoma at any primary site who are undergoing staging, either following primary presentation of disease or following recurrence of disease. We will include studies that include mixed populations of patients or where the clinical pathway cannot be determined and examine any effect on test accuracy in subgroup analysis. We will exclude studies in which test results for participants with melanoma cannot be differentiated from those of participants with other diagnoses.

For studies of sentinel lymph node biopsy (SLNB), outcomes must be presented for both sentinel lymph node positive and sentinel lymph node negative participants. For studies of imaging tests, we will include studies focusing on either sentinel lymph node positive or sentinel lymph node negative participants.

Index tests

We will undertake individual reviews for SLNB and for the following imaging tests, either alone or in combination.

• Ultrasound (with or without subsequent fine needle aspiration cytology (FNAC) or core biopsy)

• Computed tomography (CT) (non‐contrast or contrast‐enhanced)

• Positron emission tomography (PET) or PET‐CT (¹⁸FDG only)

• Magnetic resonance imaging (MRI) (non‐contrast or contrast‐enhanced)

SLNB studies may assess the effectiveness of methods of detection of sentinel lymph nodes, for example using different tracers or dyes or alternative imaging approaches. These will often compare approaches in terms of number of diseased nodes identified and we will exclude these unless an eligible reference standard, as described below, has been used.

We will produce a comparative overview review to compare the accuracy of tests, either alone or in combination.

Target conditions

The target condition for the SLNB review will necessarily be defined differently according to the result of the index test as:

• for sentinel lymph node positive participants, the presence of micrometastasis in the nodal basin investigated by the SLNB procedure; or

• for sentinel lymph node negative participants, the emergence of clinically detectable nodal disease or macrometastases in the nodal basin investigated by the SLNB procedure in the absence of evidence of distant metastases (i.e. a false negative SLNB result will be considered to have occurred if disease is proven to emerge only in the regional lymph nodes with no spread to distant sites).

In the event of inadequate data, we will drop the requirement to demonstrate the absence of distant metastases in sentinel lymph node negative participants with regional nodal recurrence, and we will consider the emergence of any nodal disease in the nodal basin investigated by the SLNB procedure a sufficient definition of a false negative result.

The target conditions for the imaging test reviews are the detection of:

• any metastases,

• any nodal metastases, or

• any distant metastases.

The use of the same tests for the staging of cutaneous squamous cell carcinoma (cSCC) is the subject of a separate protocol (Dinnes 2017).

Reference standards

Acceptable reference standards include:

• histology of lymph node or distant specimens, with samples obtained by core biopsy, SLNB or lymph node dissection, for index test positive participants;

• cytology of lymph node specimens, with samples obtained by core biopsy, or fine needle aspiration, for index test positive participants;

• clinical or radiological follow‐up to identify nodal or distant recurrence of at least six months, for index test negative participants; or

• any combination of the above.

Studies using cross‐sectional imaging‐based reference standards, i.e. a direct comparison of the index test compared to an alternative reference standard imaging test, will not be eligible.

Electronic searches

We have conducted a single large literature search for the programme grant, covering all conditions and tests. This allowed for the screening of search results for potentially relevant papers for all reviews at the same time. We formulated a MEDLINE scoping search combining disease‐related terms with terms related to the test names, using both text words and subject headings. As most records were related to the searches for tests for the staging of disease, we applied a filter using terms related to cancer staging and to accuracy indices to the staging test search, to try to eliminate irrelevant studies, e.g. those using imaging tests to assess treatment effectiveness.

We screened a sample of 300 records that would be missed by applying this filter and adjusted the filter to make sure that we would not miss any potentially relevant studies. The final search filter (Appendix 2) reduces the overall numbers retrieved from MEDLINE by around 6000. We cross‐checked the final search result against the list of studies included in five systematic reviews; our search identified all but one of the studies, and this study is not indexed on MEDLINE. The Information Specialist, Susan Bayliss, has devised the search strategy, with input from the Information Specialist from Cochrane Skin, Elizabeth Doney. We used no additional limits.

We undertook further scoping searches to identify any relevant systematic reviews or health technology assessments. In addition to general bibliographic databases, we also accessed specialist databases with a focus on reviews of diagnostic test accuracy, such as ARIF.

We have now searched the following bibliographic databases, retrieving a total of 33,994 unique records:

Searching other resources

Due to time restrictions and the volume of evidence retrieved from the electronic searches, we will not conduct any handsearching of conference proceedings. By searching CENTRAL, we will retrieve relevant records identified by regular handsearching by Cochrane Skin. The handsearched conferences and journals are listed here: www.skin.cochrane.org/resources‐handsearchers.

We will include information about potentially relevant ongoing studies in the 'Characteristics of ongoing studies' tables. We will screen any relevant systematic reviews identified by the searches for their included primary studies, and we will include any that our searches have missed in the review. We will check the reference lists of all included papers, and subject experts within the author team will review the final list of included studies. We may use citation‐searching for key references when we consider it appropriate.

Selection of studies

Due to the volume of records retrieved, at least one review author (JDi or NC) has undertaken screening of the titles and abstracts, with any queries on the part of either reviewer discussed and resolved by consensus. A pilot exercise independently screening 539 references from MEDLINE showed a good level of agreement (89% with a kappa of 0.77). To date, 822 records have been selected for full text review for the staging reviews. At least two review authors, including methodologists (JDi or NC) and clinical reviewers, using a study eligibility screening proforma based on prespecified inclusion criteria, will independently undertake subsequent assessment of potentially relevant full text articles for the staging reviews (Appendix 3). Where differences in opinion exist, a third party drawing on the clinical and methodological expertise in the team, as appropriate to the content of the query (JDe, CD, HW, and RM), will help with resolution. We will compile a list of otherwise eligible studies for which insufficient data were presented to allow for the construction of a 2x2 contingency table, and we will contact study authors, asking them to provide the relevant data. We will describe the study selection process in an adapted PRISMA flowchart (Liberati 2009). At the full text inclusion stage, we will tag studies according to their target condition (melanoma or cSCC) and index test.

Data extraction and management

We will carry out data extraction using a predesigned and piloted data extraction form using Excel to ensure that we collect relevant data. At least two review authors will independently extract data concerning details of the study design, participants, index test(s) or test combinations and criteria for index test positivity, reference standards, and data required to populate a 2x2 diagnostic contingency table for each index test. We will record where data are available at several index test thresholds. A third party drawing on clinical and methodological expertise in the team as appropriate to the content of the query will resolve discrepancies.

We will attempt to contact authors of included studies where information that is considered key to one or more of the assessments of the quality of an included study, investigation of heterogeneity, or completion of a 2x2 diagnostic contingency table is missing. We will follow up studies published only as conference abstracts to identify whether a subsequent full paper has been published. Where possible, we will contact the authors of conference abstracts published from 2015 to 2016 and ask whether full data are available. If we can identify no full paper, we will mark conference abstracts as 'pending' and revisit them. Experience of contacting authors for information about missing data in DTA reviews is limited. Therefore, where we seek missing data, we will document the outcome of contact with the authors.

Assessment of methodological quality

We will assess applicability and risk of bias of included studies using the QUADAS‐2 checklist (Whiting 2011), which has been tailored to the review topic (see Appendix 4).

Statistical analysis and data synthesis

For the SLNB reviews, our primary analysis will focus on the detection of metastases in the investigated nodal basin. For the imaging test reviews, we will conduct separate analyses firstly according to whether study participants are recruited on primary presentation of melanoma or with a disease recurrence, and secondly according to our primary and secondary objectives, i.e. detection of any metastasis (which must include both nodal and distant recurrence) and detection of nodal metastasis alone or detection of any distant metastasis, as defined under 'Target condition'). SLNB is not employed for staging of recurrence in skin cancer.

Studies may report test accuracy per‐lesion or per‐patient. Our unit of analysis for the primary analyses will be the patient as study participants may have multiple metastatic sites at any one time, such that a per lesion analysis may over‐estimate test accuracy. We will include data from studies that reported per‐lesion level data in secondary analyses, such that per lesion and per patient data from different studies would be combined together, using per patient data in preference where both are reported within a study. The estimation of the accuracy metrics to be used in our reviews are detailed in Appendix 5.

For the SLNB review, both index test and reference standard positivity are defined histologically. In the absence of an additional suitable reference standard for SLNB test positivity, it will not be possible to estimate false positive cases and specificity will always be 100%. We will therefore perform meta‐analysis of only sensitivities by using a univariate random effects logistic regression model. We will also estimate the pooled negative predictive value in a secondary analysis (the positive predictive value not being possible to calculate due to false positives not being estimable). The definitions for each cell of the 2x2 contingency tables for the SLNB review are as follows.

• TP = sentinel lymph node positive (i.e. all patients with a positive sentinel lymph node regardless of any subsequent recurrence).

• FP = not possible to estimate.

• FN = sentinel lymph node negative patients who experience clinical emergence of disease in the same nodal basin, in the absence of disseminated disease.

• TN = sentinel lymph node negative patients who do not experience clinical emergence of disease in the same nodal basin.

For the imaging test reviews, we will estimate sensitivity and specificity in the usual way. We will initially explore the data by plotting estimates of sensitivity and specificity on coupled forest plots and in receiver operating characteristic (ROC) space for each index test under consideration. We will use hierarchical models to perform meta‐analyses (Macaskill 2010). Where commonly used thresholds are reported, we will produce summary operating points (summary sensitivities and specificities) with 95% confidence and prediction regions using the method in Reitsma 2005. Where different thresholds are used, we will fit a summary curve using the hierarchical summary ROC model (Rutter 2001). When few studies are available for meta‐analysis, we will simplify hierarchical models as appropriate, depending on whether the focus of inference is a summary point or summary curve (Takwoingi 2015). It is anticipated that results from multiple thresholds within a single study may be reported in many instances. Where multiple thresholds are to be selected for the review, data from the same participants may be used more than once in each analysis. For the analysis of summary curves, however, we will select standard or most commonly used thresholds from each study; failing that, we will select one threshold at random.

For the overview, we will perform both direct and indirect test comparisons, the latter being required because it is anticipated that comparative studies may be scarce (Takwoingi 2013). To formally compare index tests, we will add a covariate for test type to a hierarchical model. We will use likelihood ratio tests to assess the statistical significance of differences in test accuracy (sensitivity and specificity) for analyses of summary points and shape and accuracy for analyses of summary curves, by comparing models without the covariate terms with models containing the covariate terms.

We will conduct analyses using Review Manager 5 (Review Manager 2014), the NLMIXED procedure in the statistical software SAS version 9.4 (SAS 2016) and the meqrlogit command in the statistical software STATA 15 (STATA 2017).

Investigations of heterogeneity

We will initially examine heterogeneity between studies by visually inspecting the forest plots of sensitivity and specificity and summary ROC plots. Where a sufficient number of studies has assessed the same index test and the characteristics of interest (see Secondary objectives) were adequately reported to enable analyses, we will perform meta‐regression by adding the potential source of heterogeneity as a covariate to a hierarchical model. We will apply a minimum requirement of at least five studies in each subgroup; we will only report heterogeneity analyses with less than five studies per group when we can be convinced that models have achieved adequate convergence and that the distribution of studies across groups is adequate to provide valid estimates. Where factors to be investigated (e.g. AJCC stage of disease) could vary between participants within a study, we will rely on the inclusion criteria set out by study authors (such as restriction to stage II or to stage III or IV melanoma), or use the results of any subgroup analyses within a study to examine the effect of that covariate. We will assess each of the factors listed under the secondary objectives where possible.

Sensitivity analyses

If a sufficient number of studies assess the same index test, we will perform sensitivity analyses restricting analyses according to:

• those with direct test comparisons (where the period of application between the index tests was within one month);

• where concerns around applicability for participant selection are low;

• where there was low risk of bias for the index test; and

• where there was low risk of bias for the reference standard.

As for the Investigations of heterogeneity above, we will require a minimum of at least five studies before we conduct sensitivity analyses.

Assessment of reporting bias

Because of uncertainty about the determinants of publication bias for diagnostic accuracy studies and the inadequacy of tests for detecting funnel plot asymmetry (Deeks 2005), we will not perform tests to detect publication bias.