Types of studies

All prospective diagnostic studies with blind comparison of index and reference tests i.e. where the index test was read without knowledge of the reference test and vice versa, with sufficient data to extract the following will be considered for inclusion:

• The number of true positives (TP), i.e. patients categorised as diseased by both the reference and index test

• The number of false negatives (FN), i.e. patients categorised as diseased by the reference test, but as non‐diseased by the index test

• The number of true negatives (TN), i.e. patients categorised as non‐diseased by both the reference and index tests

• The number of false positives (FP), i.e. patients categorised as non‐diseased by the reference test, but as diseased by the index test

We will also restrict the studies to those that used the index and reference tests within six months of each other to limit bias due to changes in patients’ stenoses over time. Patients will be from an unbiased sample, e.g. a random sample or a consecutive series. There will be no language restriction imposed, though it is recognised that resources may not allow for all non‐English language studies to be translated or included. Initially, we will limit the review to studies with 20 or more patients, and only include the smaller studies if resources allow.

Participants

The patient group will be people presenting with symptoms consistent with a transient ischaemic attack, minor stroke, amaurosis fugax, or retinal artery occlusion. They may or may not have had a prior ultrasound test. The settings are dedicated transient ischaemic attack and minor stroke services, neurology services, vascular services, geriatric services, and A & E departments. Studies using asymptomatic patients, or a mixture of symptomatic and asymptomatic patients where the proportion of symptomatic patients is less than 90%, will be excluded.

Index tests

The index tests are colour (DUS) (but not power Doppler or B‐flow imaging), CT angiography, MR angiography, and contrast‐enhanced MR angiography in the assessment of severe carotid stenosis.

Colour DUS is non‐invasive, requiring no injections. A probe is held on the skin over the carotid arteries with a gel on the skin to provide acoustic coupling, and images of the carotid and vertebral arteries, as well as readings of the blood velocities at different points in the blood vessels obtained. Colour DUS depends on the skill and experience of the operator and provides limited images for others to review independently. The completeness of the examination may be restricted by the patient physique, the jaw may prevent visualisation of the carotid bifurcation. CT angiography and contrast‐enhanced MR angiography both require intravenous injections routinely given by mechanical pumps timed to coincide with rapid imaging through the carotid arteries as the contrast agent is carried through as a bolus in the blood. Both require post‐processing techniques to provide anatomical images of the artery outlines and subtract the veins which can obscure the arteries. CT contrast agents may be precluded in patients with allergies, heart failure, renal failure or diabetes (all common in older people). MR angiography uses special properties of flowing blood to “magnetically tag the blood” and allows visualisation without intravenous injections. It also requires post‐processing. MR angiography and contrast‐enhanced MR angiography produce images which many others can view independently so do not have the same operator dependence but are best performed in units with regular and sufficient patient throughput to develop proficiency.

Target conditions

Severe carotid stenosis is defined as 70%‐99% NASCET (80%‐99% ECST) and moderate stenosis as 50%‐69% NASCET (70%‐79% ECST) method (ECST 1998; NASCET 1991). Studies without data consistent with this definition, or which could not be converted to it, will be excluded. The data from studies using the ECST or CCA method to define stenosis will be transformed to 70%‐99% or 50%‐69% by the NASCET method using ECST or CCA = 0.6NASCET + 40% (Rothwell 1994).

Reference standards

Intra‐arterial angiography (including digital subtraction and cut film but not rotational intra‐arterial angiography). This investigation is no longer routinely performed in many centres because of the morbidity associated with arterial puncture. Also the need for inpatient stay amongst older patients creates delays. Intra‐arterial angiography was the test used to measure carotid stenosis in all the carotid endarterectomy trials. It is therefore the technique that links stenosis to stroke risk and is thus the reference standard against which all other imaging techniques are judged. It produces high definition anatomical outlines of the arteries and the disease.

Electronic searches

Electronic searches will be made in MEDLINE and EMBASE from 1990 to end 2005 using the search strategies given in Appendix 1. The earlier review found no relevant and methodologically robust articles prior to 1990 (Wardlaw 2006). The search strategies used in MEDLINE and EMBASE incorporated a diagnostic studies filter to reduce the retrieval of non‐relevant studies, as otherwise the number of studies was unworkable. However, great care was taken to ensure that relevant studies were not lost by this process. The search strategies were validated by comparison with handsearching, and experts in the field were asked to give details of any missing references.

Searching other resources

The bibliographies of review articles will be checked for extra references.

The search strategy will be revalidated by handsearching of the following key journals: Radiology, Neuroradiology, American Journal of Roentgenology, Stroke, and European Journal of Vascular and Endovascular Surgery from 1990 to 2005 inclusive. The search strategy was used in an earlier systematic review (Wardlaw 2006), where only one extra reference was found by handsearching alone, and this reference was not included in the final analysis.

Selection of studies

Two reviewer authors will independently select abstracts for inclusion in the review using the above criteria. Abstracts will be sorted into “definitely exclude” and “possibly include” categories, recognising that sometimes it will not be possible to judge from the abstract whether a reference will fulfil the criteria or not. All abstracts selected by at least one review author will be in the “possible include” category.

Data extraction and management

For each study, the numbers of TP, FN, TN, and FP will be extracted. A positive test result will be defined as 70%‐99% stenosis by the NASCET method. Where possible, TP, FN, TN, and FP will also be extracted using 50%‐69% stenosis by the NASCET method as the definition of a positive test result. These data will be extracted by one review author and checked by a second. Other data to be recorded are:

• all items from the QUADAS checklist (spectrum of patients, reference test, time period between reference and index tests, if the results of the index test influenced whether patients underwent the reference test, whether patients received different forms of the reference test, whether the results of the index test formed part of the final diagnosis, whether the index test was read blind to the result of the reference test and vice versa, what clinical data were available to the readers of the tests, how were uninterpretable or intermediate tests reported, how were dropouts reported).

• imaging modality

• age range of patients

• symptoms of patients

• male to female ratio of the patients

• any prior selection by US was used to decide if patients were to enter the study or not, and if so, the degree of stenosis required for inclusion

• whether patients contributed data from both the symptomatic and asymptomatic sides

• the proportion of and total number of patients accounted for in the analysis, and any reports of adverse events due to the tests.

• the method of defining stenosis: NASCET, ECST, or CCA

• study setting

We will record data in Microsoft Excel initially, prior to importing into other packages such as SAS.

Assessment of methodological quality

Two review authors will independently assess the full‐text of each study in the “possible include” category. Discrepancies will be referred to a third author. As abstracts may not give the information necessary to decide whether a study fulfils the inclusion criteria or not, studies will also be excluded at this stage if the full‐text version shows they do not fulfil the inclusion criteria for considering studies for review as outline above. The review authors will use the QUADAS checklist (Whiting 2006) to assess the susceptibility to bias of the studies (Appendix 2), with additional questions on imaging and patient population.

Statistical analysis and data synthesis

Given the adverse effects and cost of intra‐arterial angiography, we need to know whether the less invasive tests can replace the reference test, whether singly or in combination. Reflecting current clinical practice, the combinations of index tests will be DUS followed by another less invasive test (DUS, CT angiography, MR angiography, and contrast enhanced MR angiography).

Providing statistical modelling assumptions are met, the data will be combined using the empirical Bayes hierarchical summary ROC curve method (Macaskill 2004) as this method can be used to produced both ROC curves and point estimates of sensitivity and specificity, and also to investigate the effect of sources of heterogeneity on diagnostic accuracy.

If possible, we will analyse the data from studies where more than one non‐invasive modality has been used on the same patient group, i.e. direct comparison of non‐invasive modalities. In a separate analysis, we will compare diagnostic accuracy using data from studies that used a single non‐invasive test, i.e. make an indirect comparison of diagnostic accuracy.

Investigations of heterogeneity

The investigations are primarily concerned with exploring heterogeneity in sensitivity and specificity as these are the quantities we hope to estimate. Therefore, we will use:

• Forest plots to look for evidence of heterogeneity within sensitivity and within specificity

• ROC plots to look for evidence of a threshold effect and heterogeneity due to differences in accuracy.

• Plots of logit‐transformed sensitivity and specificity as described by Moses et al (Moses 1993) to assess variation in diagnostic odds with change in diagnostic threshold.

Sensitivity analyses

In many centres, ultrasound is used to identify patients with stenosis, and only patients with potentially significant degrees of stenosis are referred for further testing. We will therefore also investigate how screening with prior ultrasound affects diagnostic accuracy as the non‐invasive tests may perform differently in patients who have been shown to have a significant stenosis. We will also investigate the effect of missing data, i.e. where patients have been recruited into the study, but are not included in the final analysis as these may be patients for whom the tests did not work. The analyses will be done by adding covariates to the nonlinear mixed model used in the hierarchical summary ROC curve method.