Item

Judgment

Description

1. Random sequence generation (selection bias)

Low risk

The investigators describe a random component in the sequence generation process such as: random number table; computer random number generator; coin tossing; shuffling cards or envelopes; throwing dice; drawing of lots; minimisation.

High risk

The investigators describe a non‐random component in the sequence generation process such as: odd or even date of birth; date (or day) of admission; hospital or clinic record number; alternation; judgment of the clinician; results of a laboratory test or a series of tests; availability of the intervention.

Unclear risk

Insufficient information about the sequence generation process to permit judgment of low or high risk

2. Allocation concealment (selection bias)

Low risk

Investigators enrolling participants could not foresee assignment because one of the following, or an equivalent method, was used to conceal allocation: central allocation (including telephone, web‐based, and pharmacy‐controlled, randomisation); sequentially‐numbered drug containers of identical appearance; sequentially‐numbered, opaque, sealed envelopes.

High risk

Investigators enrolling participants could possibly foresee assignments because one of the following method was used: open random allocation schedule (e.g. a list of random numbers); assignment envelopes without appropriate safeguards (e.g. if envelopes were unsealed or non­opaque or not sequentially numbered); alternation or rotation; date of birth; case record number; any other explicitly unconcealed procedure.

Unclear risk

Insufficient information to permit judgment of low or high risk. This is usually the case if the method of concealment is not described or not described in sufficient detail to allow a definite judgment.

3. Blinding of participants and providers (performance bias)

Objective outcomes

Low risk

No blinding or incomplete blinding, but the review authors judge that the outcome is not likely to be influenced by lack of blinding

Blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken

High risk

No blinding or incomplete blinding, and the outcome is likely to be influenced by lack of blinding

Blinding of key study participants and personnel attempted, but likely that the blinding could have been broken, and the outcome is likely to be influenced by lack of blinding

Unclear risk

Insufficient information to permit judgment of low or high risk

4. Blinding of participants and providers (performance bias)

Subjective outcomes

Low risk

Blinding of participants and providers ensured and unlikely that the blinding could have been broken

High risk

No blinding or incomplete blinding, and the outcome is likely to be influenced by lack of blinding

Blinding of key study participants and personnel attempted, but likely that the blinding could have been broken, and the outcome is likely to be influenced by lack of blinding

Unclear risk

Insufficient information to permit judgment of low or high risk

5. Blinding of outcome assessor (detection bias)

Objective outcomes

Low risk

No blinding of outcome assessment, but the review authors judge that the outcome measurement is not likely to be influenced by lack of blinding

Blinding of outcome assessment ensured, and unlikely that the blinding could have been broken

High risk

No blinding of outcome assessment, and the outcome measurement is likely to be influenced by lack of blinding

Blinding of outcome assessment, but likely that the blinding could have been broken, and the outcome measurement is likely to be influenced by lack of blinding

Unclear risk

Insufficient information to permit judgment of low or high risk

6.Blinding of outcome assessor (detection bias)

Subjective outcomes

Low risk

Blinding of outcome assessment ensured, and unlikely that the blinding could have been broken

High risk

No blinding of outcome assessment, and the outcome measurement is likely to be influenced by lack of blinding

Blinding of outcome assessment, but likely that the blinding could have been broken, and the outcome measurement is likely to be influenced by lack of blinding

Unclear risk

Insufficient information to permit judgment of low or high risk

7. Incomplete outcome data (attrition bias)

For all outcomes except retention in treatment or drop out

Low risk

No missing outcome data

Reasons for missing outcome data unlikely to be related to true outcome (for survival data, censoring unlikely to be introducing bias)

Missing outcome data balanced in numbers across intervention groups, with similar reasons for missing data across groups

For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk not enough to have a clinically‐relevant impact on the intervention effect estimate

For continuous outcome data, plausible effect size (difference in means or standardised difference in means) among missing outcomes not enough to have a clinically‐relevant impact on observed effect size

Missing data have been imputed using appropriate methods

All randomised patients are reported/analysed in the group they were allocated to by randomisation irrespective of non‐compliance and co‐interventions (intention to treat)

High risk

Reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across intervention groups

For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk enough to induce clinically relevant bias in intervention effect estimate

For continuous outcome data, plausible effect size (difference in means or standardised difference in means) among missing outcomes enough to induce clinically relevant bias in observed effect size

'As‐treated' analysis done with substantial departure of the intervention received from that assigned at randomisation

Unclear risk

Insufficient information to permit judgment of low or high risk (e.g. number randomised not stated, no reasons for missing data provided; number of dropout not reported for each group)

8 Selective reporting (reporting bias)

Low risk

The study protocol is available and all of the study's pre‐specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre‐specified way

The study protocol is not available but it is clear that the published reports include all expected outcomes, including those that were pre‐specified (convincing text of this nature may be uncommon)

High risk

Not all of the study's pre‐specified primary outcomes have been reported

One or more primary outcomes is reported using measurements, analysis methods or subsets of the data (e.g. subscales) that were not pre‐specified

One or more reported primary outcomes were not pre‐specified (unless clear justification for their reporting is provided, such as an unexpected adverse effect);

One or more outcomes of interest in the review are reported incompletely so that they cannot be entered in a meta‐analysis

The study report fails to include results for a key outcome that would be expected to have been reported for such a study

Unclear risk

Insufficient information to permit judgment of low or high risk

Domain

Judgment

Description

Bias due to confounding

Low risk (the study is comparable to a well‐performed randomised trial with regard to this domain)

No confounding expected

Moderate risk (the study is

sound for a non‐randomised study with regard to this domain but cannot be considered comparable to a well performed randomised trial)

Confounding expected, all known critically important confounding domains appropriately measured and adjusted for;

and

Reliability and validity of measurement of a critically important domains were sufficient that we do not expect serious residual confounding.

Serious risk (the study has some important problems)

At least one known critically important domain not appropriately measured, or not adjusted for;

or

Reliability or validity of measurement of a critically important domain was low enough that we expect serious residual confounding.

Critical risk (the study is too problematic to provide any useful evidence on the effects of intervention)

Confounding inherently not controllable, or use of negative controls strongly suggests unmeasured confounding

No information on which to base a judgment about risk of bias for this domain

No information on whether confounding might be present

Bias in selection of participants into the study

Low risk

All participants who would have been eligible for the target trial were included in the study and start of follow‐up and start of intervention coincide for all participants

Moderate risk

Selection into the study may have been related to intervention and outcome, but the authors used appropriate methods to adjust for the selection bias;

or

Start of follow‐up and start of intervention do not coincide for all participants, but the proportion of participants for which this was the case was too low to induce important bias; the authors used appropriate methods to adjust for the selection bias; or the review authors are confident that the rate (hazard) ratio for the effect of intervention remains constant over time.

Serious risk

Selection into the study was related to intervention and outcome;

or

Start of follow‐up and start of intervention do not coincide, and a potentially important amount of follow‐up time is missing from analyses, and the rate ratio is not constant over time.

Critical risk

Selection into the study was strongly related to intervention and outcome;

or

A substantial amount of follow‐up time is likely to be missing from analyses, and the rate ratio is not constant over time.

No information

No information is reported about selection of participants into the study or whether start of follow‐up and start of intervention coincide

Bias in measurement of interventions

Low risk

Intervention status is well defined and based solely on information collected at the time of intervention

Moderate risk

Intervention status is well defined but some aspects of the assignments of intervention status were determined retrospectively

Serious risk

Intervention status is not well defined, or major aspects of the assignments of intervention status were determined in a way that could have been affected by knowledge of the outcome

Critical risk

(Unusual) An extremely high amount of misclassification of intervention status, e.g. because of unusually strong recall biases

No information

No definition of intervention or no explanation of the source of information about intervention status

Bias due to departures from intended

interventions

Low risk

No bias due to departure from the intended intervention is expected, for example if both the intervention and comparator are implemented over a short time period, and subsequent interventions are part of routine medical care, or if the specified comparison relates to initiation of intervention regardless of whether it is continued

Moderate risk

Bias due to departure from the intended intervention is expected, and switches, co‐interventions, and some problems with intervention fidelity are appropriately measured and adjusted for in the analyses. Alternatively, most (but not all) departures from intended intervention reflect the natural course of events after initiation of intervention.

Serious risk

Switches in treatment, co‐interventions, or problems with implementation fidelity are apparent and are not adjusted for in the analyses.

Critical risk

Substantial departures from the intended intervention are present and are not adjusted for in the analysis.

No information

No information is reported on whether there is departure from the intended intervention.

Bias due to missing data

Low risk

Data were reasonably complete;

or

Proportions and reasons of missing participants were similar across intervention groups;

or

Analyses that addressed missing data are likely to have removed any risk of bias.

Moderate risk

Proportions of missing participants differ across

interventions or reasons for missingness differ minimally across interventions;

and

Missing data were not addressed in the analysis.

Serious risk

Proportions of missing participants differ substantially across interventions or reasons for missingness differ substantially across interventions;

and

Missing data were addressed inappropriately in the analysis;

or

The nature of the missing data means that the risk of

bias cannot be removed through appropriate analysis.

Critical risk

(Unusual) There were critical differences between

interventions in participants with missing data that were not, or could not, be addressed through appropriate analysis.

No information

No information is reported about missing data or the

potential for data to be missing

Bias in measurement of outcomes

Low risk

The methods of outcome assessment were comparable across intervention groups;

and

The outcome measure was unlikely to be influenced by knowledge of the intervention received by study participants (i.e. is objective) or the outcome assessors were unaware of the intervention received by study participants;

and

Any error in measuring the outcome is unrelated to

intervention status.

Moderate risk

The methods of outcome assessment were comparable across intervention groups;

and

The outcome measure is only minimally influenced by knowledge of the intervention received by study

participants;

and

Any error in measuring the outcome is only minimally related to intervention status.

Serious risk

The methods of outcome assessment were not comparable across intervention groups;

or

The outcome measure was subjective (i.e. likely to be

influenced by knowledge of the intervention received by study participants) and was assessed by outcome assessors aware of the intervention received by study participants;

or

Error in measuring the outcome was related to intervention status.

Critical risk

The methods of outcome assessment were so different that they cannot reasonably be compared across intervention groups.

No information

No information is reported about the methods of outcome assessment.

Bias in selection of the reported result

Low risk

There is clear evidence (usually through examination of a pre‐registered protocol or statistical analysis plan) that all reported results correspond to all intended outcomes, analyses and sub‐cohorts.

Moderate risk

The outcome measurements and analyses are consistent with an a priori plan; or are clearly defined, and internally and externally consistent;

and

There is no indication of selection of the reported analysis from among multiple analyses;

and

There is no indication of selection of the cohort or subgroups for analysis and reporting on the basis of the results.

Serious risk

Outcome measurements or analyses are internally or

externally inconsistent;

or

There is a high risk of selective reporting from among

multiple analyses;

or

The cohort or subgroup is selected from a larger study for analysis and appears to be reported on the basis of the results.

Critical risk

There is evidence or strong suspicion of selective reporting of results, and the unreported results are likely to be substantially different from the reported results.

No information

There is too little information to make a judgment, for example if only an abstract is available for the study.

Overall judgment about risk of bias

Low risk

The study is judged to be at low risk of bias for all domains.

Moderate risk

The study is judged to be at low or moderate risk of bias for all domains.

Serious risk

The study is judged to be at serious risk of bias in at least one domain, but not at critical risk of bias in any domain.

Critical risk

The study is judged to be at critical risk of bias in at least one domain.

No information

There is no clear indication that the study is at

serious or critical risk of bias and there is a lack of information in one or more key domains of bias (a judgment is required for this).