Types of studies

We will include randomised controlled trials (RCTs), cluster RCTs (cRCTs) and quasi‐RCTs of offline eLearning for medical doctors, in which efficacy outcomes are expressed in quantitative terms. We will include RCTs with unclear or high risk of bias for sequence generation, but will exclude cross‐over trials due to the high likelihood of a carry‐over effect (Higgins 2011). We will include eligible trials reported in conference proceedings and abstracts. When considering cost‐effectiveness we will only use economic data collected as part of efficacy studies that meet our eligibility criteria.

Types of participants

We will include studies in which participants (learners) are medical doctors (including interns) enrolled in a post‐graduate medical education programme, namely, any type of study after a university qualification that is recognised by the relevant governmental or professional bodies and enables the qualification holder primary entry into, or continuation of work in, the healthcare workforce in a more independent or senior role. This includes learners from medicine, and excludes learners of nursing and midwifery, medical diagnostic and treatment technology, therapy and rehabilitation, pharmacy, and traditional, alternative and complementary medicine.

Participants will not be excluded on the basis of age, sex or any other socio‐demographic characteristic. We will exclude studies with mixed participant groups, such as medical doctors, nurses, pharmacists as well as pre‐ and post‐registration healthcare professionals, in which results are not presented separately.

Types of interventions

We will include studies in which offline computer‐based eLearning interventions were used to deliver learning content. CME and CPD programs that involve the use of offline eLearning interventions will be included. CME will be defined as "all educational activities which serve to maintain, develop, or increase the knowledge, skills, and professional performance and relationships that a physician uses to provide services for patients, the public, or the profession" (AAMC 2015) and CPD as "a range of learning activities through which health and care professionals maintain and develop throughout their career to ensure that they retain their capacity to practice safely, effectively and legally within their evolving scope of practice" (HCPC 2015).

Offline computer‐based eLearning interventions refer to the use of personal computers (PCs) or laptops to assist in the delivery of stand‐alone multimedia materials, without the need for internet or LAN connections (Ward 2001; Rasmussen 2014). Trials of offline interventions where a computer was not explicitly used but could potentially be used for delivery (e.g. a videotape) will be included. Content can be delivered via videoconferencing, emails and audio‐visual learning material; and kept in either magnetic storage (floppy discs), optical storage (CD/DVD), flash memory, multimedia cards, external hard discs or information pre‐downloaded from a networked connection, as long as the learning activities do not rely on this connection (Rasmussen 2014). This includes studies where the offline eLearning methods are the sole means by which the interventions are delivered, or where the offline eLearning methods are part of a multi‐component intervention (i.e. blended learning), provided that this is the only method of eLearning used alongside classroom learning. If the eLearning intervention could fit into multiple categories (e.g. both offline and DGBL), the study will be excluded from this review and subsequently included in the more focused review categories.

We will only include studies that make one of the following intervention comparisons:

• offline intervention versus traditional face‐to‐face learning;

• offline intervention versus no intervention;

• blended learning (with an offline component) versus traditional face‐to‐face learning;

• offline intervention versus other types of eLearning interventions;

• offline intervention versus another offline intervention; and

• offline intervention versus blended learning.

Types of outcome measures

In this review we will investigate patient‐related outcomes, and intervention user outcomes, such as knowledge, comprehension, intellectual skill and its application, analysis, synthesis, evaluation and attitudes (cognitive and affective domains). Studies of offline eLearning which assess motor skills and skills‐based learning (psychomotor domain) will be excluded, as they are analysed in a separate review in the eLearning series (currently pre‐publication). However, we will include studies assessing intellectually‐based skills, including reading instructions, solving problems and a variety of other tasks that involve recalling and processing information (Clark 2002). We will include studies reporting at least one of the following primary or secondary outcomes.

Electronic searches

We will search the following databases:

• Cochrane Central Register of Controlled Trials (CENTRAL), (Cochrane Library)

• MEDLINE (via Ovid)

• Embase (via Ovid)

• Web of Science

• Educational Resource Information Centre (ERIC) (via Ovid)

• PsycINFO (Ovid)

• CINAHL (via EBSCO)

• ProQuest Dissertation and Theses Database

The MEDLINE search strategy is presented in Appendix 1. This strategy will be adapted to the other databases. This strategy will be used to search for studies eligible for the whole series of reviews assessing the use of eLearning modalities for pre‐ and post‐registration health professionals’ education and training. Databases will be searched from and including the year 1990 to present. The reason for selecting 1990 as the starting year for our search is because prior to that year, use of the computer and internet was limited to very basic tasks. We will search in English, but will include papers in any language.

Searching other resources

We will search the reference lists of all included studies and relevant systematic reviews. We will also search the International Clinical Trials Registry Platform Search Portal and Current Controlled Trials metaRegister of Controlled Trials to identify unpublished trials, and contact the relevant investigators for further information.

Selection of studies

We will implement the search strategy as described under the ‘Electronic searches’ section and import all references identified to EndNote X7.4. The search results from the different electronic databases will be combined in a single EndNote library and duplicate records of the same reports will be removed. Two review authors will independently screen titles and abstracts to identify studies that potentially meet the inclusion criteria. The full‐text versions of these articles will be retrieved and read in full. Finally, two review authors will independently assess these articles against the eligibility criteria. Any disagreements will be resolved through discussion between the two authors. If no agreement can be reached, a third author will act as arbiter. Studies that initially appeared to be relevant but are excluded at this stage will be listed in the table ‘Characteristics of excluded studies’, where a reason for exclusion will be noted. Two review authors will verify the final list of included studies.

Data extraction and management

Two review authors will independently extract and manage the data for each of the included studies using a structured data recording form. We will pilot the data extraction form and amend it according to the received feedback. In addition to the usual information on study design and participant demographics, we will extract data on relevant fields, including the type of device used, delivery channel/method (CD‐ROM, external hard disc, USB stick etc.), type of content (video, text, images, etc.), and mode of offline eLearning (active or passive, linear or dynamic). We plan to contact study authors in the case of any unclear or missing information. Disagreements between review authors will be resolved by discussion. A third review author will act as an arbiter in cases where disagreements cannot be resolved.

For economic analysis, we will separately extract estimates of resource use associated with interventions and comparators, and estimates of their unit costs. We will also collect information on price per year and the currency used to calculate estimates of costs and incremental costs. We will aim to collect measures of incremental resource use and costs at the individual student level. We will extract both a point estimate and a measure of uncertainty (e.g. standard error or confidence interval (CI)) for measures of incremental resource use, costs and cost‐effectiveness, if reported. Additionally, we will also collect details of any sensitivity analyses undertaken, and any information regarding the impact of varying assumptions on the magnitude and direction of results.

Assessment of risk of bias in included studies

Two review authors will independently assess the risk of bias of the included studies using the Cochrane Collaboration’s ‘Risk of bias’ tool (Higgins 2011). Studies will be assessed for risk of bias in the following domains: random sequence generation; allocation concealment; blinding (participants, personnel); blinding (outcome assessment); completeness of outcome data (attrition bias), selective outcome reporting (reporting bias); and other sources of bias (e.g. baseline imbalance, inappropriate administration of an intervention, and contamination). For cRCTs, we will also assess and report on the risk of bias associated with an additional domain: selective recruitment of cluster participants. Judgements concerning the risk of bias for each study will fall into three categories: high, low, or unclear risk of bias. We will incorporate the results of the 'Risk of bias' assessment into the review using 'Risk of bias' tables, graphs and summaries.

We will also critically appraise any included economic evidence using the British Medical Journal Checklist for authors and peer reviewers of economic submissions (Drummond 1996).

Measures of treatment effect

For continuous data, we will estimate mean differences and 95% CIs for each study. For dichotomous outcomes, we will calculate risk ratios (RR) and 95% CIs. Where possible, all data will be analysed on an intention‐to‐treat basis.

Unit of analysis issues

When a study has more than one active intervention arm, we will label the study arms as ‘a’, ‘b’, ‘c’ and so on. If more than one intervention arm is relevant for a single comparison, we will compare the relevant eLearning arm with the least active control arm, so that double counting of data does not occur. We will list the arms that were not used for the comparison in the 'Notes' section in the table ‘Characteristics of included studies’.

For cRCTs, we will attempt to obtain data at the student level. Where the effects of clustering have already been adjusted for, we will extract the reported estimates and use them directly for our analyses. Otherwise, we will check for unit of analysis errors; and if such errors are found, and sufficient information is available, we will reanalyse the data using the appropriate unit of analysis, by taking account of the intra‐cluster correlation coefficients (ICCs). We will obtain estimates of ICCs by imputing them using estimates from external sources. If it is not possible to obtain sufficient information to reanalyse the data, we will report effect estimates and annotate ‘unit of analysis error' (Ukoumunne 1999).

Dealing with missing data

We will conduct an intention‐to‐treat analysis, including all those who were randomised to either the intervention group or comparator, regardless of losses to follow‐up and withdrawals. Whenever possible, we will attempt to obtain missing data (e.g. number of participants in each group, outcomes and summary statistics) from the original author(s). For dichotomous outcomes, data imputed as ‘complete case analysis’ can be used to fill in the missing values (Higgins 2008). When standard deviations of continuous outcome data are missing, we will try to calculate the standard deviations from relevant statistics, such as 95% CIs, standard errors, or P values. If these are unavailable, then we will contact study authors or impute the standard deviations from other similar studies.

Assessment of heterogeneity

Where studies are judged homogeneous enough (in terms of their populations, interventions, comparator groups, outcomes and study designs) to draw meaningful conclusions when combined, we will consider pooling them together in a meta‐analysis. If such meta‐analyses are feasible, we will assess heterogeneity through a visual inspection of the overlap of forest plots and by calculating the Chi² tests and the I² inconsistency statistics (Higgins 2011). If there is a moderate level of heterogeneity (I² = 50% to 75%), we will explore possible reasons for inconsistency by conducting subgroup analysis. Where substantial clinical, methodological or statistical heterogeneity (I² > 75%) is detected, we will not present pooled data, but will use a narrative approach to data synthesis instead. In this event, we will attempt to explore possible clinical or methodological reasons for this variation by grouping studies that are similar in terms of populations, intervention features, methodological features, or other factors to account for differences in intervention effects (described below).

Assessment of reporting biases

We will assess reporting bias qualitatively based on the characteristics of the included studies (small studies with positive findings), and if information that we obtain from contacting experts and authors or studies suggests that there are relevant unpublished studies. If we include at least 10 studies, we will assess reporting bias using a funnel plot and regression weighted by the inverse of the pooled variance. A regression slope of zero will be interpreted as an absence of small study bias.

Data synthesis

Data will be reported using Review Manager software (RevMan 2014). Extracted data will be entered into tables grouped by study design and type of intervention to create a descriptive synthesis. The results of individual RCTs and cRCTs will be reported as mean differences for continuous variables and risk ratios for dichotomous variables with a 95% CI, as stated above. Using Miller's classification of clinical competence (Miller 1990), the different types of tests for learners' knowledge and skills will be grouped and analysed together. For example, multiple choice questions (MCQs) assessing knowledge (i.e. knows) will be analysed together, and essay questions assessing competence (i.e. knows how) will be analysed together. The focus will therefore be on the testing method rather than the delivery method (i.e. if skills are assessed by a knowledge test, it will be categorised as knowledge).

For learners' attitudes, the different types of assessment will be grouped and analysed as cognitive attitudes, behavioural attitudes or affective attitudes as described by Martin et al (Martin 2002). Learners’ satisfaction will include the satisfaction with and attitudes towards the learning intervention to which they were exposed. Learners’ attitudes and satisfaction will only be assessed in a narrative fashion, as the preliminary work conducted by the Global eHealth Unit suggests that there is a high level of heterogeneity in the operational definition of these outcomes across different studies (George 2014; Rasmussen 2014).

If adverse, unintended, or undesirable effects are reported numerically, for example, the number of participants that dropped out due to social isolation, anxiety from using the computer and/or mean anxiety score, we will analyse the adverse effects using risk ratios for dichotomous outcomes; however for continuous variables we will use mean difference. In the case that adverse, unintended, or undesirable effects are not expressed numerically, we will narratively synthesise the evidence. We also intend to narratively synthesise economic evidence and to identify key determinants of resource use, costs and/or cost‐effectiveness. We will then analyse how these determinants may be distributed within and between different settings to allow readers to determine the implications of findings of presented data in their own settings and to inform their particular context‐specific decisions. We will follow the Cochrane Handbook guidance on incorporating economic evidence in systematic reviews (Higgins 2011).

Where studies report more than one measure for each outcome, we will use the primary measure as defined by the primary study authors in the analysis. Where no primary measure has been reported, a mean value of all the measures for the outcome will be calculated and used in the analysis. The choice of model used to pool data will depend on the level of heterogeneity observed (assessed as described in the Assessment of heterogeneity section) between studies to be included in the meta‐analysis. Where trials and outcomes appear to be homogenous, we will use fixed‐effect models; however if trials appear to be heterogenous then we will use random‐effects models, which provide a more conservative estimate of effect and can be used where there is moderate heterogeneity.

Subgroup analysis and investigation of heterogeneity

We will conduct the following subgroup analyses in this review where possible.

• By countries’ income (LMICs versus high‐income countries).

• By sub‐speciality, using the International Standard Classification of Occupations (ISCO‐08).

• By number of repeated interventions (one‐off versus repeated interventions).

• Comparing quartiles of adherence/time spent on the intervention. We will recalculate and present the measure of adherence/time spent on the intervention as a percentage to account for the difference in intervention duration between studies.

We acknowledge that there are further subgroup analyses that could be performed, e.g. comparing interventions according to learning objectives and interactivity of interventions. In future reviews conducted after completion of our series of initial reviews, authors will be in the best position to look at these subgroup analyses because such comparisons would be most meaningful from the perspective of an educator if multiple methods of eLearning were to be compared.

Sensitivity analysis

Sensitivity analyses will be considered to explore the impact of the 'Risk of bias' dimensions on the outcomes of the review. We will remove studies from the analysis that are deemed to be at high risk of bias after examination of individual study characteristics. To examine the effect on the pooled effects of the intervention we will exclude studies according to the following filters.

• High risk of bias studies (we will include RCTs with unclear or high risk of bias for sequence generation, however where meta‐analysis is feasible and appropriate we will conduct sensitivity analyses excluding these trials to examine the robustness of the meta‐analysis results to methodological limitations of the included studies).

• Small studies (less than 30 participants per randomisation group).

• Source of funding (industry sponsorship (solely industry funded) or mixed sponsorship (public and industry funded, including the free provision of study material only).

• Time lapse between end of intervention and first post‐test (quartiles) as well as last post‐test.

• If studies compared more than one offline eLearning intervention or blended learning intervention to traditional learning, we will perform a sensitivity analysis to assess the impact of successively replacing the results of each intervention group on the measure of effect. Additionally, we will average the mean scores for each intervention group and use this average in the meta‐analysis. We will then compare the difference between the two approaches.

'Summary of findings' table

We intend to prepare 'Summary of findings' tables to present results of the most relevant comparisons, as described in chapter 11 of theCochrane Handbook (Schünemann 2011). We will present the results of meta‐analysis for each of the primary outcomes, as well as potential adverse effects, as defined in the Types of outcome measures section. We will provide a source and rationale for each assumed risk cited in the table(s). We will consider the following limitations: risk of bias, inconsistency of results, indirectness of the evidence, imprecision, and publication bias, and subsequently downgrade the quality of evidence contributing to outcomes where appropriate. Two authors will apply the GRADE criteria independently of one another to assess the quality of evidence using the GRADEprofiler (GRADEpro) software (Schünemann 2011). If meta‐analysis is not feasible, we will present results in a narrative ‘Summary of findings’ table format, as demonstrated by Chan 2011 (see also Cochrane 2014).