Data extraction and management

Two review authors will independently undertake data extraction from all included studies using a modified version of the Cochrane EPOC data extraction template (EPOC 2015). We will resolve discrepancies in judgment by discussion and consensus, or by consulting a third review author if necessary. Where information will be missing, we will attempt to contact the study authors, and if data are not available, the study will be recorded as an 'Included study without data'. In addition to capturing EPOC standardised data extraction criteria, we will consider the following study or intervention characteristics: a) clinical area addressed; b) type of healthcare professionals (e.g. physician, nurse, physiotherapist, student nurse, resident); c) pre‐licensure healthcare professionals (student nurses, residents); d) type of healthcare stakeholders (e.g. patients, consumer, decision maker, researcher); e) conceptual or theoretical underpinnings of the intervention (i.e. part of a theory based intervention); f) known effectiveness of the intervention for changing of healthcare professional behaviours (e.g. evidence‐based intervention); g) number of components included in the intervention; h) source and authors of the intervention (e.g. professional organisation, governmental agency, health professionals training schools); i) mode of delivery (e.g. individuals or groups); j) frequency and timing of the intervention and its duration; k) type of CWA (i.e. Wiki, Google Knol, Google Docs, Google Wave); l) presence of a moderator; m) presence of a pre‐existing community of practice or community of learners; n) open versus closed wiki or CWA; o) authorship clearly stated; p) type of conversational features integrated with the CWA (e.g. discussion pages, email, comment page); q) format of knowledge being shared within the CWA (e.g. reminders, care pathways, textbook format); r) formal teaching about how to use a wiki. For economic analysis data extraction, we plan to develop a data collection form specifically for use with health economic studies, based on the template used to produce the National Health Service (NHS) Economic Evaluation Database (EED) structured abstracts (Craig 2007).

Assessment of risk of bias in included studies

We plan for two review authors to independently assess the risk of bias of included studies, with disagreements resolved by discussion and consensus. For RCT, NRCT, and CBA study designs, we intend to assess and report the risk of bias for the following nine domains in accordance with Cochrane EPOC guidance (EPOC 2016): 1) adequate sequence generation; 2) allocation concealment; 3) blinding; 4) incomplete outcome data addressed; 5) free of selective reporting; 6) free of other bias; 7) baseline outcomes similar; 8) free of contamination; and 9) baseline characteristics similar.

For ITS studies and repeated measure studies, we plan to assess and report the risk of bias for the following seven domains (EPOC 2016): 1) independence from other changes in time; 2) shape of the intervention effect pre‐specified; 3) intervention unlikely to affect data collection; 4) blinding; 5) incomplete data outcome; 6) free from selective outcome reporting; and 7) free from other biases.

We plan to assign an overall assessment of the risk of bias (high, moderate, or low risk of bias) to each of the included studies using the approach suggested in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2011). We will prepare a summary table presenting an assessment of the risk of bias within and across studies for important outcomes.

For health economic studies, we plan to assess risk of bias using Cochrane's tool for assessing risk of bias as well as the Consolidated health economic evaluation reporting standards (CHEERS) developed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR (Higgins 2011; Husereau 2013)).

Measures of treatment effect

Reporting

For each study, we will report data in natural units. Where baseline results are available from RCTs, NRCTs, CBA studies, ITS studies, and RMSs, we will report pre‐ and post‐intervention means or proportions for both study and control groups. We will calculate the unadjusted and adjusted (for any baseline imbalance) absolute change from baseline with 95% confidence intervals (CIs). For ITS studies, we will report the main outcomes in natural units and two effect sizes: the change in the level of outcome immediately after the introduction of the intervention, and the change in the slopes of the regression lines. Both of these estimates will be necessary for interpreting the results of each comparison. For example, there could be no change in the level immediately after the intervention, but there could be a significant change in slope.

Analytic approach

We will use the statistical methods proposed in Grimshaw 2004 to guide data analysis and presentation. Our statistical analysis will be based upon consideration of dichotomous practitioner process outcomes, continuous practitioner process outcomes, dichotomous patient outcome measures, and continuous patient outcome measures. In cases where there are insufficient data to calculate these effect sizes, we will present results of studies as reported by the authors.

Where studies report more than one outcome, we will extract data on the primary outcome (as defined by the authors of the study). However, if the study reports more than one outcome and none of them are denoted as the primary variable, we will rank the effect sizes for all the variables and take the median value. In all cases, the direction of the effect size will be standardised so that a positive difference between post‐intervention percentages or means will indicate a positive outcome.

Primary analyses

Where possible, we will present the results for all comparisons using a standard method of presentation. We will report results for categorical and continuous outcomes separately. For comparisons of RCTs, NRCTs, CBA studies, ITS studies, and RMSs, we will report (separately for each study design) median effect sizes across included studies, interquartile ranges (IQRs) of effect sizes across included studies, and the range of effect sizes across included studies. We will calculate standardized effect sizes for continuous measures by dividing the difference in mean scores between the intervention and comparison group in each study by the standard deviation of the comparison group. This results in a ‘scale free’ estimate of the effect for each study, which can be interpreted and pooled across studies regardless of the original scale of measurement used in each study (Laird 1990). When available, we will also report the pre‐ and post‐intervention study and control data in natural units, with statistical significance provided across groups. For ITS studies, we will follow the recommendation of Ramsay 2003 and compute, where possible, a difference in slopes and/or a level effect.

Secondary analyses

Secondary analyses will explore the consistency of primary analyses with other types of endpoints.

Unit of analysis issues

Methods for re‐analysis of RCTs, NRCTs and CBA studies with potential unit of analysis errors

Comparisons that randomise or allocate clusters (healthcare professionals or organisations) but do not account for clustering during analysis have 'potential unit of analysis errors' resulting in artificially extreme P values and overly narrow CIs (Ukoumunne 1999). Therefore, when possible, we will attempt to re‐analyse studies with potential unit of analysis errors, by contacting primary authors for missing information. If a comparison is re‐analysed, we will quote the P value and will annotate it as being 're‐analysed'. If this is not possible, we will only report the point estimate.

Methods for re‐analysis of ITS studies comparisons with inappropriate analysis

Where possible, we will use time series regression to re‐analyse each comparison. We will estimate the best‐fit pre‐intervention and post‐intervention lines using linear regression and autocorrelation adjusted for using the Cochrane‐Orcutt method where appropriate (Draper 1981). We will test for first‐order autocorrelation statistically using the Durbin‐Watson statistic and higher order autocorrelations will be investigated using the autocorrelation and partial autocorrelation function.

Dealing with missing data

All efforts will be made to contact the original authors if missing data are noted in an included study. If this is impossible, we will clearly state if the data seemed to be ‘missing at random’. If this is the case, we will analyse only the available data and ignore the missing data (Higgins 2011). If data seems ‘not to be missing at random’, we will clearly state our assumptions and we will either impute the missing data with replacement values, and treat these as if they were observed (e.g. last observation carried forward, imputing an assumed outcome such as assessing all were poor outcomes, imputing the mean, imputing based on predicted values from a regression analysis). We will also explore the impact of missing values on our results with the use of statistical models that allow for missing data, making clearly stated assumptions about their relationship with the available data. We will perform sensitivity analyses to assess how sensitive our results are to reasonable changes in the stated assumptions (Sensitivity analysis). Finally, we will address the potential of missing data on the findings of our review in our discussion section.

Assessment of heterogeneity

We will analyse included studies to determine whether there were studies sufficiently similar in participants characteristics (e.g. age, gender), study design, type of CWA (e.g. wiki, Google Docs), type of knowledge contained in CWA (e.g. wikis sharing reminders), environmental setting (e.g. critical care), health condition (e.g. acute disease), and outcome measurement, to allow for a meta‐analysis of their combined data using a random‐effects model. If studies are too heterogeneous, we will present a descriptive review of the included studies, using a narrative summary along with extracted data in tables and figures.

Where meta‐analysis is possible, we will assess statistical heterogeneity between trials using the Chi² test and the I² statistic (a Chi² P value of less than 0.05, or an I² value of 50% or higher will be considered to indicate substantial heterogeneity). If heterogeneity is identified, we will undertake a subgroup analysis to investigate its possible sources (Subgroup analysis and investigation of heterogeneity). We will also conduct a meta‐regression if there are enough studies to assess the effect of the possible sources of heterogeneity.

Assessment of reporting biases

If sufficient studies are found, we will assess publication bias graphically using funnel plots and statistically using Begg and Egger tests (Egger 1997). While funnel plot asymmetry may indicate publication bias, this is not always the case. We will consider all possible explanations for this asymmetry and discuss them in the review.

Data synthesis

We expect to find statistical heterogeneity, given the range of disparate settings where wikis and CWAs have been tested, and the different content within these different CWAs. This makes it unlikely that statistical pooling will be feasible, but if there appears to be a body of studies amenable to meta‐analysis, then we will display their results graphically, and view them to assess heterogeneity. It is unlikely that this review will find many studies for inclusion in a meta‐analysis, and so statistical tests for heterogeneity will be insensitive. We will nevertheless carry out these tests. We will conduct any subsequent meta‐analyses using a random‐effects model. In comparisons of homogeneous groups of studies with different outcome measures, we will consider using a single effect size measure (standardised mean differences).

If meta‐analysis is not possible, we will provided narrative and qualitative summaries. We will group studies by criteria included in this protocol (e.g. type of CWA), and vote count studies grouped according to simple trichotomies on outcome: effective, no difference, or harm. We will perform formal qualitative textual analysis for each of these groups of studies, looking for common themes in the description of the intervention within each outcome group, and systematic differences between groups of studies with different outcomes.

We plan to summarise the findings using the GRADE approach with input on the most important outcomes from patients and other decision makers, and produce a 'Summary of findings' table with selected studies, according to the Cochrane Handbook for Systematic Reviews of Interventions (Guyatt 2011; Higgins 2011).

Subgroup analysis and investigation of heterogeneity

We plan to investigate heterogeneity by preparing tables and bubble plots, and comparing effect sizes of studies grouped according to potential effect modifiers to investigate heterogeneity. For example, the following variables will guide our exploration of heterogeneity: the type of CWA (wikis compared to Google Docs), the age of wiki users and editors (‘Internet generation’ (born between 1976 and 1994) compared to participants born before the ‘Internet generation’). We will also consider the following subgroups to explore the following effect modifiers of the intervention on the magnitude of effects observed across studies: a) open versus closed CWAs; b) clear authorship identification; c) the presence of a moderator; and d) being part of a theory‐based intervention.

Data allowing, we also plan to conduct the following comparisons: a) CWA intervention versus no intervention; b) CWA intervention versus any single intervention (any static intervention where users did not have the capability to edit the content of the page); c) a non‐wiki CWA intervention versus a wiki CWA intervention; d) multifaceted intervention where a CWA was included versus any multifaceted intervention without a CWA (any intervention where a CWA is a part of the intervention but was not the only active ingredient); e) a CWA with an implementation strategy versus a wiki or CWA without an implementation strategy.

A single intervention will be any static resource (e.g., a traditional web page) intervention where users do not have the capability in editing the content of the page. A multifaceted intervention will be any intervention where a CWA is a part of the intervention but is not the only active ingredient. For example, a study could explore the impact of a virtual learning environment that contains a wiki and other applications (e.g., discussion boards, social networks, live virtual classroom meetings) compared to traditional teacher‐led classroom meetings or other educational interventions like problem‐based learning groups.

Sensitivity analysis

We will undertake sensitivity analyses for the allocation of missing data by best and worst case analysis and will also undertake a sensitivity analysis on the basis of our evaluation of the risk of bias of each study. Since studies would not have been blinded, and already contained one domain with high risk of bias, our sensitivity analyses would include studies with at least two high risk of bias domains.

Economics issues

As described in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2011), we will summarise methodological characteristics and results of included economic evaluations using additional tables (Additional tables), supplemented by a narrative summary that will compare and evaluate methods used and principal results between studies. We will tabulate unit cost data when available. We will report the currency and price year that applies to measures of costs in each original study, alongside measures of costs, incremental costs, and incremental cost‐effectiveness, by study. Where details of currency and price year are available in original studies, we will convert measures of costs, incremental costs, and cost‐effectiveness to current US dollar value, using implicit price deflators for gross domestic product (GDP) and purchasing power parities (PPPs). We will use the Campbell and Cochrane Economics Methods Group (CCEMG) and the Evidence for Policy and Practice Information and Coordinating Centre (EPPI‐Centre) 'CCEMG‐EPPI Cost Converter' (eppi.ioe.ac.uk/costconversion/), which is a web‐based tool for adjusting estimates of cost expressed in one currency and price year to a specific target currency and price year (Shemilt 2010).