Types of studies

For the primary study objectives, we will include the following study designs:

• randomised trials;

• non‐randomised trials, where the assignment is not random;

• controlled before‐after studies provided they have at least two intervention sites and two control sites;

• interrupted time series studies if there is a clearly defined point in time when the intervention occurred and at least three data points before and three after the intervention.

We will include published studies, conference abstracts, and unpublished data. We will include studies irrespective of their publication status and language of publication.

To address the secondary study objectives, we will include all the studies included in the primary analyses, as well as any studies that use descriptive, qualitative, or quantitative methods to describe interventions that are aimed at improving stock‐outs of health commodities.

Types of participants

• All cadres of healthcare providers (i.e. professionals, para‐professionals, and lay health workers) or others involved in the distribution of health commodities located at any level of the health system (e.g. administrative staff, managerial and supervisory staff in purchasing or distribution, or dispensary staff);

• other individuals or groups involved in stock notification, monitoring, and tracking commodity inventories. These individuals or groups may be based in a primary healthcare facility or in the community and must be involved in supporting the delivery of primary healthcare services.

• Clients or recipients of health services

Types of interventions

We will include interventions that are aimed at improving the availability of health commodities including medicines and other medical supplies, using mobile devices, for the delivery of primary healthcare services in healthcare facilities or in the community, if they involve one or both of the following:

• strategies for tracking health commodity inventory using mobile devices. Tracking health commodity inventory may involve the use of databases and dashboards to manage the availability of health commodities and project availability of medical supplies. While some aspects of commodity tracking might involve mobile devices, the data may be linked to a logistics management information system (LMIS) or supply chain management system where inventory levels and historic data are maintained on desktops;

• notification of stock levels conducted via mobile devices. This may involve the transmission of information on stock levels by health workers within healthcare facilities or by members of the community to alert higher‐level facilities about potential stock shortages. For example, health workers at facilities/dispensaries may use text messaging, SMS, or Unstructured Supplementary Service Data (USSD)‐based systems to notify district or central authorities about stock levels. In some interventions of interest, notification of stock levels using mobile phones may be a component of a broader strategy for tracking health commodities.

By “mobile devices” we mean mobile phones of any kind (but not analogue landline telephones), as well as tablets, personal digital assistants, and smartphones.

By “primary healthcare services” we mean a combination of the following:

• the first contact point of healthcare (Awofeso 2004), including care delivered at an individual or community level or both (Muldoon 2006), by individual healthcare providers or teams of providers, and intended to bring care to where people work and live (Muldoon 2006), or coordinate or provide continuity of care (WHO 2008), or both;

• any rehabilitative, therapeutic, preventive, and promotive healthcare (Global Health Watch 2011).

The comparisons for this review will be: tracking commodity inventory and notifying stock levels conducted via mobile devices compared with standard practice (i.e. non‐digital strategies or no intervention).

We will exclude:

• studies that focus on cold chain management only and do not report on stock levels of the vaccine;

• studies where commodity tracking and notification of commodities is conducted on stationary computers or laptops only.

Where tracking or notification via mobile device, or both, is delivered as part of a wider package, we will only include the study if we have judged the mobile component to be the major component of the intervention.

Types of outcome measures

• Availability of commodities, measured, for instance, as decreased stock‐outs, lead time for drug supply, availability at point of care;

• quality of data about stock management (accuracy of data, completeness of data);

• timeliness of stock level reporting, and time between receipt/reporting of data regarding commodity status and appropriate action;

• provider acceptability or satisfaction with the intervention, measured with a validated scale if available;

• resource use (e.g. human resources/time, including additional time spent by providers when managing/transitioning dual paper and digital reporting systems; training, supplies, and equipment);

• unintended consequences that result in adverse effect of the intervention (these could include: misreading or misinterpretation of the data; transmission of inaccurate data, for instance through software formatting errors; interrupted workflow due to infrastructural constraints for battery recharge and network coverage; decreased motivation/trust by health workers in the system if stock replenishment is not reliable; loss/misuse of device).

Electronic searches

An independent Information Specialist (JE) will develop the search strategies in consultation with the review authors.

We will search the following databases for primary studies, from 2000 to the date of search:

• Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Library;

• MEDLINE, Ovid;

• Embase, Ovid;

• POPLINE, K4Health;

• Global Health Library, WHO.

Appendix 1 lists the search strategy for MEDLINE Ovid. Search strategies are comprised of keywords and controlled vocabulary terms. We will not apply any limits on language.

Searching other resources

Selection of studies

A core team of two authors (NH and HB), with assistance where necessary from one additional review author (SA), will be responsible for the selection of studies. We will download all titles and abstracts retrieved by electronic searching to a reference management database and remove duplicates. Two review authors (NH and HB) will independently screen titles and abstracts for inclusion. We will retrieve the full‐text study reports/publication and two review authors (NH and HB) will independently screen the full‐text and identify studies for inclusion and identify and record reasons for exclusion of the ineligible studies. We will resolve any disagreement through discussion or, if required, we will consult a third review author (SA).

We will list studies that initially appeared to meet the inclusion criteria but that we later excluded in the 'Characteristics of excluded studies' table. We will collate multiple reports of the same study so that each study, rather than each report, is the unit of interest in the review. We will also provide any information we can obtain about ongoing studies. We will record the selection process in sufficient detail to complete a PRISMA flow diagram (Liberati 2009).

Data extraction and management

We will modify the Cochrane Effective Practice and Organisation of Care (EPOC) standard data collection form and adapt it for study characteristics and outcome data (EPOC 2017a). We will identify key characteristics of the intervention for abstraction based on the mHealth Evidence Review and Assessment (mERA) guidelines (Agarwal 2016). We will pilot the form on at least one study in the review. Two review authors (NH and HB) will independently extract the following study characteristics from the included studies for the primary analyses:

• general information: title, reference details, author contact details, publication type, funding source, conflicts of interest of study authors;

• population and setting: country, geographical location (rural, urban, peri‐urban), healthcare setting (e.g. facility‐based, community‐based);

• methods: function of the intervention, study design, unit of allocation, study duration;

• participant characteristics: type of user (role, if in the health system; length of training, if any), description of any other participants in the intervention, withdrawals;

• interventions: intervention purpose, components, infrastructure to support the technology, type of technology (software platform), type of mobile device(s) used (smartphone, tablets with a screen size larger than 7 inch, feature phones that can run java applications, basic phone with SMS and call functions, laptops), mode of delivery, content of the intervention, participant/provider training, interoperability, compliance with national guidelines, data security, comparison, fidelity assessment, duration of intervention;

• outcomes: primary and other outcomes specified and collected, time points reported, adverse events, results of any subgroup analyses.

Two review authors (NH and HB) will independently extract outcome data from included studies. We will note in the 'Characteristics of included studies' table if outcome data were reported in an unusable way. We will resolve disagreements by consensus or by involving a third review author (SA).

For the secondary analyses, we will extract all the information listed above, if available, to describe the intervention. To understand factors affecting the implementation of relevant interventions, we will be guided by the Supporting the Use of Research Evidence (SURE) framework (Glenton 2017). This framework provides a possible list of factors that may influence the implementation of health systems intervention. We will review the included studies and extract any data that describe enabling or restricting factors related to the implementation of the intervention based on themes identified in the framework.

Assessment of risk of bias in included studies

Two review authors (NH and HB) will independently assess the risk of bias for each study included in the primary analyses using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions Section 8.5 (Higgins 2011), and guidance from the Cochrane EPOC group (EPOC 2017b). Any disagreement will be resolved by discussion or by involving a third review author (SA). We will assess risk of bias for randomised trial/non‐randomised trials and controlled before‐after studies using the following criteria‐ random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, baseline outcomes measurement, similarity of baseline characteristics, and other bias. For interrupted time series studies, we will assess the risk of bias following these seven standard criteria (EPOC 2017b): independence of the intervention from other changes, pre‐specified shape of the intervention effect, independence of the intervention and data collection procedures, objectivity of the outcomes, bias resulting from missing outcome measures, selective outcome reporting, and other biases such as seasonality.

We will judge each potential source of bias as either high, low, or unclear and provide a quote from the study report together with a justification for our judgment in the 'Risk of bias' table. We will summarize the 'Risk of bias' judgements across different studies for each of the domains listed. We will consider blinding separately for different key outcomes where necessary (e.g. for unblinded outcome assessment, risk of bias for all‐cause mortality may be very different than for a patient‐reported pain scale). Where information on risk of bias relates to unpublished data or correspondence with a trialist, we will note this in the 'Risk of bias' table. We will not exclude studies on the grounds of their risk of bias, but will clearly report the risk of bias when presenting the results of the studies. When considering treatment effects, we will take into account the risk of bias for the studies that contribute to that outcome. We will further perform assessment of quality of evidence using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach (Guyatt 2008). Findings will be summarized in the ‘Summary of findings’ tables.

We will conduct the review according to this published protocol and report any deviations from it in the 'Differences between protocol and review' section of the systematic review.

Measures of treatment effect

We will report pre‐intervention and post‐intervention means and proportions for the intervention and comparison groups. Where possible, we will estimate the effect of the intervention using risk ratios or risk differences for dichotomous data, together with the appropriate associated 95% confidence interval (CI) and mean difference or standardized mean difference for continuous data, together with the 95% appropriate associated CI. We will ensure that an increase in scores for continuous outcomes can be interpreted in the same way for each outcome, explain the direction to the reader, and report where the directions were reversed if this was necessary. For interrupted time series studies, we will report the pre‐ and post‐intervention slopes of analysis, the difference of the slopes and the differences of intercepts at the first intervention point and the predicted intercept by the intervention. If interrupted time series data has been analysed incorrectly, we will reanalyse the data where possible (EPOC 2017c). The data will be analysed in RevMan 5 (Review Manager 2014).

Unit of analysis issues

If cluster‐randomised trials or controlled before‐after studies are included in the review, we will report cluster adjusted risk ratios or differences and their 95% CIs. If the analysis was not adjusted for clustering, we will use the intracluster correlation coefficient (ICC), if available, to adjust the CI. If the ICC value is not available, we will present the results without a measure of variance or precision of effect for outcomes for which there is a unit of analysis error (EPOC 2017d)

Dealing with missing data

We will contact investigators in order to verify key study characteristics and obtain missing outcome data where possible (e.g. when a study is identified as an abstract only). If this is not possible, we will report the data as missing, note this in the 'Risk of bias' tables, and not attempt to impute the missing values. For all outcomes, we will carry out the analyses, to the extent possible, on an intention‐to‐treat basis based on the available cases. However, in assessing adverse events, we will relate the results to the actual treatment received i.e. we will base the analyses on the participants who actually received the intervention and the number of adverse events reported in the studies.

Assessment of heterogeneity

If we find a sufficient number of studies that evaluate similar interventions and report similar outcomes, we will conduct a meta‐analysis. We will examine heterogeneity by visual inspection of forest plots as well as using the I² statistic to measure heterogeneity among the trials in each analysis (Higgins 2011). If we identify heterogeneity we will explore it by pre‐specified subgroup analysis.

Assessment of reporting biases

We will attempt to contact study authors and ask them to provide missing outcome data. Where this is not possible, and the missing data are thought to introduce serious bias, we will explore the impact of including such studies in the overall assessment of results. If we are able to pool more than 10 trials, we will create funnel plots to explore possible publication biases or other causes for asymmetry (Sterne 2011). We will interpret the results of the funnel plot with caution as funnel plot calculations for dichotomous outcomes measured as risk ratios are not well developed, and statistical funnel plot results may not be representative if there are small‐study effects.

Data synthesis

For the primary analyses, when studies evaluate interventions that are similar, we will group these together and summarize the key characteristics of each study in tables, to facilitate comparison across studies. We will undertake meta‐analyses using a random‐effects model only where this is meaningful i.e. if the intervention, context, and outcomes are similar enough for pooling to make sense. A common way that trialists indicate when they have skewed data is by reporting medians and interquartile ranges. When we encounter this we will note that the data are skewed and consider the implication of this. Where multiple trial arms are reported in a single trial, we will include only the relevant arms. If two comparisons (e.g. intervention A versus usual care and intervention B versus usual care) must be entered into the same meta‐analysis, we will halve the control group to avoid double‐counting.

For the secondary analyses, we will identify common themes across all the included studies to qualitatively describe the interventions. If any data on the effectiveness of these interventions is available and not included in the primary analyses results, it will be presented with an assessment of the risk of bias. For the secondary objective on factors affecting the implementation of relevant interventions, we will identify common themes under each Supporting the Use of Research Evidence (SURE) framework category and summarized.

Subgroup analysis and investigation of heterogeneity

We will perform subgroup analysis to assess variations in the delivery of interventions across different geographical settings and intervention characteristics. The subgroup analysis will be conducted only if sufficient number of trials are available to make statistically significant comparisons between groups.

• Type of geographic setting: (for example, urban, rural, peri‐urban; LMICs), as we anticipate that the intervention may have different effects due to social and economic differences between settings;

• type of intervention characteristics: interventions aimed at reducing stock‐outs and improving the supply of commodities vary largely in their characteristics. We will describe each intervention by intervention characteristics described under the data extraction section of this protocol. If sufficient number of studies have similar characteristics, we will conduct subgroup analyses led by meaningful practice questions. For example, analysing studies on stock‐out notification system alone, versus notification system that is integrated with a commodity management system.

Sensitivity analysis

We will perform sensitivity analyses defined a priori to assess the robustness of our conclusions and explore its impact on effect sizes. This will involve restricting the analysis to published studies, removing studies that have a high risk of bias based on the 'Risk of bias' assessment from any meta‐analyses.