Tracking health commodity inventory and notifying stock levels via mobile devices: a mixed methods systematic review

Smisha Agarwal; Claire Glenton; Nicholas Henschke; Tigest Tamrat; Hanna Bergman; Marita S Fønhus; Garrett L Mehl; Simon Lewin

doi:10.1002/14651858.CD012907.pub2

Seguimiento del inventario de productos sanitarios y notificación de las existencias mediante dispositivos móviles: una revisión sistemática de métodos mixtos

Declaraciones de intereses de los autores

Versión publicada: 28 octubre 2020 Historial de versiones

https://doi.org/10.1002/14651858.CD012907.pub2

Contraer todo Desplegar todo

Resumen

disponible en

Antecedentes

Los sistemas de salud necesitan un acceso oportuno y fiable a los fármacos y productos sanitarios esenciales, pero los problemas de acceso son frecuentes en muchos contextos. Las tecnologías móviles ofrecen posibles soluciones de bajo costo al problema de la distribución de fármacos y la disponibilidad de productos básicos en los centros de atención primaria. Sin embargo, existe poca evidencia sobre el uso de los dispositivos móviles para hacer frente a la escasez de productos básicos y no ofrece un camino claro a seguir.

Objetivos

Objetivo principal

Evaluar los efectos de las estrategias para notificar los niveles de existencias y el seguimiento digital de los productos básicos relacionados con la atención sanitaria y las existencias mediante dispositivos móviles en todo el sistema de atención primaria

Objetivos secundarios

Describir las estrategias de dispositivos móviles que se utilizan actualmente para mejorar el informe y el seguimiento digital de los productos básicos sanitarios

Identificar los factores que influyen en la aplicación de intervenciones con dispositivos móviles destinadas a reducir el agotamiento de existencias de productos sanitarios

Métodos de búsqueda

Se realizaron búsquedas en CENTRAL, MEDLINE Ovid, Embase Ovid, Global Index Medicus de la OMS, POPLINE K4Health y en dos registros de ensayos en agosto de 2019. También se buscaron en Epistemonikos revisiones sistemáticas relacionadas y estudios primarios potencialmente elegibles. Se realizó una búsqueda de literatura gris utilizando mHealthevidence.org, y se emitió un llamamiento para la presentación de artículos a través de comunidades digitales práctica sanitaria famosas. Por último, se realizaron búsquedas de citas de los estudios incluidos. Se buscaron los estudios publicados después del año 2000, en cualquier idioma.

Criterios de selección

Para el objetivo primario, se incluyeron ensayos aleatorizados individuales y por conglomerados, estudios controlados tipo antes y después (before‐after studies) y estudios de series de tiempo interrumpido. En cuanto a los objetivos secundarios, se incluyó cualquier diseño de estudio, que podía ser cuantitativo, cualitativo o descriptivo, que tuviera como objetivo describir las estrategias actuales de rastreo de productos básicos o de notificación de existencias mediante dispositivos móviles; o que tuviera como objetivo explorar los factores que influyen en la aplicación de esas estrategias, incluidos los estudios de aceptabilidad o viabilidad.

Se incluyeron estudios de todos los colectivos de profesionales sanitarios, incluidos los trabajadores sanitarios legos, y otras personas que participan en la distribución de productos sanitarios (personal administrativo, personal directivo y de supervisión, personal de dispensarios); y todas las demás personas que participan en la notificación de existencias, que pueden estar localizados en un centro o en un entorno comunitario, y que participan en la prestación de servicios de atención primaria.

Se incluyeron las intervenciones destinadas a mejorar la disponibilidad de los productos sanitarios mediante el uso de dispositivos móviles en ámbitos de atención primaria. Para el objetivo principal, se incluyeron los estudios que compararon el seguimiento de los productos sanitarios o la notificación de existencias a través de dispositivos móviles con la práctica habitual. En cuanto a los objetivos secundarios, se incluyeron los estudios de seguimiento de los productos sanitarios y de notificación de existencias a través de un dispositivo móvil, si se podían obtener datos pertinentes para los objetivos secundarios.

Obtención y análisis de los datos

Dos autores, de forma independiente, revisaron todas las entradas, extrajeron los datos de los estudios incluidos y evaluaron el riesgo de sesgo. Para el análisis de los objetivos primarios, se informaron las medias y las proporciones cuando fue apropiado. Se utilizó el enfoque GRADE para evaluar la certeza de la evidencia y se preparó una tabla "Resumen de los hallazgos". Para el objetivo secundario, dos autores examinaron independientemente todos los registros, extrajeron los datos de los estudios incluidos y aplicaron un enfoque de síntesis temática para resumir los datos. La limitación metodológica se evaluó mediante la herramienta Ways of Evaluating Important and Relevant Data (WEIRD). Se utilizó el enfoque GRADE‐CERQual para evaluar la confianza en la evidencia, y se preparó una tabla "Resumen de los hallazgos cualitativos".

Resultados principales

Objetivo principal

Para el objetivo principal se incluyó un estudio controlado tipo antes y después realizado en Malawi.

No hay certeza del efecto de cStock más una mejor gestión, o de cStock más el transporte efectivo de productos sobre la disponibilidad de los productos, la calidad y la puntualidad de la gestión de las existencias, ni sobre la satisfacción y la aceptabilidad, porque se consideró que la evidencia es de certeza muy baja. El estudio no informó sobre el uso de los recursos ni sobre las consecuencias imprevistas.

Objetivo secundario

Para los objetivos secundarios se incluyeron 16 estudios, con una variedad de diseños de estudio, que describían un total de 11 intervenciones. Todos los estudios se realizaron en países de África (Tanzania, Kenya, Malawi, Ghana, Etiopía, Camerún, Zambia, Liberia, Uganda, Sudáfrica y Rwanda) y de Asia (Pakistán e India).

La mayoría de las intervenciones tenían como objetivo hacer visibles los datos sobre las existencias y los posibles casos de agotamiento de las mismas a los administradores, que podían entonces adoptar medidas correctivas para hacerles frente.

Se han identificado varios factores que pueden influir en la aplicación de la notificación y el seguimiento de las existencias a través de un dispositivo móvil.

Entre ellos figuran los desafíos relacionados con cuestiones de infraestructura, como el deficiente acceso a la electricidad o a Internet, y cuestiones más amplias relacionadas con los sistemas de salud, como la escasez de fármacos a nivel nacional, que no se puede mitigar con intervenciones en el nivel de atención primaria (confianza baja). Se determinó que hubo varios factores importantes, entre ellos las firmes asociaciones con las autoridades locales, las empresas de telecomunicaciones, los proveedores de sistemas técnicos y las organizaciones no gubernamentales (confianza muy baja); la disponibilidad de datos sobre el nivel de las existencias en todos los niveles del sistema de salud (confianza baja); el papel de la supervisión de apoyo y la gestión receptiva (confianza moderada); la familiaridad y la capacitación de los trabajadores sanitarios en el uso de los dispositivos digitales (confianza moderada); la disponibilidad de conocimientos técnicos de programación para el desarrollo inicial y el mantenimiento continuo de los sistemas digitales (confianza baja); incentivos, como el crédito telefónico para uso personal, para apoyar el uso regular del sistema (confianza baja); sistemas fáciles de usar construidos con la participación del usuario (confianza moderada); uso de teléfonos móviles básicos o personales para favorecer una adopción más fácil (confianza baja); consideración de las características de los programas informáticos, como la comunicación bidireccional (confianza baja); y disponibilidad de datos en un formato fácil de usar, como un tablero interactivo (confianza moderada).

Conclusiones de los autores

Se necesitan más estudios bien diseñados y controlados que comparen la notificación de existencias y la gestión de los productos básicos mediante dispositivos móviles con los sistemas de gestión analógico de productos básicos. Es necesario realizar más estudios para comprender los factores que pueden influir en la aplicación de esas intervenciones y la forma en que las consideraciones relacionadas con la aplicación difieren según las variaciones de la intervención.

PICO

Population

Intervention

Comparison

Outcome

El uso y la enseñanza del modelo PICO están muy extendidos en el ámbito de la atención sanitaria basada en la evidencia para formular preguntas y estrategias de búsqueda y para caracterizar estudios o metanálisis clínicos. PICO son las siglas en inglés de cuatro posibles componentes de una pregunta de investigación: paciente, población o problema; intervención; comparación; desenlace (outcome).

Para saber más sobre el uso del modelo PICO, puede consultar el Manual Cochrane.

Resumen en términos sencillos

disponible en

Uso de los teléfonos móviles para hacer un seguimiento de los medicamentos y notificar a los superiores cuando se necesitan más

¿Cuál es el objetivo de esta revisión?

En esta revisión Cochrane, se intentó evaluar si los trabajadores sanitarios tendrían un mejor acceso a los medicamentos y otros suministros, si utilizaban teléfonos móviles u otros dispositivos móviles para llevar un registro de la cantidad de material disponible, y para notificar a los superiores cuando necesitaban más suministros. También se describió cómo los trabajadores sanitarios utilizan actualmente estos sistemas móviles y qué factores influyen en su uso.

Mensajes clave

No se sabe si el uso de los teléfonos móviles para hacer un seguimiento de los materiales médicos proporciona a los trabajadores sanitarios de la atención primaria un mejor acceso a estos suministros. Esto se debe a que la certeza de la evidencia disponible fue muy baja. Sin embargo, la evidencia apunta a varios factores que podrían influir en la aplicación de esos sistemas. Entre ellos figuran el intercambio de datos a todos los niveles del sistema de salud; un acceso fiable a la electricidad y a Internet; teléfonos y sistemas fáciles de usar y funcionales; un buen soporte técnico; y suficiente capacitación y supervisión. Los sistemas digitales que funcionan bien sólo tendrán éxito si se dispone realmente de suministros médicos.

¿Qué se estudió en la revisión?

Los trabajadores sanitarios necesitan medicamentos, vacunas, jeringuillas y otros materiales para hacer su trabajo correctamente. Sin embargo, muchos centros de atención sanitaria, especialmente en los países pobres, suelen carecer de materiales porque los gobiernos no pueden permitirse comprarlos o porque no tienen buenos sistemas para distribuirlos a los lugares adecuados en el momento oportuno.

Para abordar algunos de estos problemas, los gestores y los trabajadores sanitarios pueden utilizar los teléfonos móviles para hacer un seguimiento de material, decidir cuánto más se debe pedir y cuándo, y hacer saber a las personas en los niveles superiores del sistema cuando le están agotando las existencias.

El objetivo principal de esta revisión fue determinar si los trabajadores sanitarios que utilizan estos sistemas móviles tenían mejor acceso a los materiales que los trabajadores sanitarios que utilizan otros sistemas, o que no tenían ningún sistema. También se examinó cómo se utilizan estos sistemas móviles en ámbitos de atención primaria de la salud, y los factores que influyen en su uso.

¿Cuáles son los principales resultados de la revisión?

Se encontró un estudio de Malawi que midió el efecto de los sistemas de telefonía móvil en el acceso a los medicamentos y los materiales en los centros de salud. Sin embargo, no se sabe si estos sistemas mejoran el acceso o no, porque la certeza de esta evidencia fue muy baja.

Se encontraron 16 estudios realizados en países de África y Asia, en los que los autores de los estudios identificaron varios factores que podrían ayudar a establecer sistemas exitosos.

‐ Los autores de los estudios indicaron que permitir que los funcionarios de todos los niveles del sistema de salud vean los datos podría ayudarles a responder a tiempo y evitar la escasez de material (confianza baja).

‐ Los autores de los estudios destacaron que el escaso acceso a la electricidad y a Internet podría dificultar a los trabajadores sanitarios cargar los teléfonos y enviar datos (confianza moderada).

‐ Los autores de los estudios señalaron la importancia de sistemas fáciles de utilizar, construidos con la participación de los usuarios (confianza moderada); el acceso a expertos técnicos para desarrollar y mantener el sistema (confianza baja); el apoyo y la supervisión adecuados de los trabajadores sanitarios (confianza moderada); y la capacitación de los trabajadores sanitarios en el uso del sistema móvil en sí (confianza moderada).

‐ Un autor indicó que añadir créditos telefónicos a los teléfonos móviles de los trabajadores sanitarios para su uso personal, podría motivarlos y mejorar su utilización del sistema (confianza baja). Los autores de los estudios también indicaron varias funciones de los teléfonos móviles que podrían ser útiles para los trabajadores sanitarios que utilizan estos sistemas. Estas incluyeron teléfonos que podían tomar fotografías y trazar mapas de ubicaciones geográficas, sistemas que permitían enviar mensajes de texto sin costo, comunicación bidireccional y fácil intercambio de información (confianza baja). Pero los autores de los estudios también indicaron que a los trabajadores sanitarios les podría resultar más fácil utilizar teléfonos básicos y mantener sus propios teléfonos personales (confianza baja).

‐ Los autores de los estudios subrayaron que los sistemas de notificación digital sólo darán a los trabajadores sanitarios un mejor acceso a los suministros si estos se encuentran realmente disponibles a nivel de distrito o nacional (confianza baja).

¿Cuál es el grado de actualización de esta revisión?

Se hicieron búsquedas de estudios que se habían publicado hasta agosto de 2019.

Authors' conclusions

Implications for practice

Below are a set of questions that may help health system or programme managers when implementing or planning tracking health commodity inventory and notifying stock levels via mobile devices.

1. Have you considered the availability of necessary infrastructure?

Do health workers have reliable access to electricity and internet connectivity?
Where network access is a challenge, are there systems in place so that staff can work offline until connectivity is restored?
Have you considered whether health workers might prefer to use basic or simple mobile phones, or their own personal phones, rather than smart phones, for instance because their own phones might be cheaper and easier to use?
Do you have reliable access to the medicines and supplies that local health facilities need? Are systems in place for regular procurement of medicines at the national and sub‐national level, so that supplies can be made available when requested through digital notification systems?

2. Have you taken the needs and view of users into account when developing, planning, and implementing the use of mobile devices for stock notification and tracking?

Have you considered the type and format of data and information that should be presented on a dashboard?

Will you involve users in an iterative design process, with the system evolving as the needs of users and the health system become clear?
Have you selected technology that is appropriate for your setting’s data network coverage, data needs, and local capacity for maintenance?
Have you put in place mechanisms to select software that aligns with programme needs for specific functions, such as capturing images, mapping geographical locations, and two‐way communication?

3. Have you considered how to work with key partners and how to share data?

Have you considered partnering with local authorities to facilitate effective implementation? This may include partnerships with the government, local telecommunication companies, technical systems providers, or non‐governmental organisations (NGO).
Have you ensured that data are available at all levels of the health system? Would developing digital dashboards help in making data available to managers at district, regional, and national levels? If developing dashboards is not feasible, is it possible to develop a paper‐based system for sharing stock availability reports with district and national levels?

4. Is there a plan for addressing training and support needs?

Have you ensured that health facility staff have adequate training in the use of the digital system, and where necessary, in the use of any equipment, such as smart phones?
Do health facility staff have sufficient mobile phone credit to support timely reporting of stock data and adoption of the system?
Do you have the technical programming expertise that is needed to develop, install, and maintain the system on an ongoing basis?
Have you ensured that health facility staff have access to supportive supervision and responsive management structures?

These questions were drawn, based on this review. They also align with similar implications for practice identified in a linked review on health workers’ perceptions and experiences of using mHealth technologies to deliver primary healthcare services (Odendaal 2020).

Implications for research

Further, well conducted, comparative evaluations are needed to robustly establish the effects of stock notification and commodity management via mobile devices on improved availability, improve timeliness of stock availability (average time between sending an order request and receiving health products), and reduced stockout of commodities at the point of care. Given the practical challenges in randomising such systems‐level interventions, researchers may consider alternate study designs, such as controlled before‐after studies with at least two intervention and control sites, and interrupted time series studies with at least three data points before and after the intervention.
Interventions targeted at improvements in stock management have a large amount of variability in core intervention components. Therefore, it is important that research studies describe interventions in sufficient detail that readers can discern the core components.
Currently, there is no standardisation of outcomes related to measurement of stockouts and other outcomes of interest in this review. In the studies included in this review, stockout outcomes were reported in three different ways: the proportion of health workers reporting stockout of specific drugs on the day of the interview; the proportion of health workers reporting stockout of specific drugs in the last 30 days; and the proportion of women of reproductive ages who reported a stockout (of preferred contraceptive) at the health facility (over an unspecified time period). Consistency in measurement of outcomes, and use of standardised metrics, where possible, can help facilitate comparability, pooling, and meta‐analysis of the research findings.
Comparative evaluations should be accompanied by process evaluations to enhance understanding of the mechanisms and contexts within which different mobile‐based stock notification and commodity management interventions work well, and the views and experiences of those using these systems. Understanding the conditions under which such systems adequately operate is valuable. For example, the review shows that misalignment of national stock ordering systems and local needs limits the effectiveness of such digital systems. These process evaluations need to be well conducted, and should report their methods clearly.
Studies are needed of how different mobile‐based stock notification and commodity management systems can be sustainably adopted and used. This review suggests that the use of incentives, such as mobile phone airtime credit, may be considered, and it would be valuable to test empirically the effects of this and other incentives on the adoption and long‐term use of such systems.
The cost‐effectiveness of different mobile‐based stock notification and commodity management systems, compared to paper‐based stock‐management systems, should be assessed.
While there are certain common implementation considerations for mobile‐based stock notification and commodity management systems, factors influencing implementation may vary by the exact type of intervention. Research studies should identify specific factors influencing implementation by intervention characteristics.

Summary of findings

Open in table viewer

Summary of findings 1. Primary objective: mobile stock notification with enhanced management compared to standard care

Mobile stock notification (cStock) with enhanced management (EM) compared to standard care in primary healthcare settings
Patient or population: healthcare workers and other health professionals involved in commodity or stock management Setting: primary healthcare setting in Malawi Intervention: mobile stock notification with enhanced management (cStock + EM), which involved quality improvement teams tasked with using the data supplied by the stock notification system Comparison: standard care, which involved routine stock management with mobile stock notification, or any other digital intervention
Outcomes	*Illustrative comparative risks (95% CI)**		Relative effect (95% CI)	No of participants (studies)	Certainty of the evidence (GRADE)	Results in words
Outcomes	Assumed risk with standard care	Corresponding risk with cStock + EM	Relative effect (95% CI)	No of participants (studies)	Certainty of the evidence (GRADE)	Results in words
Availability of commodities
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of cotrimoxazole to treat bacterial infections)	167 per 1000 healthcare workers	160 per 1000 healthcare workers (82 to 317)	RR 0.96 (0.49 to 1.90)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of cotrimoxazole because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of artemether‐lumefantrine 2 X 6 to treat malaria caused by Plasmodium facilparum)	189 per 1000 healthcare workers	136 per 1000 healthcare workers (68 to 272)	RR 0.72 (0.36 to 1.44)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of artemether‐lumefantrine because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of oral rehydration salts (ORS) to treat dehydration)	256 per 1000 healthcare workers	258 per 1000 healthcare workers (156 to 432)	RR 1.01 (0.61 to 1.69)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of oral rehydration salts because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of zinc 20 mg to treat diarrhoea)	211 per 1000 healthcare workers	209 per 1000 healthcare workers (118 to 376)	RR 0.99 (0.56 to 1.78)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of zinc because it is supported by very low‐certainty evidence.
Quality of stock management
Quality of data about stock management (assessed by the extent to which HSAs (intervention group participants) sent messages about the stocks on hand for all the products they managed)	In the intervention group, an average of 85% (N = 393) of the health surveillance assistants (HSA) who managed relevant medicines reported completely on stock levels. This outcome was not assessed in the comparison group.			393^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on quality of data about stock management because it is supported by very low‐certainty evidence.
Timeliness of stock management
Time between stock‐level reporting and appropriate action (measured over an 18‐month period (January 2012 to June 2013)	In the intervention group, health facilities took an average of 12.8 days to fulfil an order requested by the health surveillance assistants (lead time). This outcome was not assessed in the comparison group.			393^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on the timeliness of stock management because it is supported by very low‐certainty evidence.
Satisfaction and acceptability
Provider acceptability and satisfaction (proportion of participants who reported using the digital intervention)	In the intervention group, the proportion of participants who reported using the digital intervention (cStock) as the primary means for ordering health products was 97% (N = 81). This outcome was not assessed in the comparison group.			81^d (1 CBA)^a	⊕⊝⊝⊝ Very low^e	We are uncertain of the effect of this approach on provider satisfaction with stock management because it is supported by very low‐certainty evidence.
Resource use
No studies were identified that reported on this outcome
Unintended consequences
No studies were identified that reported on this outcome
The basis for the assumed risk (e.g. the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). 95% CI: 95% confidence interval; RR:* risk ratio; CBA: controlled before‐after trial
GRADE Working Group grades of evidence High certainty: Further research is very unlikely to change our confidence in the estimate of effect. Moderate certainty: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low certainty: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low certainty: We are very uncertain about the estimate.
^aShieshia 2014. Published and unpublished data. Study conducted in primary healthcare setting ^bDowngraded two levels for very serious concerns regarding risk of bias: unclear random sequence generation, allocation concealment, and blinding of participants not feasible given the intervention, unclear blinding of outcomes and incomplete outcome reporting ^cDowngraded one level for imprecision: small sample size ^dFor this outcome, the number of study participants was based on a different study sample to the one used for the other outcomes. These data come from ongoing data (backend data in a digital system), and comprise of all the health workers who ever reported on stock levels ^eNon‐comparable results, thus downgraded to very low

Open in table viewer

Summary of findings 2. Primary objective: mobile stock notification with effective product transport compared to standard care

Mobile stock notification (cStock) with effective product transport (EPT) compared to standard care in primary healthcare settings
Patient or population: healthcare workers and other health professionals involved in commodity and stock management Setting: primary healthcare settings in Malawi Intervention: mobile stock notification with effective product transport (cStock + EPT), which involved providing health surveillance assistants (HSA) with training and tools for bicycle maintenance Comparison: standard care, involved routine stock management with mobile stock notification or any other digital intervention
Outcomes	*Illustrative comparative risks (95% CI)**		Relative effect (95% CI)	No of Participants (studies)	Certainty of the evidence (GRADE)	Results in words
Outcomes	Assumed risk with standard care	Corresponding risk with cStock + EPT	Relative effect (95% CI)	No of Participants (studies)	Certainty of the evidence (GRADE)	Results in words
Availability of commodities
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of cotrimoxazole to treat bacterial infections)	167 per 1000 healthcare workers	218 per 1000 healthcare workers (117 to 407)	RR 1.31 (0.70 to 2.44)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of cotrimoxazole because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of artemether‐lumefantrine 2 X 6 to treat malaria caused by Plasmodium facilparum)	189 per 1000 healthcare workers	270 per 1000 healthcare workers (153 to 472)	RR 1.43 (0.81 to 2.50)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of artemether‐lumefantrine because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of oral rehydration salts (ORS) to treat dehydration)	211 per 1000 healthcare workers	129 per 1000 healthcare workers (63 to 260)	RR 0.61 (0.30 to 1.23)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of oral rehydration salts because it is supported by very low‐certainty evidence
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of zinc 20 mg to treat diarrhoea)	256 per 1000 healthcare workers	281 per 1000 healthcare workers (171 to 465)	RR 1.10 (0.67 to 1.82)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of zinc because it is supported by very low‐certainty evidence
Quality of stock management
Quality of data about stock management (assessed by the extent to which HSAs (intervention group participants) send messages about the stocks on hand for all the products they managed)	In the intervention group, an average of 65% (N = 253) of the health surveillance assistants (HSAs) who managed relevant medicines reported on stock levels. This outcome was not assessed in the comparison group.			253^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on quality of data about stock management because it is supported by very low‐certainty evidence
Timeliness of stock management
Time between stock‐level reporting and appropriate action (Measured over an 18‐month period: January 2012 to June 2013)	In the intervention group, health facilities took an average of 26 days to fulfil an order requested by the health surveillance assistants (lead time). This outcome was not assessed in the comparison group.			253^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on the timeliness of stock management because it is supported by very low‐certainty evidence
Satisfaction and acceptability
Provider acceptability and satisfaction (Proportion of participants who reported using the digital intervention)	In the intervention group, the proportion of participants who reported using the digital intervention (cStock) as the primary means for ordering health products was 91% (N = 78). This outcome was not assessed in the comparison group.			78^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on provider satisfaction with stock management because it is supported by very low‐certainty evidence
Resource use
No studies were identified that reported on this outcome
Unintended consequences
No studies were identified that reported on this outcome
The basis for the assumed risk (e.g. the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). 95% CI: 95% confidence interval; RR: risk ratio; CBA: controlled before‐after trial
GRADE Working Group grades of evidence High certainty. Further research is very unlikely to change our confidence in the estimate of effect. Moderate certainty. Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low certainty. Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low certainty. We are very uncertain about the estimate.
^aShieshia 2014 published and unpublished data. Study conducted in primary healthcare setting. ^bDowngraded two levels for very serious risk of bias concerns: unclear random sequence generation, allocation concealment, and blinding of participants not feasible given the intervention, unclear blinding of outcomes and incomplete outcome reporting ^cDowngraded one step for imprecision: small sample size ^dFor this outcome, the number of study participants is based on a different study sample to the one used for the other outcomes. These data come from ongoing data (backend data in a digital system), and comprise all the health workers who ever reported on stock levels.

Open in table viewer

Summary of findings 3. Secondary objective: summary of findings

Summary of qualitative findings for the secondary objectives^a
	Summary of review finding	Studies contributing to the review finding	Overall GRADE‐CERQual assessment of confidence in the evidence	Explanation of GRADE‐CERQual assessment^b
1	Infrastructural issues, such as challenges in charging phones, uploading and transmitting data, and loss of data due to poor access to electricity and poor or non‐existent internet connectivity were identified as key barriers to implementation.	Negandhi 2016 Asiimwe 2011; Atnafu 2017; Biemba 2017; Negandhi 2016; Shieshia 2014; Stanton 2016; USAID 2010 Atnafu 2017	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
2	Concern was expressed that digital stock notification systems used at the facility level could not mitigate several, broader health system problems, including an underlying lack of stock at the national or district level, and a mismatch between national ordering routines and local needs.	Chandani 2017; Githinji 2013; Hara 2017; Mikkelsen‐Lopez 2014	Low confidence	Reduced due to serious concerns regarding methodological limitations, and minor concerns regarding adequacy
3	Programmes could benefit from strong partnerships, such as with local authorities; with local telecommunications companies; with technical system providers; and with non‐governmental organizations (NGOs).	WHO 2013	Very low confidence	Reduced due to serious concerns regarding methodological limitations, and serious concerns regarding adequacy
4	The availability and use of data on stock levels at all levels of the health system allowed health care officials to respond to anticipated shortages.	Asiimwe 2011; Barron 2016; Biemba 2017; Shieshia 2014; Stanton 2016	Low confidence	Reduced due to serious concerns regarding methodological limitations, and concerns regarding coherence
5	Supportive supervision and responsive management played an important role in effective adoption of a digital system.	Asiimwe 2011; Barrington 2010; Chandani 2017; Negandhi 2016; Shieshia 2014 Asiimwe 2011	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
6	The extent to which health workers are familiar with smartphones and are given adequate training in using the digital system, influences the adoption of the system.	Asiimwe 2011; Barrington 2010; Githinji 2013; Negandhi 2016; Shieshia 2014; Stanton 2016	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
7	The availability of technical programming expertise for the initial development and ongoing maintenance of the digital system is an important implementation factor.	Asiimwe 2011; Biemba 2017; USAID 2010 Biemba 2017	Low confidence	Reduced due to serious concerns regarding methodological limitations, and concerns regarding adequacy
8	Incentives, such as receiving phone talk‐time credit, to improve adoption and use of the digital system are valuable.	Barrington 2010	Low confidence	Reduced due to serious concerns regarding methodological limitations, and concerns regarding adequacy
9	User‐friendly systems, built with user participation with easy‐to‐use interfaces were considered important to implementation.	Namisango 2016; Negandhi 2016; Shieshia 2014 Namisango 2016	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
10	The use of basic mobile phones or personal phones by health workers reduced challenges with data coverage and expense, and supported easier adoption of the intervention due to familiarity with the phones.	Barrington 2010; Stanton 2016	Low confidence	Reduced due to serious concerns regarding methodological limitations, and minor concerns regarding adequacy
11	Software features, such as ability to capture images, map geographic features, support two‐way communication, toll‐free text messaging, and interoperability were considered important.	Barrington 2010; Negandhi 2016; Shieshia 2014	Low confidence	Reduced due to serious concerns regarding methodological limitations, and minor concerns regarding adequacy
12	Dashboard design and data visualisation played important roles in effective implementation. Managers should have access to data in an easy‐to‐use format, such as an interactive dashboard.	Negandhi 2016; Shieshia 2014; USAID 2010	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
^aThe study authors referred to several factors that may influence the implementation, uptake, or efficient use of interventions targeted at improving stock management ^bThe GRADE‐CERQual evidence profile for each finding is available in Table 1

Open in table viewer

1. GRADE‐CERQual evidence profile

Summary of review finding	Studies contributing to the review finding	Methodological limitations	Coherence	Adequacy	Relevance	GRADE‐CERQual assessment of confidence in the evidence	Explanation of GRADE‐CERQual assessment
1. Infrastructural issues, such as challenges in charging phones, uploading and transmitting data, and loss of data due to poor access to electricity and poor or non‐existent internet connectivity were identified as key barriers to implementation.	Asiimwe 2011; Atnafu 2017; Biemba 2017; Negandhi 2016; Shieshia 2014; Stanton 2016; USAID 2016	Serious concerns, because 6 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to methodological limitations, as all source material did not include empirical data.
2. Concern was expressed that digital stock notification systems used at the facility level could not mitigate several, broader health system problems, including an underlying lack of stock at the national or district level, and a mismatch between national ordering routines and local needs.	Chandani 2017; Githinji 2013; Hara 2017; Mikkelsen‐Lopez 2014	Serious concerns, because 3 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	Minor concerns about adequacy, as 3 studies refer to the same intervention administered in different settings	No, or very minor concerns about relevance	Low confidence	Reduced due to methodological limitations, and concerns about adequacy, as conclusions are based on few studies.
3. Programmes could benefit from strong partnerships, such as with local authorities; with local telecommunications companies; with technical system providers; and with non‐governmental organizations (NGOs).	WHO 2013	Serious concerns, because one study had serious methodological limitations (insufficient evidence to support findings, and unclear description of the intervention)	No, or very minor concerns about coherence	Concerns about adequacy, as only one study contributed to the finding	No, or very minor concerns about relevance	Very low confidence	Reduced due to methodological limitation, as source material did not include empirical data, and concerns about adequacy, as conclusions are based on one study with thin data
4. The availability and use of data on stock levels at all levels of the health system allowed health care officials to respond to anticipated shortages.	Asiimwe 2011; Barron 2016; Biemba 2017; Shieshia 2014; Stanton 2016	Serious concerns, because 4 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	Concerns about coherence due to contradictory findings	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Low confidence	Reduced due to methodological limitations, and concerns about coherence of the data.
5. Supportive supervision and responsive management played an important role in effective adoption of a digital system.	Asiimwe 2011; Barrington 2010; Chandani 2017; Negandhi 2016; Shieshia 2014	Serious concerns, because 4 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations
6. The extent to which health workers are familiar with smartphones and are given adequate training in using the digital system, influences the adoption of the system.	Asiimwe 2011; Barrington 2010; Githinji 2013; Negandhi 2016; Shieshia 2014; Stanton 2016	Serious concerns, because 5 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations.
7. The availability of technical programming expertise for the initial development and ongoing maintenance of the digital system is an important implementation factor.	Asiimwe 2011; Biemba 2017; USAID 2016	Serious concerns, as 3 studies had serious methodological limitations (insufficient evidence to support findings)	No, or very minor concerns about coherence	Concerns about adequacy, as only 3 studies contributed to the finding, and the presented data are sparse.	No, or very minor concerns about relevance	Low confidence	Reduced due to concerns about methodological limitations and adequacy, as conclusions are based on three studies with thin data.
8. Incentives, such as receiving phone talk‐time credit, to improve adoption and use of the digital system are valuable.	Barrington 2010	Serious concerns, because 1 study had serious methodological limitations (insufficient evidence to support findings)	No, or very minor concerns about coherence	Concerns about adequacy, as only 1 study contributed to the finding	No, or very minor concerns about relevance	Low confidence	Due to methodological limitations and concerns about adequacy, as conclusions are based on few studies.
9. User‐friendly systems, built with user participation with easy‐to‐use interfaces were considered important to implementation.	Namisango 2016; Negandhi 2016; Shieshia 2014	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations.
10. The use of basic mobile phones or personal phones by health workers reduced challenges with data coverage and expense, and supported easier adoption of the intervention due to familiarity with the phones.	Barrington 2010; Stanton 2016	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings)	No, or very minor concerns about coherence	Concerns about adequacy, as only 2 studies contributed to the finding	No, or very minor concerns about relevance	Low confidence	Reduced due to methodological limitations, and concerns about adequacy, as conclusions are based on few studies.
11. Software features, such as ability to capture images, map geographic features, support two‐way communication, toll‐free text messaging, and interoperability were considered important.	Barrington 2010; Negandhi 2016; Shieshia 2014	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	Minor concerns about adequacy, due to few studies and the relevant data are sparse.	No, or very minor concerns about relevance	Low confidence	Due to concerns about methodological limitations, and concerns about adequacy, as conclusions are based on few studies.
12. Dashboard design and data visualisation played important roles in effective implementation. Managers should have access to data in an easy‐to‐use format, such as an interactive dashboard.	Negandhi 2016; Shieshia 2014; USAID 2016	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations.

Background

Access to medicines and other health commodities remains one of the most serious global public health problems and results in critical gaps in delivery of healthcare services.

Description of the condition

Reliable availability of health commodities is fundamental to diagnosing and treating illnesses in primary healthcare settings. Health commodities include health products, health and medical supplies, and other items that may be needed for the provision of health services, including medicines; vaccines; medical supplies, such as contraceptives, dressings, needles, and syringes; and laboratory and diagnostic consumables (Tran 2015; WHO 2017). The World Health Organization (WHO) Global Strategy for Women’s and Children’s Health highlights the importance of equitable access to life‐saving medicines and other health commodities (WHO 2010). A hallmark of functioning health systems is the availability of essential medicines in adequate amounts, appropriate dosage forms, and assured quality – at a price that is affordable for the local community (Tran 2015; WHO 2016). However, stockouts of critical medical commodities, such as medicines, are widespread, especially in low‐ and middle‐income countries (LMICs).

At least one third of the world’s population does not have regular access to medicines, which makes health care highly inequitable (WHO 2011). A survey of the national AIDS programmes in 12 countries, by the Pan American Health Organization (PAHO), found that between January 2011 and April 2012, over 67% of the countries reported experiencing stockouts of at least one drug, lasting an average of 40 days each (Sued 2011). Another study, in Kenya, reported that over 75% of health facilities had shortages of one component of the combination of drugs used to treat malaria, while one in four reported a lack of all related drugs (Kangwana 2009). This lack of access to critical drugs, caused by a stockout, has profound effects on the ongoing treatment of diseases. A study in Côte d’Ivoire reported that people who experienced interruptions in their HIV treatment, caused by drug shortages, were twice as likely to permanently discontinue treatment or die (Pasquet 2010). Drug stockouts have been linked to increases in morbidity and mortality across several disease states in low‐resource settings, including malaria (Chuma 2010), HIV (Pasquet 2010), and the prevention or treatment of pregnancy complications (Hill 2006).

Lack of access to medicines and other health commodities is often symptomatic of broader systemic problems. For example, access to medicines is determined by rational use of medicines, affordable pricing, sustainable financing, and reliable health and supply systems (WHO 2004; WHO 2015). A reliable medicine supply system should include appropriate procurement and distribution. A good distribution system ensures timely availability of medicines across all levels of the healthcare system and prevention of stockouts (WHO 2017).

Description of the intervention

The rapid global expansion of mobile technology has provided a potential low‐cost solution to the challenge of drug distribution and stockouts. Plummeting costs of mobile handsets and services have made mobile phone technology accessible to people living in rural and underserved areas. Mobile interventions may address stockouts of medicines and health commodities primarily through two strategies: supply chain management, and assessment and reporting of essential commodities (Mehl 2017 [pers comm]).

Supply chain management involves approaches for monitoring and reporting stock levels, consumption and distribution of medical commodities, as well as approaches to analyse and project usage of medical commodities. This can include the use of communication systems, such as short message service (SMS) and data dashboards, to manage and report on supply levels of medical commodities. Some specific examples where mobile tools may be used to improve supply chain management include tracking inventory of health commodities, notifying stock levels of health commodities, monitoring cold‐chain sensitive commodities, and managing distribution of health commodities.

Digital approaches for assessment and reporting of essential commodities are often used for reporting and tracking the authenticity and quality of medical commodities. This can include using mobile functions, such as barcode readers and short message service (SMS) communication to validate an authentication code on the drug packaging (Frøen 2016), as well as to report on adverse drug effects. Specific examples where mobile devices may be used for assessment and reporting of commodities include reporting on stock levels, reporting counterfeit or substandard drugs, reporting adverse drug interactions, and registering licensed drugs and health commodities.

How the intervention might work

Mobile devices are being used for supply chain management to improve data visibility, improve decision‐making, and help to address the availability of commodities. There is a substantial amount of variation in how such systems might operate. At the most basic level, interventions may involve citizens reporting counterfeit medicines, using SMS sent to a toll‐free phone number. Such interventions may use a mobile product authentication (MPA) application, or a barcode that allows consumers to text a set of unique numbers to a toll‐free phone number, to verify if a medicine is authentic. In response, consumers may receive a SMS that indicates the legitimacy of the medicine.

Other interventions may involve frontline healthcare workers or healthcare administrators in primary healthcare settings using mobile devices to collect data on stock levels, so that data can be instantly digitised and used to predict and prevent stockouts, and respond to drug shortages. More comprehensive interventions may aim to develop a technology‐based system for reporting of drug stock levels, and change the culture around the use of data (on stock levels), and accountability for responding to projected shortages. For example, cStock is an open‐source internet‐accessible logistics management information system that targets the availability of health commodities at the community level, in Malawi (Shieshia 2014). Health surveillance assistants (HSAs), who typically deliver primary healthcare services in the community, send information about the amount of medicine stocks they have on hand, via a text message to a toll‐free number. The logistics management system automatically collates this data from multiple HSAs, calculates the total quantities of commodities needed, and sends a text message to the HSAs when the medicines are available at the nearest health centre. These data are also available on a internet‐accessible dashboard, with simple, easy‐to‐use reports, showing stock levels, HSA reporting rates, and alerts from central and district level health managers. Making real‐time data available, with regular reports of stock levels, allows managers and other stakeholders to co‐ordinate, plan, and identify solutions to better meet community needs in a timely manner.

Why it is important to do this review

There is rapid progress in the use of mobile devices to address systemic challenges in the delivery of healthcare services. Despite the exponential growth of mobile device‐based interventions and their potential, there remain several unanswered questions about the effectiveness of such interventions. The reliable availability of essential medicines and health commodities is foundational to a responsive health system, and an area that is of much interest to governments, especially in LMICs. However, the evidence on the use of mobile devices to address drug and commodity shortages is sparse, and offers no clear way forward. We are not aware of any existing systematic reviews that assess the effectiveness of strategies to improve stock notification, through either digital or non‐digital approaches. The WHO recently published guidelines to inform investments in digital health applications for strengthening health systems (WHO 2019). Through a consultative process, assessing the impact of mobile interventions to address stockouts was identified as one of the several areas to be included in the guidelines. This Cochrane Review is one of a suite of reviews that contributed to these guidelines. We aimed to assess the effectiveness of using mobile devices to address stockouts of drugs and essential health commodities, and the acceptability, resource use, and unintended consequences of such interventions.

Objectives

Primary

To assess the effects of strategies for notifying stock levels and digitally tracking healthcare‐related commodities and inventory, via mobile devices, across the primary healthcare system.

Secondary

To describe what mobile device strategies are currently being used to improve reporting and digital tracking of health commodities;
To identify factors influencing the implementation of mobile device interventions targeted at reducing stockouts of health commodities.

Methods

Criteria for considering studies for this review

Types of studies

Primary objective

For the review’s primary objective, we included these study designs:

Randomised trials;
Non‐randomised trials;
Controlled before‐after studies, provided they had at least two intervention sites and two control sites;
Interrupted time series studies, if there was a clearly defined point in time when the intervention occurred, and at least three data points before and three after the intervention.

We included published studies, conference abstracts, and unpublished data. We included studies, regardless of their publication status, or language of publication.

Secondary objectives

For the review's secondary objectives, we included any studies that used descriptive, qualitative, or quantitative methods to describe interventions that were aimed at improving stockouts of health commodities.

Types of participants

For the review's primary and secondary objectives, we included studies with these participants:

All cadres of healthcare providers (i.e. professionals, para‐professionals, and lay health workers), or others involved in the distribution of health commodities, located at any level of the health system (e.g. administrative staff, managerial and supervisory staff in purchasing or distribution, or dispensary staff);
Other individuals or groups involved in stock notification, monitoring, and tracking commodity inventories. These individuals or groups may be based in a primary healthcare facility or in the community, and must be involved in supporting the delivery of primary healthcare services.
Clients or recipients of health services

Types of interventions

For the review's primary and secondary objectives, we included interventions that were aimed at improving the availability of health commodities, including medicines and other medical supplies, using mobile devices for the delivery of primary healthcare services in healthcare facilities or in the community, if they involved one or both of the following:

Strategies for tracking health commodity inventory using mobile devices. Tracking health commodity inventory may have involved the use of databases and dashboards to manage the availability of health commodities and project availability of medical supplies. While some aspects of commodity tracking might have involved mobile devices, the data may have been linked to a logistics management information system (LMIS) or supply chain management system, where inventory levels and historic data were maintained on desktops;
Notification of stock levels conducted via mobile devices. This may have involved the transmission of information on stock levels by health workers within healthcare facilities or by members of the community, to alert higher‐level facilities about potential stock shortages. For example, health workers at facilities or dispensaries may have used text messaging, short message service (SMSa), or unstructured supplementary service data (USSD)‐based systems to notify district or central authorities about stock levels. In some interventions of interest, notification of stock levels using mobile phones may have been a component of a broader strategy for tracking health commodities.

By mobile devices, we meant mobile phones of any kind (but not analogue landline telephones), tablets, personal digital assistants, and smartphones.

By primary healthcare services, we meant a combination of the following:

The first contact point of healthcare (Awofeso 2004), including care delivered at an individual or community level, or both, by individual healthcare providers or teams of providers, and intended to bring care to where people worked and lived (Muldoon 2006), or the co‐ordination or provision of continuity of care, or both (WHO 2008);
Any rehabilitative, therapeutic, preventive, or promotional healthcare (Global Health Watch 2011).

The key comparison for this review was tracking commodity inventory and notifying stock levels via mobile devices compared with standard practice (i.e. non‐digital strategies or no intervention).

We excluded:

Studies that focused on cold chain management only, and did not report on stock levels of the vaccines;
Studies where commodity tracking and notification of commodities was conducted on stationary computers or laptops only.

Where tracking or notification via mobile device, or both, was delivered as part of a wider package, we included the study if we judged the mobile component to be the major component of the intervention.

Types of outcome measures

Primary objective

For the review's primary objective, we included studies that assessed the following outcome measures:

Availability of commodities, measured, for instance, as decreased stockouts, lead time for drug supply, availability at point of care;
Quality of data about stock management (accuracy of data, completeness of data);
Timeliness of stock level reporting, and time between receipt and reporting data regarding commodity status and appropriate action;
Provider acceptability or satisfaction with the intervention, measured with a validated scale if available;
Resource use (e.g. human resources or time, including additional time spent by providers when managing or transitioning dual paper and digital reporting systems; training, supplies, and equipment);
Unintended consequences that may result in the intervention having adverse effects (these could include: misreading or misinterpreting the data; transmitting inaccurate data, for instance through software formatting errors; interrupted workflow due to infrastructure constraints for battery recharge and network coverage; decreased motivation or trust in the system by health workers, if stock replenishment is not reliable; loss or misuse of mobile device).

Search methods for identification of studies

We started the search in 2000. This was based on the increased availability and penetration of mobile devices in LMICs from 2000 onwards (ITU 2015).

Electronic searches

An independent Information Specialist (JE) developed the search strategies in consultation with the review authors.

We searched the following databases for primary studies, from 2000 to the date of search:

Cochrane Central Register of Controlled Trials (CENTRAL; 2019, Issue 8), in the Cochrane Library (searched 7 August 2019);
MEDLINE Ovid and Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations and Daily (1946 to 7 August 2019);
Embase Ovid (1974 to 2019 Week 31);
POPLINE, K4Health (searched 7 August 2019);
Global Index Medicus, World Health Organization (WHO; searched 7 August 2019).

Appendix 1 lists the search strategies we used to search all the databases. Search strategies were comprised of keywords and controlled vocabulary terms. We did not apply any limits on language.

Searching other resources

Trial registries

We searched for ongoing trials in the following trial registries, and contacted authors for further information and data, if available:

WHO International Clinical Trials Registry Platform (ICTRP; www.who.int/ictrp; searched 7 August 2019);
US National Institutes of Health Ongoing Trials Register ClinicalTrials.gov (www.clinicaltrials.gov; searched 7 August 2019).

We searched Epistemonikos (www.epistemonikos.org; searched 27 January 2020) for relevant systematic reviews and potentially eligible primary studies. Additionally, the WHO issued a call for papers through popular digital health communities of practice, such as the Global Digital Health Network, to identify additional primary studies and grey literature.

Grey literature

We searched www.mhealthevidence.org for grey literature. The search portal for mhealthevidence.org was more limited; therefore, we reviewed the titles and abstracts of all contributed literature that was not referenced in MEDLINE Ovid (searched 15 August 2017; the database was discontinued in 2018).

We reviewed reference lists of all included studies and relevant systematic reviews for additional, potentially eligible primary studies. We also conducted a citation search for studies that had cited any included studies (searched 27 January 2020). We contacted authors of included studies and reviews to clarify reported published information, and to seek unpublished results and data.

Data collection and analysis

Selection of studies

A core team of two authors (NH and HB), with assistance where necessary from one additional review author (SA), were responsible for the selection of studies. We downloaded all titles and abstracts retrieved by electronic searching to a reference management database and removed duplicates. Two review authors (NH and HB) independently screened titles and abstracts for inclusion for the primary and secondary objectives. We retrieved the full‐text study reports and publications for titles and abstracts that were assessed as potentially eligible. Two review authors (NH and HB) for the primary objective, and one review author for the secondary objectives (SA), independently screened the full‐text, and identified studies for inclusion, and recorded reasons for exclusion of the ineligible studies. We resolved any disagreement through discussion; if required, we consulted a third review author.

We listed studies that initially appeared to meet the inclusion criteria but that we later excluded in the 'Characteristics of excluded studies' table. We collated multiple reports of the same study, so that each study, rather than each report, was the unit of interest in the review. We also provided any information we obtained about ongoing studies. We recorded the selection process in sufficient detail to complete a PRISMA flow diagram (Liberati 2009).

Data extraction and management

We modified the Cochrane Effective Practice and Organisation of Care (EPOC) standard data collection form and adapted it for study characteristics and outcome data (EPOC 2017a). We identified key characteristics of the intervention for abstraction based on the mHealth Evidence Review and Assessment (mERA) guidelines (Agarwal 2016). We piloted the form on one study in the review.

Primary objective

Two review authors (NH and HB) independently extracted the following study characteristics from the studies that were included for the primary objective:

general information: title, reference details, author contact details, publication type, funding source, conflicts of interest of study authors;
population and setting: country, geographical location (rural, urban, peri‐urban), healthcare setting (e.g. facility‐based, community‐based);
methods: function of the intervention, study design, unit of allocation, study duration;
participant characteristics: type of user (role, if in the health system; length of training, if any), description of any other participants in the intervention, withdrawals;
interventions: intervention purpose, components, infrastructure to support the technology, type of technology (software platform), type of mobile device(s) used (smartphone, tablets with a screen size larger than 7 inches, feature phones that can run java applications, basic phone with SMS and call functions, laptops), mode of delivery, content of the intervention, participant and provider training, interoperability, compliance with national guidelines, data security, comparison, fidelity assessment, duration of intervention;
outcomes: primary and other outcomes specified and collected, time points reported, adverse events, results of any subgroup analyses.

We noted in the 'Characteristics of included studies' table if outcome data were reported in a way that was not usable.

Secondary objective

For the secondary objectives, we extracted all the information listed above, if available, to describe the intervention. To understand factors affecting the implementation of relevant interventions, we had planned to use the Supporting the Use of Research Evidence (SURE) framework (SURE 2011; Glenton 2017); however, we found that the themes identified in the framework did not apply well to the contents of the included studies. We also explored the use of other implementation research frameworks, such as the consolidated framework for implementation research (CFIR; (Damschroder 2015), but found minimal overlap between the themes identified in the data. Therefore, we read and re‐read the included studies to identify new codes to tag the abstracted data.

Assessment of risk of bias in included studies

Assessment of risk of bias in included study for the primary objective

For the primary objective, two review authors (NH and HB) independently assessed the risk of bias for the included study, using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions Section 8.5 (Higgins 2011), and guidance from the Cochrane EPOC group (EPOC 2017b). We assessed risk of bias for the included controlled before‐after study using the following criteria: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, baseline outcomes measurement, similarity of baseline characteristics, and other bias.

We judged each potential source of bias as either high, low, or unclear, and provided a quote from the study together with a justification for our judgment, in Table 2. We considered blinding separately for different key outcomes where necessary (e.g. for unblinded outcome assessment, risk of bias for all‐cause mortality may be very different than for a participant‐reported pain scale). When considering treatment effects, we took into account the risk of bias for the study that contributed to that outcome.

Open in table viewer

Table 2. Risk of bias in the included study for the primary objective (Shieshia 2014)

Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"randomly assigned", but method of sequence generation not reported
Allocation concealment (selection bias)	Unclear risk	Not reported
Blinding of participants and personnel (performance bias)	High risk	Blinding was not possible due to the nature of the intervention
Blinding of outcome assessment (detection bias)	Unclear risk	Not reported
Incomplete outcome data (attrition bias)	Unclear risk	Different sample sizes for different outcomes; more participants were analysed at follow‐up than at baseline; and only a sample of participants were included in the analysis. It was unclear whether the researchers were able to collect data from all respondents.
Selective reporting (reporting bias)	Low risk	All outcomes in the protocol were reported in the published results
Other bias	Low risk	No other bias identified

Assessment of methodological limitations of included studies for the secondary objectives

For the secondary objectives, the included studies comprised a multitude of study designs and study aims, including case studies that were primarily descriptive. We were unable to find an accepted tool designed to appraise methodological limitations that could accommodate this variation in study design. Therefore, we piloted a newly developed tool for assessing the methodological limitations of sources, such as programme reports, that do not use typical empirical research designs. Two review authors (SA and CG) independently assessed the methodological limitations of the studies using the Ways of Evaluating Important and Relevant Data (WEIRD) tool (Lewin 2019). The tool, which is currently being piloted in EPOC and other systematic reviews, is available in Appendix 2.

For each item and question in the tool, the review author selected one of the following response options:

Yes – the item was addressed adequately in the source
Unclear – it is not clear if the item was addressed adequately in the source
No – the item was not addressed adequately in the source
Not applicable – the item is not relevant to the source being assessed

The assessments for each WEIRD tool item for each relevant study are reported in Table 3.

Open in table viewer

Table 3. Methodological limitations of the included studies for the secondary objectivesa

Study ID	Clearly stated aim, objective or purpose?	Clear description of the source of the information (transparency)?	Clear description of the programme,intervention, policy or reform?	Clear description of the context/s?	Is the information accurate?	Is the evidence representative?	Any limitations of the information and/or methods discussed?	Is evidence provided to support any findings or conclusions made?	Relevant rights and ethics considerations described (empirical studies only)	Interests declared and any potential conflicts of interest noted?	Overall assessment^d
Asiimwe 2011	Unclear^b	Unclear	Yes	Yes	Unclear	Unclear	No	No	N/A	Unclear ‐ funding source declared but no conflict of interest declaration	Major limitations^e,f,g,h
Atnafu 2017	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes ‐ none to declare	No or few limitations^e
Barrington 2010	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ conflicts reported	Major limitations^e,g,h
Barron 2016	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h
Biemba 2017	Unclear^b	Yes	Yes	Yes	Yes	Unclear^c	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Chandani 2017	Yes	Yes	Yes	Yes	Yes	Unclear^c	Yes	Yes	Yes	No reporting on conflict of interest (however, a related study has reported on COI)	Minor limitations^h,i
Githinji 2013	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ conflicts reported	Major limitations^e,g,h
Hara 2017	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Mikkelsen‐Lopez 2014	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ conflicts reported	Major limitations^e,g,h
Namisango 2016	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Negandhi 2016	Yes	Yes	Yes	Yes	Yes	Unclear^c	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Shieshia 2014	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes ‐ none to declare	Minor limitations^e
Stanton 2016	Unclear^b	Yes	Yes	Yes	Yes	Yes	No	No	N/A	Yes ‐ none to declare	Major limitations^e,g
USAID 2010	No	No	No	No	No	No	No	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h,i,j
USAID 2016	No	No	No	No	No	No	No	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h,i,j
WHO 2013	No	No	No	No	No	No	No	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h,i,j

^aDetails of the WEIRD tool assessment criteria and prompts are available in Appendix 2

^bStated aim does not include assessing implementation factors

^cSmall sample size

^dNo or few limitations: when the answer to most questions in the tool is YES
Minor limitations: when the answer to most questions in the tool is YES or UNCLEAR
Significant / major limitations: when the answer to one or more questions in the tool is NO

Explanation of overall assessments

^eConcerns related to relevance of study aim to review objectives
^fConcerns related to source of information reported
^gConcerns related to evidence to support findings
^hConcerns related to lack of COI declaration
ⁱConcerns related to generalizability
^jConcerns related to description of source of information, aims, programme and context

Based on the assessments for each WEIRD tool item, we made an overall assessment of the methodological limitations of the source as follows:

Where the assessments for most items in the tool were 'yes' – no or few limitations
Where the assessments for most items in the tool were 'yes' or 'unclear' – minor limitations
Where the assessments for one or more questions in the tool were 'no' – major limitations

For each source, our assessment of whether most of the WEIRD tool items were addressed or not was a judgement. To make these judgements as explicit and transparent as possible, we have provided explanations of our reasoning in Table 3.

We then used the overall assessment for each source as part of the GRADE‐CERQual assessment of how much confidence to place in the findings for each secondary objective.

Measures of treatment effect

For the review's primary objective, we report pre‐intervention and post‐intervention means and proportions for the intervention and comparison groups, where possible. We estimated the effect of the intervention using risk ratios for dichotomous data, together with the appropriate associated 95% confidence interval (CI) and mean difference.

Unit of analysis issues

For the controlled before‐after studies included in the review, we had planned to report cluster adjusted risk ratios and their 95% CIs. However, the analysis of the one included cluster trial was not adjusted for clustering, and no intracluster correlation coefficient (ICC) was available (Shieshia 2014). Therefore, we presented the results without a measure of variance or precision of effect for outcomes for which there is a unit of analysis error (EPOC 2017c)

Dealing with missing data

We contacted investigators in order to verify key study characteristics and obtain missing outcome data where possible (e.g. when a study was identified as an abstract only).

Assessment of heterogeneity

We did not undertake a meta‐analysis, as we only included one study for the primary objective.

Assessment of reporting biases

We did not explore reporting bias statistically, as we only included one study for the primary objective.

Data synthesis

We presented a narrative overview of the findings, together with tabular summaries of extracted data, for the primary objective. We used Mantel‐Haenszel risk ratios to present results from dichotomous data, where sufficient data were available.

As part of the data synthesis, we had planned to explore how we could integrate the findings from our primary objective with those of the secondary objective. However, this was not feasible, as only one study was eligible for the primary objective and we assessed the findings from the primary objective to be of very low certainty.

For the secondary objectives, we had originally planned to use the SURE framework. However, we found that the themes identified in the framework did not apply well to the contents of the included studies. Therefore, we applied a thematic analysis approach. We read and re‐read the included studies, coded the data, and generated themes. We then identified common themes across all included studies, and consolidated themes where they had overlapping data, and divided themes further if the data captured disparate ideas. Thematic synthesis is a standard approach that has been used across several qualitative evidence summaries. We only reported themes emerging from the data; we did not apply any other organizing frameworks.

Once the review findings were completed, one author went through each finding, identified factors that may influence the implementation of the intervention, and developed prompts for future implementers. These prompts were reviewed by at least one other review author. These prompts are not intended to be recommendations, but instead, are phrased as questions to help implementers consider the implications of the review findings in their context. The questions are presented in the 'Implications for practice' section.

Subgroup analysis and investigation of heterogeneity

Sub‐group analysis was not possible as we only included one study for the primary objective, and it did not have data relevant to any planned sub‐group analyses.

Sensitivity analysis

We did not identify a sufficient number of studies to perform sensitivity analyses.

Summary of findings and assessment of the certainty of the evidence

We created 'Summary of findings' tables for the main intervention comparison(s) and included the most important outcomes in order to draw conclusions about the certainty of the evidence within the text of the review:

Availability of commodities (e.g. proportion of health workers or facilities reporting drug stockouts, time between stockout and availability of commodities);
Quality of data about stock management (e.g. accuracy of data, completeness of data);
Timeliness of stock‐level reporting;
Provider acceptability or satisfaction with the intervention.
Resource use
Unintended consequences

For the primary objective, two review authors independently assessed the certainty of the evidence (high, moderate, low, or very low) using the five GRADE considerations (risk of bias, consistency of effect, imprecision, indirectness, and publication bias) (Guyatt 2008). We used methods and recommendations described in Section 8.5 and Chapter 12 of the Cochrane Handbook for Systematic Reviews of interventions (Higgins 2011), and the Cochrane EPOC worksheets (EPOC 2017d), and used GRADEpro software (GRADEpro GDT). We provided justification for decisions to downgrade or upgrade the ratings using footnotes in the table. We used plain language statements to report these findings in the review (EPOC 2017e).

For the secondary objectives, two authors (SA, CG) used the GRADE‐CERQual approach to assess our confidence in each finding (Lewin 2018). GRADE‐CERQual assesses confidence in the evidence, based on the following four key components: methodological limitations of included studies; coherence of the review finding; adequacy of the data contributing to a review finding; and relevance of the included studies to the review question. After assessing each of the four components, we made a judgement about the overall confidence in the evidence supporting the review finding. We judged confidence as high, moderate, low, or very low. The final assessment was based on consensus among the review authors. The GRADE‐CERQual evidence profile tables supporting the assessment of confidence in each finding can be found in Table 1.

Results

Description of studies

Results of the search

We conducted a systematic literature search to August 2019. We identified a total of 4886 references after removing duplicates. We excluded 4778 references for the primary and secondary objectives, following a review of the titles and abstracts. We retrieved the full texts of 92 articles for the primary and secondary objectives for detailed eligibility assessment.

We included one study that fulfilled our inclusion criteria for the review’s primary objective (Shieshia 2014, published and unpublished data).

We included 16 papers that fulfilled our inclusion criteria for the review’s secondary objectives, including the one study that was also included for the primary objective.

We excluded 76 articles for reasons described in Figure 1. We did not identify any ongoing studies.

Figure 1

Study flow diagram

Included studies

Primary objective

We included one controlled before‐after study that met our inclusion criteria for the primary objective: to assess the effects of the intervention (Shieshia 2014). We determined that the study was a controlled before‐after study, based on our assessment of published and unpublished data. We present key characteristics of the included studies in the Characteristics of included studies table.

Secondary objective

We included sixteen studies that fulfilled our inclusion criteria for the review’s secondary objectives; 13 studies were peer‐reviewed articles and 5 were published reports. These described a total of 11 interventions targeted at stock notification and digital tracking of healthcare commodities. See Characteristics of included studies.

Several studies described interventions that were implemented in multiple countries. For example, one intervention called ‘SMS for Life’, described by four studies, was tested as a pilot in Tanzania (Barrington 2010; Mikkelsen‐Lopez 2014), and Kenya (Githinji 2013), and then rolled out at a larger scale across Tanzania, Kenya, Ghana, and Cameroon (WHO 2013). A few types of interventions, targeted at improving health commodity inventory, were administered under the DELIVER project. One type of intervention was described by two studies; one implemented in Zambia (USAID 2016), and the second in Tanzania, Ghana, and Liberia (USAID 2010). A study describing a second intervention was implemented in Ethiopia and Pakistan (Hara 2017). Two studies (and two additional unpublished documents) described variants of an intervention called ‘cStock’ in Malawi and Rwanda (Chandani 2017; Shieshia 2014). Two studies were implemented in Uganda (Asiimwe 2011; Namisango 2016), one in South Africa (Barron 2016), one in Ethiopia (Atnafu 2017), one in Zambia (Biemba 2017), one in India (Negandhi 2016), and one in Malawi and Ghana (Stanton 2016).

Interventions

The study that addressed the primary objective aimed to improve reporting, collation, and visibility of stock data. Shieshia 2014 describes two interventions, both with a common mobile web‐based reporting system called ‘cStock’. cStock is used for community–level reporting of stocks for 19 health products managed by health surveillance assistants (HSAs). In addition to cStock, one of the interventions included an enhanced management (EM) component, comprised of quality improvement teams that used data supplied by cStock, to monitor performance of the supply chain and make informed supply chain decisions (Comparison 1). The second intervention combined cStock with efficient product transport (EPT), which consisted of training all HSAs on bicycle maintenance, and providing a basic tool kit (Comparison 2).

The interventions described in all included studies for the primary and secondary objectives were targeted at notifying and managing stock levels of a range of drugs and commodities — cStock aimed to reduce stockouts of drugs for the Integrated Management of Childhood illnesses (IMCI; (Chandani 2017; Shieshia 2014)), and SMS for Life aimed to reduce stockouts of essential drugs for malaria treatment (Barrington 2010; Githinji 2013; Mikkelsen‐Lopez 2014; WHO 2013). Other interventions targeted effective vaccine management (Negandhi 2016), commodity and drug management for neglected tropical diseases (Stanton 2016), management of drugs for integrated community case management (Biemba 2017), palliative care drugs (Namisango 2016), rapid diagnostic tests and drugs for malaria (Asiimwe 2011), contraceptive products (Atnafu 2017), management of essential drugs and supplies (Barron 2016; USAID 2010; USAID 2016), and use of barcodes to improve procurement and supply coordination of health commodities (Hara 2017).

Outcomes

Shieshia 2014, included under the primary objective, reported on the availability of seven medicines for the treatment of childhood illnesses, timeliness of reporting on stock levels, and the acceptability of digital intervention to providers. Stock availability was measured in two ways: (1) Percentage of eligible HSA’s who reported stockout of required medicines on the day of visit; and (2) Percentage of eligible HSA’s who reported stockouts of specific medicines over the last 30 days. Timeliness of reporting on stock levels was only reported for the two intervention groups, and not for the comparison group. Acceptability of cStock was evaluated by looking at its level of routine use (e.g. HSA’s who used cStock as the primary means for ordering health products), and benefits perceived by the users. Again, these results were reported only for the two intervention groups.

All the studies included for the secondary objectives described the interventions targeted at reducing stockouts with varying levels of clarity. None of the studies aimed to formally assess the barriers and enables of implementation.

Excluded studies

For the primary objective, we excluded 77 articles after full‐text screening for one of the following reasons: the article did not meet the criteria for study design (N = 33); the intervention did not include a mobile device component (N = 1); the intervention did not directly target stock notification or tracking (N = 39); the intervention was not used by a healthcare provider (N = 3); or the intervention was not used in primary care (N = 1). Details of 15 potentially relevant studies, which were excluded, are provided in the 'Characteristics of excluded studies' table.

Risk of bias in included studies

Risk of bias in included studies for the primary objective

For the primary objective, we have presented the risk of bias assessments for the included study in Table 2 . For Shieshia 2014, intervention groups were purposively assigned, and owing to the nature of the intervention, blinding was not possible. The study had different sample sizes for the different outcomes assessed, and more participants were included in the analyses at follow‐up compared to the baseline. Methods for random sequence generation, allocation concealment, and blinding of outcome assessment were not described in the methods. Random sequence was generated by lottery among twelve socioeconomically and topographically comparable districts. Blinding was not feasible, as the intervention involved distribution of mobile phones. Outcome data were not reported for all participants.

Methodological limitations of included studies for the secondary objectives

For the secondary objectives, the included studies comprised a multitude of study designs. Some were case studies that described the development and components of the intervention (Biemba 2017; Hara 2017; Stanton 2016; USAID 2010; WHO 2013). One study measured the outcome of interest before and after the intervention, without a control group (Namisango 2016). Most of the studies used operational data to describe changes in key outcomes, as the interventions were rolled out over time, and without a control group (Asiimwe 2011; Barrington 2010; Barron 2016; Chandani 2017; Githinji 2013; Mikkelsen‐Lopez 2014; Negandhi 2016; USAID 2016). All but two of the studies included for the secondary objective had significant methodological limitations – they did not include empirical data, and provided unclear descriptions of the source of the information, with limited evidence to support their findings (Atnafu 2017; Shieshia 2014). These studies described the interventions, and the conclusions were typically drawn from authors’ experiences in implementing the intervention. We have reported our assessments for each WEIRD tool item and the overall assessment for each relevant study in Table 3.

Effects of interventions

See: Summary of findings 1 Primary objective: mobile stock notification with enhanced management compared to standard care; Summary of findings 2 Primary objective: mobile stock notification with effective product transport compared to standard care; Summary of findings 3 Secondary objective: summary of findings

Primary objective

We included one study that met our primary objective: to assess the effects of the intervention.

In Shieshia 2014, health surveillance assistants (HSAs) used their mobile phones for community‐level reporting of data about nineteen drugs and products through a structured SMS – a system referred to as cStock. This was combined with two additional intervention components. In the enhanced management group (EM), district product availability teams were trained to use the data, monitor performance, and make informed supply chain decisions (Comparison 1). In the Efficient Product Transport (EPT) group, HSAs received a toolkit and training on bicycle maintenance (Comparison 2).

Comparison 1: Mobile stock notifications with enhanced management (EM) compared to standard care

See summary of findings Table 1

Availability of commodities

We are uncertain of the effect of mobile stock notification with enhanced management on the availability of commodities (stockout of drugs in the last 30 days) compared to standard care (Shieshia 2014; very low‐certainty evidence). Stockout of drugs in the last 30 days was measured for cotrimaxazole to treat bacterial infections (Analysis 1.1), artemether‐lumefantrine to treat malaria caused by Plasmodium falciparum (Analysis 1.2; Analysis 1.3), oral rehydration drugs to treat dehydration (Analysis 1.4), and zinc to treat diarrhoea (Analysis 1.5).

Quality of stock management

Based on data from the intervention group only, we are uncertain of the effect of mobile stock notification with enhanced management on quality of stock management (Shieshia 2014; very low‐certainty evidence). Quality of data about stock management was assessed as the extent to which HSAs reported data about stocks that they had available. In the mobile stock notification with EM group, on average, 85% (N = 393) of the intervention group participants reported on stock levels for all the products that they managed.

Timeliness of stock management

Based on data from the intervention group only, we are uncertain of the effect of mobile stock notification with enhanced management on timeliness of stock management (Shieshia 2014; very low‐certainty evidence). The effect of the intervention on the timeliness of stock management was measured at the level of the health facilities. Health facilities in the stock notification with EM group took an average of 12.8 days to fill an order requested by healthcare providers.

Satisfaction and acceptability

Based on data from the intervention group only, we are uncertain of the effect of mobile stock notification with enhanced management on provider satisfaction (Shieshia 2014; very low‐certainty evidence). Provider satisfaction with the intervention was evaluated based on routine use. Ninety‐seven percent (N = 81) of HSAs in the stock notification with EM group reported using cStock as the primary means for ordering health products from the resupply point.

Resource Use

The included study did not report on the effect of the intervention on resource use.

Unintended consequences

The included study did not report on the effect of the intervention on unintended consequences.

Comparison 2: Mobile stock notification with efficient product transport (EPT) compared to standard care

See summary of findings Table 2

Availability of commodities

We are uncertain of the effect of mobile stock notification with EPT on the availability of commodities (stockout of drugs in the last 30 days) compared to standard care (Shieshia 2014; very low‐certainty evidence) stockout of drugs in the last 30 days was measured for cotrimaxazole to treat bacterial infections (Analysis 2.1), artemether‐lumefantrine to treat malaria caused by Plasmodium falciparum (Analysis 2.2; Analysis 2.3), oral rehydration drugs to treat dehydration (Analysis 2.4), and zinc to treat diarrhoea (Analysis 2.5).

Quality of stock management

Based on data from the intervention group only, we are uncertain of the effect of mobile stock notification with EPT on the quality of stock management (Shieshia 2014; very low‐certainty evidence). Quality of data was measured as the extent to which HSAs reported data about stocks that they had available. In the mobile stock notification with EPT group, on average, 65% (N = 253) of the HSAs reported on stock levels for all the products that they managed.

Timeliness of stock management

Based on data from the intervention group only, we are uncertain of the effect of mobile stock notification with EPT on the timeliness of stock management (Shieshia 2014; very low‐certainty evidence). Effect of the intervention on the timeliness of stock management was measured at the level of the health facilities. Health facilities in the stock notification with EPT group took an average of 26 days to fill an order requested by healthcare providers.

Satisfaction and Acceptability

Based on data from the intervention group only, we are uncertain of the effect of mobile stock notification with EPT on provider satisfaction (Shieshia 2014; very low‐certainty evidence). Provider satisfaction with the intervention was evaluated based on routine use. Ninety‐one percent (N = 78) of the HSAs in the stock notification with EPT group reported using cStock as the primary means for ordering health products from the resupply point.

Resource Use

The included study did not report on the effect of the intervention on resource use.

Unintended consequences

The included study did not report on the effect of the intervention on unintended consequences.

Secondary objectives

Current use of mobile strategies to improve reporting and digital tracking of health commodities

We included 16 studies that met the first of our secondary objectives; to describe how these types of mobile strategies are currently being used. These studies described eleven different interventions, all of which aimed to reduce stockouts. See Study characteristics for the secondary objectives in Characteristics of included studies.

Figure 2 summarises the key intervention components that we identified in eight of these eleven interventions. The overarching purpose of each of these interventions was to make data about stock levels and potential stockouts of commodities visible to managers, who could then take corrective action to address them. Each of these interventions required the use of a mobile device by healthcare workers, either to report stock levels as a text message (Asiimwe 2011; Barrington 2010; Chandani 2017; Githinji 2013; Mikkelsen‐Lopez 2014; Shieshia 2014; WHO 2013), or to collect data about stock levels, using a digital data collection form or an app (Atnafu 2017; Biemba 2017; Negandhi 2016; Stanton 2016; USAID 2010; USAID 2016). If a text message was used to report stock levels using the healthcare worker’s personal mobile phone, the messages were sent to a short code that was free of cost to the health care worker (Asiimwe 2011; Barrington 2010; Chandani 2017; Githinji 2013; Mikkelsen‐Lopez 2014; Shieshia 2014). These data were collected and aggregated on a central server, and made available on a webpage as an electronic database (Stanton 2016; USAID 2010; USAID 2016), in the form of an interactive dashboard (Asiimwe 2011; Barrington 2010; Chandani 2017; Githinji 2013; Mikkelsen‐Lopez 2014; Negandhi 2016; Shieshia 2014; WHO 2013), or as routine reports (Atnafu 2017; Biemba 2017). It was intended that this webpage be monitored by a higher level (e.g. district level) health official, who was responsible for taking corrective action to address extant or expected stock or commodity shortages. Some authors reported instituting active measures to ensure that the online data were used and responded to in a timely fashion. Shieshia 2014 described how additional staff, with clearly defined roles, were hired to monitor the data and respond to them. In ‘SMS for Life’, weekly summary reports were provided to district medical officers and pharmacists in addition to the dashboards, to support data use (Barrington 2010; Githinji 2013; Mikkelsen‐Lopez 2014; WHO 2013).

Figure 2

Common key intervention components of interventions targeted at reducing stock‐outs

The following features were salient to some of the eight interventions described in Figure 2.

Reminders to healthcare workers to send reports. Three of the eight interventions described a feature that sent reminders to healthcare workers to send their weekly stock‐level reports (Barrington 2010; Githinji 2013; Mikkelsen‐Lopez 2014; Shieshia 2014).
Incentives to health workers. A few studies described incorporating performance‐based incentives for the healthcare workers to send timely stock reports. For the implementation of SMS for Life, in Tanzania, Mikkelsen‐Lopez 2014 reported that if the healthcare worker sent a report on time, they received a phone credit of TZS 1000, or USD 0.70, for personal use. In Kenya, 50 Kenyan Shillings, or USD 0.6 worth of airtime was transferred to healthcare workers who sent timely stock reports (Githinji 2013).
Two‐way digital communication. In some of the interventions, there were checks in place to ensure that the messages sent by the healthcare workers were free of error. In the case of an error in the format of the message, the health worker would receive an error message, advising them to correct their message (Asiimwe 2011; Barrington 2010; Githinji 2013). When there were no errors, the healthcare worker received a confirmation message, stating that their weekly stock report had been received (Asiimwe 2011; Barrington 2010; Chandani 2017; Githinji 2013; Shieshia 2014).
Multi‐faceted interventions. Some of the interventions targeted at reducing stockouts were part of a broader multi‐faceted intervention. In addition to digital data reporting, aggregation, and visualisation, cStock invested in training district and central‐level staff to use computers to access web‐based dashboards for reporting, and quality improvement teams to use the data supplied by cStock to monitor performance of the supply chain and make informed supply‐chain decisions (Shieshia 2014). The LMS Suite, described by Stanton 2016, comprised three tools, each targeted at different aspects of managing cases of lymphatic filariasis, including: (1) 'MeasureSMS‐MDA' to support text message mass drug administration (MDA) and reporting of cases that had been treated for elephantiasis using appropriate anti‐parasitic medicines; (2) 'Measure‐SMS‐morbidity' to report on new cases of elephantiasis and their demographic information; and (3) 'Ly‐MSS' lymphedema management support system aimed to maintain the supply chain of care packages (such as washbasins, towels, soaps, antibacterial cream).

Three of the eleven interventions did not include the key components described in Figure 2. One of these interventions was MomConnect, described by Barron 2016. MomConnect is a national‐level service in South Africa that is targeted at connecting pregnant women to health services. Registration on MomConnect puts demographic and pregnancy‐specific information about women on an electronic database. Women then receive free informational messages, as per their stage of pregnancy, till their infant is one year old. Using this service, women can also interact with a help desk, located in the South African Department of Health (DoH) to (1) answer a brief survey about the quality of services received at the health facility, using unstructured supplementary service data (USSD) at no cost to them; (2) ask any question related to their pregnancy, using a text message; and (3) log a complaint or a compliment, using a free of cost text message. The complaints could be about commodity or medicine shortages during their last visit to the health facility, and are routinely monitored and addressed by the help desk.

The second intervention was the palliative care management software reported in Namisango 2016. This intervention included data collection, an electronic health record for the patient, and functions for supply chain management and provider work‐planning and scheduling. The health care provider could use a tablet computer to check their mobile application screen for which drugs were available in the pharmacy before prescribing the drug. Providers could enter patient and pharmacy data on the application. These data were linked to the database available to the pharmacist, who was responsible for tracking inventory based on the number of prescriptions written for a certain drug.

The third intervention was the use of bar codes to record and track health products reported in Hara 2017. They used an automatic identification and data capture system (AIDC), using barcodes based on global standard to improve end‐to‐end supply chain visibility of health commodities. The bar codes uniquely identified each health product and were linked to the batch and serial numbers of the products, and expiration dates. They described a process in which a smartphone application was used, in Ethiopia, to scan bar codes using the mobile phone’s camera, and push the captured data to a central logistics management software. This made real time tracking of commodities feasible, and built efficiencies in the system to streamline the availability of products, where needed.

Factors influencing implementation

We also used these 16 studies to address the second of our secondary objectives: to identify factors influencing the implementation of mobile interventions targeted at reducing stockouts of health commodities. All but one of these studies lacked empirical data, and clear descriptions of the source of the information, resulting in limited evidence to support the findings (Shieshia 2014). The studies described the interventions used, and the conclusions were typically drawn from the study authors’ experiences in implementing the intervention. We considered these limitations in our GRADE‐CERQual assessment of confidence in our findings. We also noted in the results that the perspective of the findings is, in general, that of the study authors.

The study authors referred to several factors that may influence the implementation, uptake, or efficient use of interventions targeted at improving stock management. In Figure 3, we grouped these factors under three categories: macro‐level factors that constitute a supporting ecosystem; programmatic factors associated with implementation; and factors directly pertaining to the technological component of the intervention. We also used these as a summary of findings for the secondary objective in summary of findings Table 3.

Figure 3

Enablers and barriers of implementation of interventions targeted at commodity stock‐outs

Supporting ecosystem

a. Infrastructure

Study authors identified several infrastructural issues that they suggested directly influence the implementation of mobile interventions, targeted at improving notification and tracking of commodities. These included problems with poor access to electricity, and poor or non‐existent internet connectivity, leading to challenges in charging phones, uploading and transmitting data, and loss of data (Asiimwe 2011; Atnafu 2017; Negandhi 2016; Shieshia 2014; Stanton 2016; USAID 2010; moderate‐certainty evidence; Finding 1; summary of findings Table 3).

About one‐fifth of all participants in one study reported challenges in sending text messages due to poor network (Shieshia 2014). Authors identified how some of these issues could be mitigated: using solar energy packs for charging phones (Namisango 2016), developing systems for working offline during internet outages (Namisango 2016), training health workers to resend their reports the next day, in case network coverage is not available initially (Asiimwe 2011), and taking steps upfront to ensure that network coverage is available within a few hours of the health facilities where the intervention is being implemented, so that reports can be submitted when health workers reach areas with better connectivity (Barrington 2010; Biemba 2017). However, health facilities that are responsible for receiving patient referral data from the community need to have regular connectivity (Biemba 2017).

b. Procurement and distribution systems

Study authors were concerned that digital stock notification systems used at the facility level could not mitigate several broader health system problems, including an underlying lack of stock at the national or district level, and a mismatch between national ordering routines and local needs (low‐certainty evidence; Finding 2; summary of findings Table 3).

Nationally instituted procurement and distribution systems (Chandani 2017), and inventory management information systems were identified as important considerations for implementation (Hara 2017). For example, the procurement of health commodities is influenced by donor policies. Mikkelsen‐Lopez 2014 reported that during the implementation of their intervention ‘SMS for Life’, the Tanzanian government had reduced its malaria budget significantly, because over half the malaria drugs were provided by donors. However, a two‐year delay in one of the donor funding cycles resulted in a national, critical, unanticipated drug shortage. Shortages at the national and regional levels cannot be mitigated by any corrective action taken at the district level. Such shortages also result in poor morale of managers (Githinji 2013).

Poor reconciliation between national and district drug procurement systems and the medicine ordering system makes it challenging to order the correct amount of drugs (Mikkelsen‐Lopez 2014). For example, in Tanzania, health facility drug orders are made quarterly, based on the patterns of the previous quarter. However, this does not account for the seasonality of diseases like malaria (Mikkelsen‐Lopez 2014).

c. Partnerships

Study authors described how programmes could benefit from strong partnerships, including with local authorities who could provide training for healthcare staff and district managers; with local telecommunications companies who could offer good rates for SMS transfer; with technical system providers who could support the development of the system; and with non‐governmental organizations (NGOs) who could support the roll‐out and training of the intervention (WHO 2013; very low‐certainty evidence; Finding 3; summary of findings Table 3).

Programmatic Support

a. Data use, supervision, and management

Study authors suggested that the availability and use of data on stock levels at all levels of the health system allowed healthcare officials to respond to anticipated shortages (low‐certainty evidence; Finding 4; summary of findings Table 3). The use of digital data on stock availability was one of the key features of all the interventions identified by this review. Barron and colleagues suggested that data visibility and use are important at all levels of the health system, from the district to the national level (Barron 2016). Quick sharing of data across health workers and facilities was found useful by healthcare workers in Ghana and Malawi (Stanton 2016).

Shieshia and colleagues reported that in addition to data dashboards, comprising of information on stock levels, being available online in a digital format, stock reports were printed routinely at the district level. These stock reports and health facility performance and challenges were discussed at district‐level meetings, allowing health workers across levels of the health system to become aware of the stock management procedures (Shieshia 2014). While most studies found the availability of stock data across levels of the health system to be useful, authors of one study highlighted a risk of making data accessible in real time across multiple levels of management. For instance, authors of a study in Uganda reported that district health officials are typically used to having greater control over the data that they report to the national level. With a digital system to track and report stock levels, data becomes simultaneously available to the district and national‐level stakeholders, and takes the opportunity away from the district officials to contextualise the data or explain shortcomings (Asiimwe 2011).

Authors emphasised the role of supportive supervision and responsive management for effective adoption of a digital system (moderate‐certainty evidence; Finding 5; summary of findings Table 3). Adequate supervision of the stock notification systems and associated data were identified as vital to successful implementation (Barrington 2010; Negandhi 2016; Shieshia 2014). For example, supervision of district level staff was needed to ensure that the data on stock levels were used, and appropriate corrective action was taken in a timely manner (Shieshia 2014). Routine visits and meetings of supervisors and healthcare workers can facilitate problem‐solving, and ensure timely communication to discuss any resupply procedures (Chandani 2017). In structuring the management, programmes should consider that health workers must be motivated to report the data, and supervisors must be motivated to use the data (Chandani 2017). Some study authors highlighted the importance of having well defined roles and responsibilities for the management staff (Asiimwe 2011; Barrington 2010; Shieshia 2014), and strict timelines for the roll‐out of the intervention, to further accountability (Barrington 2010). In some areas in Uganda, district health officials, who were expected to monitor stock levels and respond to them, only became involved in an ad‐hoc manner. The authors reported that this could be circumvented by having clearer roles defined by the Ministry of Health (Asiimwe 2011).

b. Knowledge, skills, and training

Some of these factors associated with implementation were tied to healthcare workers’ and staff members’ knowledge and skills, including the extent to which they were familiar with smartphones, comfortable using mobile data services (Stanton 2016), and the extent to which they were given adequate training in using the digital system (Negandhi 2016; Shieshia 2014; moderate‐certainty evidence; Finding 6; summary of findings Table 3).

Some authors highlighted the value of training to improve the uptake of the digital intervention, including training of the frontline health workers, such as Health Surveillance Assistants (HSA), who may be involved in sending stock reports (Shieshia 2014), training of support staff, such as cold chain technicians, who might be involved with stock management (Negandhi 2016), and facility workers, who count the stock levels (Barrington 2010; Githinji 2013). Authors suggested that staff should be trained in the composition and use of text messages (Asiimwe 2011; Barrington 2010; Githinji 2013). Training should also be provided to management staff at the national and district levels in the use of data and online dashboards (Barrington 2010; Githinji 2013).

c. Human resources and incentives

Authors identified that to support successful implementation of a digital intervention, it was important to have technical programming expertise available to develop and install the digital programme, and to maintain the system on an ongoing basis (Asiimwe 2011; USAID 2010). Asiimwe 2011 suggested that locally available expertise in software programming was important to responsively develop and test the mobile applications. Having ongoing technology support was important to address software bugs and other problems once the system was piloted (USAID 2010; low‐certainty evidence; Finding 7; summary of findings Table 3).

As discussed earlier, several studies offered performance‐based incentives of mobile phone credit to health workers, for timely reporting. While the value of such performance‐based incentives was not formally assessed, one study author reported that they found that incentives in the form of airtime credit to healthcare workers was helpful in encouraging timely SMS reporting of stock levels (Barrington 2010; low‐certainty evidence; Finding 8; summary of findings Table 3).

Technology inputs

a. Design of the digital systems

Several factors affecting implementation were tied to the design of the digital system, including the extent to which the systems were user‐friendly, with easy‐to‐use interfaces, and built with user participation (Namisango 2016; Negandhi 2016; Shieshia 2014), and the extent to which they were aligned with the country's existing health information reporting systems (Shieshia 2014). Two study authors emphasised the importance of iteratively designing the platform with user feedback and input to improve the acceptability and adoption of the digital intervention (Namisango 2016; Shieshia 2014; moderate‐certainty evidence; Finding 9; summary of findings Table 3).

b. Digital hardware and software

Study authors considered the use of basic mobile phones or personal phones by health workers to reduce challenges with data coverage and expense, and support easier adoption of the intervention due to familiarity with the phones (low‐certainty evidence; Finding 10; summary of findings Table 3). One study author suggested that programmes might consider using basic phones in lieu of android phones, as data network coverage is limited in remote locations, and data packages are prohibitively expensive (Stanton 2016). Another author suggested that having health workers use their personal mobile phones mitigates problems with phone maintenance, familiarity, and issues of ownership (Barrington 2010). Managers might be provided with a Blackberry or a similar device, so they can access dashboards, especially in places where desktop computer access is limited (Barrington 2010).

Study authors highlighted software features, such as ability to capture images, map geographic features, support two‐way communication, toll‐free text messaging, and interoperability (low‐certainty evidence; Finding 11; summary of findings Table 3). Software that had multiple features, such as the ability to capture images and map geographical locations, was amenable to programming, and could be used for different programmes was preferable (Negandhi 2016). Having a function for two‐way communication with the healthcare workers, either to confirm receipt of their stock reports, or to send them updates on stock availability, helped them to take necessary action, and supported morale (Shieshia 2014). One author emphasised the value of a toll‐free number for text messaging, so that health workers were not deterred by anticipated costs in sending text message on stock‐level updates (Barrington 2010). Negandhi and colleagues identified interoperability of the stock management systems as important for success (Negandhi 2016). The authors suggested that logistics management systems should be linked to health management systems, so that linkages could be made between supply and demand, which should in turn, could reduce waste.

c. Data visualisation

Several factors that influenced implementation were tied to the design of the dashboards, and data visualisation. Authors emphasised that healthcare mangers should have access to data in an easy‐to‐use format (Shieshia 2014), with an effective display of data using factsheets, and graphical and tabular illustrations (Negandhi 2016; moderate‐certainty evidence; Finding 12; summary of findings Table 3).

In order to accommodate this, Shieshia 2014 reported redesigning the dashboards several months after the system was set up, so that the users had a better understanding of the metrics and visuals, and could incorporate their experiences interacting with the system into the redesign. The management of data should be detail‐oriented, with regular reviews of the database (USAID 2010). To facilitate visualisation of data, healthcare personnel at other levels of the healthcare system also need access to functioning smart phones, laptops, or desktop computers (Biemba 2017).

Discussion

Summary of main results

Our review provides limited evidence on the primary objective, to assess the effect of tracking health commodity inventory and notifying stock levels via mobile devices on improvements in availability of commodities, quality of data about stock management, timeliness of stock‐level reporting, and provider acceptability. We identified one study, conducted in Malawi, that used a before‐after study design to answer these questions (Shieshia 2014). However, we are uncertain of the effect of these interventions on the outcomes of interest, because we assessed the certainty of this evidence as very low.

For the secondary objectives, we included 16 studies that described a total of eleven interventions. All studies were conducted in Africa (Tanzania, Kenya, Malawi, Ghana, Ethiopia, Cameroon, Zambia, Liberia, Uganda, South Africa, and Rwanda) and Asia (Pakistan and India). Most of the interventions aimed to make data about stock levels and potential stockouts visible to managers, who could then take corrective action to address them. We identified several factors that may influence the implementation of stock notification and tracking via mobile device. These included challenges tied to infrastructural issues, such as poor access to electricity or internet, and broader health systems issues, such as drug shortages at the national level, which could not be mitigated by interventions at the primary healthcare level. Several factors were identified as important, including strong partnerships with local authorities, telecommunication companies, technical system providers, and non‐governmental organizations; availability of stock‐level data at all levels of the health system; the role of supportive supervision and responsive management; familiarity and training of health workers in the use of the digital devices; availability of technical programming expertise for initial development and ongoing maintenance; incentives, such as phone credit, to support regular use of the system; easy‐to‐use systems built with user participation; use of basic or personal mobile phones to support easier adoption; consideration for software features, such as two‐way communication; and data availability in an easy‐to‐use format, such as an interactive dashboard.

Overall completeness and applicability of evidence

We identified insufficient high‐quality studies to address the primary objective of the review. Evidence was insufficient to recommend the use of mobile tools to track health commodity inventory and stock notification. We did not identify any data on use of resources for such interventions, or unintended consequences. Despite the proliferation of large scale, mobile‐based interventions to support stock notification and management, we did not identify any ongoing studies to address questions on the effectiveness of such interventions.

For the secondary study objective, the studies that described mobile interventions targeted at stock management had some common features, involving timely collection of stock data, visibility of stock data at different levels of the health system, and use and responsiveness to these data. Several implementation challenges that were identified by this review are consistent with the global evidence that points to general considerations for the implementation of digital interventions, including problems with network connectivity, access to electricity, device usability, and access to health worker training. Several 'best practices' were identified, based on the experiences of the study authors in implementing the interventions. Given the lack of empirical data from which these conclusions were drawn, and the high level of contextual and infrastructural variability within which such interventions might be implemented, these findings have limited external validity, and should be cautiously interpreted.

Quality of the evidence

We used the GRADE methodology to assess the quality of evidence for the primary objective, and GRADE‐CERQual to assess the quality of evidence for the secondary objective. The quality of evidence relating to all five outcomes under the primary objective were downgraded two levels, due to very serious risk of bias concerns, and one level for imprecision, due to a small sample size. Outcomes of quality and timeliness of stock management, and satisfaction and acceptability of the intervention by providers, were downgraded to very low‐quality evidence, as these were based on data from the intervention group only.

All but one study included for the secondary objective had significant methodological limitations – they did not include empirical data, had unclear descriptions of the source of the information, resulting in limited evidence to support the findings (Shieshia 2014). These studies described the interventions, and the conclusions were typically drawn from authors’ experiences in implementing the intervention. Our confidence in the evidence for the secondary objective was typically downgraded due to methodological limitations of the studies, and adequacy of the findings, owing to a small number of studies contributing to specific findings.

Potential biases in the review process

We do not believe that the potential for bias in the review process for the primary objective was high. The authors meticulously followed the protocol. Where necessary, we attempted to contact the study authors to request missing relevant information.

For the secondary objective, while we followed the study protocol, the inclusion criteria were broadly defined. This could have resulted in the omission of certain articles and reports, especially if these were not published on any of the search engines that we outlined in our approach. In some cases, where reporting on the details of the intervention or factors affecting its implementation was unclear or incompletely reported, the review authors attempted to infer relevant findings from the study authors’ presented opinions.

The review team represents diverse professional backgrounds, which could have influence our input in conducting this review. Three of the review authors (SA, TT, GM) have been closely involved with the development and deployment of digital interventions in low‐ and middle‐income countries, and have experience in conducting primary research to evaluate digital health interventions. One review author (SA) is a co‐author on the study included in the primary objective. While these experiences provided us with a platform for understanding the complexities and nuances of evaluating such interventions, they may also have influenced our analyses of the studies included in this review. We tried to moderate this influence by working closely with other review authors. SA questioned the weight she attributed to certain data that resonated with her experiences, and ensured that all data were equally weighted in the final set of findings. Other members of the review team were called upon to verify the findings and ensure that they were supported by the data. As is standard practice in qualitative research, two authors conducted the GRADE‐CERQual assessment.

Agreements and disagreements with other studies or reviews

To our knowledge, this is the first systematic review of mobile phone‐based interventions for tracking health commodity inventory and stock notification, and trying to understand factors that affect implementation of interventions targeted at improving stock availability. One literature review assessed the potential impact of mobile‐based interventions on drug supply chain and stock management as one of several outcomes (Aranda‐Jan 2014). The review narratively summarised results from two studies that were excluded from the primary objective, as they did not meet the study design inclusion criteria (Barrington 2010; Githinji 2013). It concluded that evidence was insufficient to assess the impact of mobile devices on drug stock management.

Figure 1

Study flow diagram

Ir a la figura de la revisiónAbrir en una pestaña nueva

Figure 2

Common key intervention components of interventions targeted at reducing stock‐outs

Ir a la figura de la revisiónAbrir en una pestaña nueva

Figure 3

Enablers and barriers of implementation of interventions targeted at commodity stock‐outs

Ir a la figura de la revisiónAbrir en una pestaña nueva

Analysis 1.1

Comparison 1: Mobile stock notification and enhanced management (cStock + EM) vs standard care, Outcome 1: Stockout of drugs in the last 30 days

Ir a la figura de la revisiónAbrir en una pestaña nueva

Analysis 1.2

Comparison 1: Mobile stock notification and enhanced management (cStock + EM) vs standard care, Outcome 2: Stockout of drugs on the day of visit

Ir a la figura de la revisiónAbrir en una pestaña nueva

Analysis 2.1

Comparison 2: Mobile stock notification and efficient product transport (cStock + EPT) vs standard care, Outcome 1: Stockout of drugs in the last 30 days

Ir a la figura de la revisiónAbrir en una pestaña nueva

Analysis 2.2

Comparison 2: Mobile stock notification and efficient product transport (cStock + EPT) vs standard care, Outcome 2: Stockout of drugs on the day of visit

Ir a la figura de la revisiónAbrir en una pestaña nueva

Summary of findings 1. Primary objective: mobile stock notification with enhanced management compared to standard care

Mobile stock notification (cStock) with enhanced management (EM) compared to standard care in primary healthcare settings
Patient or population: healthcare workers and other health professionals involved in commodity or stock management Setting: primary healthcare setting in Malawi Intervention: mobile stock notification with enhanced management (cStock + EM), which involved quality improvement teams tasked with using the data supplied by the stock notification system Comparison: standard care, which involved routine stock management with mobile stock notification, or any other digital intervention
Outcomes	*Illustrative comparative risks (95% CI)**		Relative effect (95% CI)	No of participants (studies)	Certainty of the evidence (GRADE)	Results in words
Outcomes	Assumed risk with standard care	Corresponding risk with cStock + EM	Relative effect (95% CI)	No of participants (studies)	Certainty of the evidence (GRADE)	Results in words
Availability of commodities
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of cotrimoxazole to treat bacterial infections)	167 per 1000 healthcare workers	160 per 1000 healthcare workers (82 to 317)	RR 0.96 (0.49 to 1.90)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of cotrimoxazole because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of artemether‐lumefantrine 2 X 6 to treat malaria caused by Plasmodium facilparum)	189 per 1000 healthcare workers	136 per 1000 healthcare workers (68 to 272)	RR 0.72 (0.36 to 1.44)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of artemether‐lumefantrine because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of oral rehydration salts (ORS) to treat dehydration)	256 per 1000 healthcare workers	258 per 1000 healthcare workers (156 to 432)	RR 1.01 (0.61 to 1.69)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of oral rehydration salts because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of zinc 20 mg to treat diarrhoea)	211 per 1000 healthcare workers	209 per 1000 healthcare workers (118 to 376)	RR 0.99 (0.56 to 1.78)	171 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of zinc because it is supported by very low‐certainty evidence.
Quality of stock management
Quality of data about stock management (assessed by the extent to which HSAs (intervention group participants) sent messages about the stocks on hand for all the products they managed)	In the intervention group, an average of 85% (N = 393) of the health surveillance assistants (HSA) who managed relevant medicines reported completely on stock levels. This outcome was not assessed in the comparison group.			393^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on quality of data about stock management because it is supported by very low‐certainty evidence.
Timeliness of stock management
Time between stock‐level reporting and appropriate action (measured over an 18‐month period (January 2012 to June 2013)	In the intervention group, health facilities took an average of 12.8 days to fulfil an order requested by the health surveillance assistants (lead time). This outcome was not assessed in the comparison group.			393^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on the timeliness of stock management because it is supported by very low‐certainty evidence.
Satisfaction and acceptability
Provider acceptability and satisfaction (proportion of participants who reported using the digital intervention)	In the intervention group, the proportion of participants who reported using the digital intervention (cStock) as the primary means for ordering health products was 97% (N = 81). This outcome was not assessed in the comparison group.			81^d (1 CBA)^a	⊕⊝⊝⊝ Very low^e	We are uncertain of the effect of this approach on provider satisfaction with stock management because it is supported by very low‐certainty evidence.
Resource use
No studies were identified that reported on this outcome
Unintended consequences
No studies were identified that reported on this outcome
The basis for the assumed risk (e.g. the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). 95% CI: 95% confidence interval; RR:* risk ratio; CBA: controlled before‐after trial
GRADE Working Group grades of evidence High certainty: Further research is very unlikely to change our confidence in the estimate of effect. Moderate certainty: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low certainty: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low certainty: We are very uncertain about the estimate.
^aShieshia 2014. Published and unpublished data. Study conducted in primary healthcare setting ^bDowngraded two levels for very serious concerns regarding risk of bias: unclear random sequence generation, allocation concealment, and blinding of participants not feasible given the intervention, unclear blinding of outcomes and incomplete outcome reporting ^cDowngraded one level for imprecision: small sample size ^dFor this outcome, the number of study participants was based on a different study sample to the one used for the other outcomes. These data come from ongoing data (backend data in a digital system), and comprise of all the health workers who ever reported on stock levels ^eNon‐comparable results, thus downgraded to very low

Summary of findings 1. Primary objective: mobile stock notification with enhanced management compared to standard care

Ir a la tabla de la revisión

Summary of findings 2. Primary objective: mobile stock notification with effective product transport compared to standard care

Mobile stock notification (cStock) with effective product transport (EPT) compared to standard care in primary healthcare settings
Patient or population: healthcare workers and other health professionals involved in commodity and stock management Setting: primary healthcare settings in Malawi Intervention: mobile stock notification with effective product transport (cStock + EPT), which involved providing health surveillance assistants (HSA) with training and tools for bicycle maintenance Comparison: standard care, involved routine stock management with mobile stock notification or any other digital intervention
Outcomes	*Illustrative comparative risks (95% CI)**		Relative effect (95% CI)	No of Participants (studies)	Certainty of the evidence (GRADE)	Results in words
Outcomes	Assumed risk with standard care	Corresponding risk with cStock + EPT	Relative effect (95% CI)	No of Participants (studies)	Certainty of the evidence (GRADE)	Results in words
Availability of commodities
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of cotrimoxazole to treat bacterial infections)	167 per 1000 healthcare workers	218 per 1000 healthcare workers (117 to 407)	RR 1.31 (0.70 to 2.44)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of cotrimoxazole because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of artemether‐lumefantrine 2 X 6 to treat malaria caused by Plasmodium facilparum)	189 per 1000 healthcare workers	270 per 1000 healthcare workers (153 to 472)	RR 1.43 (0.81 to 2.50)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of artemether‐lumefantrine because it is supported by very low‐certainty evidence.
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of oral rehydration salts (ORS) to treat dehydration)	211 per 1000 healthcare workers	129 per 1000 healthcare workers (63 to 260)	RR 0.61 (0.30 to 1.23)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of oral rehydration salts because it is supported by very low‐certainty evidence
Proportion of healthcare workers who reported a stockout of drugs in the last 30 days (stockout of zinc 20 mg to treat diarrhoea)	256 per 1000 healthcare workers	281 per 1000 healthcare workers (171 to 465)	RR 1.10 (0.67 to 1.82)	168 (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on stockout of zinc because it is supported by very low‐certainty evidence
Quality of stock management
Quality of data about stock management (assessed by the extent to which HSAs (intervention group participants) send messages about the stocks on hand for all the products they managed)	In the intervention group, an average of 65% (N = 253) of the health surveillance assistants (HSAs) who managed relevant medicines reported on stock levels. This outcome was not assessed in the comparison group.			253^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on quality of data about stock management because it is supported by very low‐certainty evidence
Timeliness of stock management
Time between stock‐level reporting and appropriate action (Measured over an 18‐month period: January 2012 to June 2013)	In the intervention group, health facilities took an average of 26 days to fulfil an order requested by the health surveillance assistants (lead time). This outcome was not assessed in the comparison group.			253^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on the timeliness of stock management because it is supported by very low‐certainty evidence
Satisfaction and acceptability
Provider acceptability and satisfaction (Proportion of participants who reported using the digital intervention)	In the intervention group, the proportion of participants who reported using the digital intervention (cStock) as the primary means for ordering health products was 91% (N = 78). This outcome was not assessed in the comparison group.			78^d (1 CBA)^a	⊕⊝⊝⊝ Very low^b,c	We are uncertain of the effect of this approach on provider satisfaction with stock management because it is supported by very low‐certainty evidence
Resource use
No studies were identified that reported on this outcome
Unintended consequences
No studies were identified that reported on this outcome
The basis for the assumed risk (e.g. the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). 95% CI: 95% confidence interval; RR: risk ratio; CBA: controlled before‐after trial
GRADE Working Group grades of evidence High certainty. Further research is very unlikely to change our confidence in the estimate of effect. Moderate certainty. Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low certainty. Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low certainty. We are very uncertain about the estimate.
^aShieshia 2014 published and unpublished data. Study conducted in primary healthcare setting. ^bDowngraded two levels for very serious risk of bias concerns: unclear random sequence generation, allocation concealment, and blinding of participants not feasible given the intervention, unclear blinding of outcomes and incomplete outcome reporting ^cDowngraded one step for imprecision: small sample size ^dFor this outcome, the number of study participants is based on a different study sample to the one used for the other outcomes. These data come from ongoing data (backend data in a digital system), and comprise all the health workers who ever reported on stock levels.

Summary of findings 2. Primary objective: mobile stock notification with effective product transport compared to standard care

Ir a la tabla de la revisión

Summary of findings 3. Secondary objective: summary of findings

Summary of qualitative findings for the secondary objectives^a
	Summary of review finding	Studies contributing to the review finding	Overall GRADE‐CERQual assessment of confidence in the evidence	Explanation of GRADE‐CERQual assessment^b
1	Infrastructural issues, such as challenges in charging phones, uploading and transmitting data, and loss of data due to poor access to electricity and poor or non‐existent internet connectivity were identified as key barriers to implementation.	Negandhi 2016 Asiimwe 2011; Atnafu 2017; Biemba 2017; Negandhi 2016; Shieshia 2014; Stanton 2016; USAID 2010 Atnafu 2017	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
2	Concern was expressed that digital stock notification systems used at the facility level could not mitigate several, broader health system problems, including an underlying lack of stock at the national or district level, and a mismatch between national ordering routines and local needs.	Chandani 2017; Githinji 2013; Hara 2017; Mikkelsen‐Lopez 2014	Low confidence	Reduced due to serious concerns regarding methodological limitations, and minor concerns regarding adequacy
3	Programmes could benefit from strong partnerships, such as with local authorities; with local telecommunications companies; with technical system providers; and with non‐governmental organizations (NGOs).	WHO 2013	Very low confidence	Reduced due to serious concerns regarding methodological limitations, and serious concerns regarding adequacy
4	The availability and use of data on stock levels at all levels of the health system allowed health care officials to respond to anticipated shortages.	Asiimwe 2011; Barron 2016; Biemba 2017; Shieshia 2014; Stanton 2016	Low confidence	Reduced due to serious concerns regarding methodological limitations, and concerns regarding coherence
5	Supportive supervision and responsive management played an important role in effective adoption of a digital system.	Asiimwe 2011; Barrington 2010; Chandani 2017; Negandhi 2016; Shieshia 2014 Asiimwe 2011	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
6	The extent to which health workers are familiar with smartphones and are given adequate training in using the digital system, influences the adoption of the system.	Asiimwe 2011; Barrington 2010; Githinji 2013; Negandhi 2016; Shieshia 2014; Stanton 2016	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
7	The availability of technical programming expertise for the initial development and ongoing maintenance of the digital system is an important implementation factor.	Asiimwe 2011; Biemba 2017; USAID 2010 Biemba 2017	Low confidence	Reduced due to serious concerns regarding methodological limitations, and concerns regarding adequacy
8	Incentives, such as receiving phone talk‐time credit, to improve adoption and use of the digital system are valuable.	Barrington 2010	Low confidence	Reduced due to serious concerns regarding methodological limitations, and concerns regarding adequacy
9	User‐friendly systems, built with user participation with easy‐to‐use interfaces were considered important to implementation.	Namisango 2016; Negandhi 2016; Shieshia 2014 Namisango 2016	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
10	The use of basic mobile phones or personal phones by health workers reduced challenges with data coverage and expense, and supported easier adoption of the intervention due to familiarity with the phones.	Barrington 2010; Stanton 2016	Low confidence	Reduced due to serious concerns regarding methodological limitations, and minor concerns regarding adequacy
11	Software features, such as ability to capture images, map geographic features, support two‐way communication, toll‐free text messaging, and interoperability were considered important.	Barrington 2010; Negandhi 2016; Shieshia 2014	Low confidence	Reduced due to serious concerns regarding methodological limitations, and minor concerns regarding adequacy
12	Dashboard design and data visualisation played important roles in effective implementation. Managers should have access to data in an easy‐to‐use format, such as an interactive dashboard.	Negandhi 2016; Shieshia 2014; USAID 2010	Moderate confidence	Reduced due to serious concerns regarding methodological limitations
^aThe study authors referred to several factors that may influence the implementation, uptake, or efficient use of interventions targeted at improving stock management ^bThe GRADE‐CERQual evidence profile for each finding is available in Table 1

Summary of findings 3. Secondary objective: summary of findings

Ir a la tabla de la revisión

Table 1. GRADE‐CERQual evidence profile

Summary of review finding	Studies contributing to the review finding	Methodological limitations	Coherence	Adequacy	Relevance	GRADE‐CERQual assessment of confidence in the evidence	Explanation of GRADE‐CERQual assessment
1. Infrastructural issues, such as challenges in charging phones, uploading and transmitting data, and loss of data due to poor access to electricity and poor or non‐existent internet connectivity were identified as key barriers to implementation.	Asiimwe 2011; Atnafu 2017; Biemba 2017; Negandhi 2016; Shieshia 2014; Stanton 2016; USAID 2016	Serious concerns, because 6 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to methodological limitations, as all source material did not include empirical data.
2. Concern was expressed that digital stock notification systems used at the facility level could not mitigate several, broader health system problems, including an underlying lack of stock at the national or district level, and a mismatch between national ordering routines and local needs.	Chandani 2017; Githinji 2013; Hara 2017; Mikkelsen‐Lopez 2014	Serious concerns, because 3 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	Minor concerns about adequacy, as 3 studies refer to the same intervention administered in different settings	No, or very minor concerns about relevance	Low confidence	Reduced due to methodological limitations, and concerns about adequacy, as conclusions are based on few studies.
3. Programmes could benefit from strong partnerships, such as with local authorities; with local telecommunications companies; with technical system providers; and with non‐governmental organizations (NGOs).	WHO 2013	Serious concerns, because one study had serious methodological limitations (insufficient evidence to support findings, and unclear description of the intervention)	No, or very minor concerns about coherence	Concerns about adequacy, as only one study contributed to the finding	No, or very minor concerns about relevance	Very low confidence	Reduced due to methodological limitation, as source material did not include empirical data, and concerns about adequacy, as conclusions are based on one study with thin data
4. The availability and use of data on stock levels at all levels of the health system allowed health care officials to respond to anticipated shortages.	Asiimwe 2011; Barron 2016; Biemba 2017; Shieshia 2014; Stanton 2016	Serious concerns, because 4 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	Concerns about coherence due to contradictory findings	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Low confidence	Reduced due to methodological limitations, and concerns about coherence of the data.
5. Supportive supervision and responsive management played an important role in effective adoption of a digital system.	Asiimwe 2011; Barrington 2010; Chandani 2017; Negandhi 2016; Shieshia 2014	Serious concerns, because 4 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations
6. The extent to which health workers are familiar with smartphones and are given adequate training in using the digital system, influences the adoption of the system.	Asiimwe 2011; Barrington 2010; Githinji 2013; Negandhi 2016; Shieshia 2014; Stanton 2016	Serious concerns, because 5 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations.
7. The availability of technical programming expertise for the initial development and ongoing maintenance of the digital system is an important implementation factor.	Asiimwe 2011; Biemba 2017; USAID 2016	Serious concerns, as 3 studies had serious methodological limitations (insufficient evidence to support findings)	No, or very minor concerns about coherence	Concerns about adequacy, as only 3 studies contributed to the finding, and the presented data are sparse.	No, or very minor concerns about relevance	Low confidence	Reduced due to concerns about methodological limitations and adequacy, as conclusions are based on three studies with thin data.
8. Incentives, such as receiving phone talk‐time credit, to improve adoption and use of the digital system are valuable.	Barrington 2010	Serious concerns, because 1 study had serious methodological limitations (insufficient evidence to support findings)	No, or very minor concerns about coherence	Concerns about adequacy, as only 1 study contributed to the finding	No, or very minor concerns about relevance	Low confidence	Due to methodological limitations and concerns about adequacy, as conclusions are based on few studies.
9. User‐friendly systems, built with user participation with easy‐to‐use interfaces were considered important to implementation.	Namisango 2016; Negandhi 2016; Shieshia 2014	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations.
10. The use of basic mobile phones or personal phones by health workers reduced challenges with data coverage and expense, and supported easier adoption of the intervention due to familiarity with the phones.	Barrington 2010; Stanton 2016	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings)	No, or very minor concerns about coherence	Concerns about adequacy, as only 2 studies contributed to the finding	No, or very minor concerns about relevance	Low confidence	Reduced due to methodological limitations, and concerns about adequacy, as conclusions are based on few studies.
11. Software features, such as ability to capture images, map geographic features, support two‐way communication, toll‐free text messaging, and interoperability were considered important.	Barrington 2010; Negandhi 2016; Shieshia 2014	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	Minor concerns about adequacy, due to few studies and the relevant data are sparse.	No, or very minor concerns about relevance	Low confidence	Due to concerns about methodological limitations, and concerns about adequacy, as conclusions are based on few studies.
12. Dashboard design and data visualisation played important roles in effective implementation. Managers should have access to data in an easy‐to‐use format, such as an interactive dashboard.	Negandhi 2016; Shieshia 2014; USAID 2016	Serious concerns, because 2 studies had serious methodological limitations (insufficient evidence to support findings), and one study had minor methodologic limitations	No, or very minor concerns about coherence	No, or very minor concerns about adequacy	No, or very minor concerns about relevance	Moderate confidence	Reduced due to concerns about methodological limitations.

Table 1. GRADE‐CERQual evidence profile

Ir a la tabla de la revisión

Table 2. Risk of bias in the included study for the primary objective (Shieshia 2014)

Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"randomly assigned", but method of sequence generation not reported
Allocation concealment (selection bias)	Unclear risk	Not reported
Blinding of participants and personnel (performance bias)	High risk	Blinding was not possible due to the nature of the intervention
Blinding of outcome assessment (detection bias)	Unclear risk	Not reported
Incomplete outcome data (attrition bias)	Unclear risk	Different sample sizes for different outcomes; more participants were analysed at follow‐up than at baseline; and only a sample of participants were included in the analysis. It was unclear whether the researchers were able to collect data from all respondents.
Selective reporting (reporting bias)	Low risk	All outcomes in the protocol were reported in the published results
Other bias	Low risk	No other bias identified

Table 2. Risk of bias in the included study for the primary objective (Shieshia 2014)

Ir a la tabla de la revisión

Table 3. Methodological limitations of the included studies for the secondary objectivesa

Study ID	Clearly stated aim, objective or purpose?	Clear description of the source of the information (transparency)?	Clear description of the programme,intervention, policy or reform?	Clear description of the context/s?	Is the information accurate?	Is the evidence representative?	Any limitations of the information and/or methods discussed?	Is evidence provided to support any findings or conclusions made?	Relevant rights and ethics considerations described (empirical studies only)	Interests declared and any potential conflicts of interest noted?	Overall assessment^d
Asiimwe 2011	Unclear^b	Unclear	Yes	Yes	Unclear	Unclear	No	No	N/A	Unclear ‐ funding source declared but no conflict of interest declaration	Major limitations^e,f,g,h
Atnafu 2017	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes ‐ none to declare	No or few limitations^e
Barrington 2010	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ conflicts reported	Major limitations^e,g,h
Barron 2016	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h
Biemba 2017	Unclear^b	Yes	Yes	Yes	Yes	Unclear^c	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Chandani 2017	Yes	Yes	Yes	Yes	Yes	Unclear^c	Yes	Yes	Yes	No reporting on conflict of interest (however, a related study has reported on COI)	Minor limitations^h,i
Githinji 2013	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ conflicts reported	Major limitations^e,g,h
Hara 2017	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Mikkelsen‐Lopez 2014	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ conflicts reported	Major limitations^e,g,h
Namisango 2016	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Negandhi 2016	Yes	Yes	Yes	Yes	Yes	Unclear^c	Yes	No	N/A	Yes ‐ none to declare	Major limitations^e,g
Shieshia 2014	Unclear^b	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes ‐ none to declare	Minor limitations^e
Stanton 2016	Unclear^b	Yes	Yes	Yes	Yes	Yes	No	No	N/A	Yes ‐ none to declare	Major limitations^e,g
USAID 2010	No	No	No	No	No	No	No	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h,i,j
USAID 2016	No	No	No	No	No	No	No	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h,i,j
WHO 2013	No	No	No	No	No	No	No	No	N/A	No reporting on conflict of interest	Major limitations^e,g,h,i,j
^aDetails of the WEIRD tool assessment criteria and prompts are available in Appendix 2 ^bStated aim does not include assessing implementation factors ^cSmall sample size ^dNo or few limitations: when the answer to most questions in the tool is YES Minor limitations: when the answer to most questions in the tool is YES or UNCLEAR Significant / major limitations: when the answer to one or more questions in the tool is NO Explanation of overall assessments ^eConcerns related to relevance of study aim to review objectives ^fConcerns related to source of information reported ^gConcerns related to evidence to support findings ^hConcerns related to lack of COI declaration ⁱConcerns related to generalizability ^jConcerns related to description of source of information, aims, programme and context

Table 3. Methodological limitations of the included studies for the secondary objectivesa

Ir a la tabla de la revisión

Comparison 1. Mobile stock notification and enhanced management (cStock + EM) vs standard care

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1.1 Stockout of drugs in the last 30 days Show forest plot	1		Risk Ratio (M‐H, Random, 95% CI)	Subtotals only

1.1.1 Stockout of cotrimoxazole 480 mg	1	171	Risk Ratio (M‐H, Random, 95% CI)	0.96 [0.49, 1.90]
1.1.2 Stockout of artemether‐lumefantrine 1 X 6	1	171	Risk Ratio (M‐H, Random, 95% CI)	0.66 [0.35, 1.22]
1.1.3 Stockout of artemether‐lumefantrine 2 X 6	1	171	Risk Ratio (M‐H, Random, 95% CI)	0.72 [0.36, 1.44]
1.1.4 Stockout of ORS sachets	1	171	Risk Ratio (M‐H, Random, 95% CI)	1.01 [0.61, 1.69]
1.1.5 Stockout of zinc 20 mg	1	171	Risk Ratio (M‐H, Random, 95% CI)	0.99 [0.56, 1.78]
1.2 Stockout of drugs on the day of visit Show forest plot	1		Risk Ratio (M‐H, Random, 95% CI)	Subtotals only

1.2.1 Stockout of cotrimoxazole 480 mg	1	167	Risk Ratio (M‐H, Random, 95% CI)	0.83 [0.32, 2.11]
1.2.2 Stockout of artemether‐lumefantrine 1 X 6	1	167	Risk Ratio (M‐H, Random, 95% CI)	0.38 [0.14, 1.01]
1.2.3 Stockout of artemether‐lumefantrine 2 X 6	1	167	Risk Ratio (M‐H, Random, 95% CI)	1.21 [0.63, 2.32]
1.2.4 Stockout of ORS sachets	1	167	Risk Ratio (M‐H, Random, 95% CI)	1.64 [0.82, 3.29]
1.2.5 Stockout of zinc 20 mg	1	167	Risk Ratio (M‐H, Random, 95% CI)	1.17 [0.52, 2.60]

Comparison 1. Mobile stock notification and enhanced management (cStock + EM) vs standard care

Ir a la tabla de la revisión

Comparison 2. Mobile stock notification and efficient product transport (cStock + EPT) vs standard care

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
2.1 Stockout of drugs in the last 30 days Show forest plot	1		Risk Ratio (M‐H, Random, 95% CI)	Subtotals only

2.1.1 Stockout of cotrimoxazole 480 mg	1	168	Risk Ratio (M‐H, Random, 95% CI)	1.31 [0.70, 2.44]
2.1.2 stockout of artemether‐lumefantrine 1 X 6	1	168	Risk Ratio (M‐H, Random, 95% CI)	1.21 [0.73, 1.99]
2.1.3 Stockout of artemether‐lumefantrine 2 X 6	1	168	Risk Ratio (M‐H, Random, 95% CI)	1.43 [0.81, 2.50]
2.1.4 Stockout of ORS sachets	1	168	Risk Ratio (M‐H, Random, 95% CI)	1.10 [0.67, 1.82]
2.1.5 Stockout of zinc 20 mg	1	168	Risk Ratio (M‐H, Random, 95% CI)	0.61 [0.30, 1.23]
2.2 Stockout of drugs on the day of visit Show forest plot	1		Risk Ratio (M‐H, Random, 95% CI)	Subtotals only

2.2.1 Stockout of cotrimoxazole 480 mg	1	164	Risk Ratio (M‐H, Random, 95% CI)	0.86 [0.34, 2.19]
2.2.2 Stockout of artemether‐lumefantrine 1 X 6	1	164	Risk Ratio (M‐H, Random, 95% CI)	0.71 [0.33, 1.55]
2.2.3 Stockout of artemether‐lumefantrine 2 X 6	1	164	Risk Ratio (M‐H, Random, 95% CI)	1.34 [0.71, 2.53]
2.2.4 Stockout of ORS sachets	1	164	Risk Ratio (M‐H, Random, 95% CI)	0.90 [0.39, 2.06]
2.2.5 Stockout of zinc 20 mg	1	164	Risk Ratio (M‐H, Random, 95% CI)	0.33 [0.09, 1.16]

Comparison 2. Mobile stock notification and efficient product transport (cStock + EPT) vs standard care

Ir a la tabla de la revisión

	Idioma de la Revisión Cochrane Escoja su idioma de preferencia para las revisiones Cochrane y otros contenidos. Las secciones sin una traducción aparecerán en inglés.

	Idioma de la web Escoja su idioma de preferencia para la web de la Biblioteca Cochrane.

Idioma de la Revisión Cochrane

Idioma de la web

Resumen

Antecedentes

Objetivos

Métodos de búsqueda

Criterios de selección

Obtención y análisis de los datos

Resultados principales

Conclusiones de los autores

PICO

PICO

Population

Intervention

Comparison

Outcome

Resumen en términos sencillos

Uso de los teléfonos móviles para hacer un seguimiento de los medicamentos y notificar a los superiores cuando se necesitan más

Resumen visual

Authors' conclusions

Implications for practice

Implications for research

Summary of findings

Background

Description of the condition

Description of the intervention

How the intervention might work

Why it is important to do this review

Objectives

Primary

Secondary

Methods

Criteria for considering studies for this review

Types of studies

Primary objective

Secondary objectives

Types of participants

Types of interventions

Types of outcome measures

Primary objective

Search methods for identification of studies

Electronic searches

Searching other resources

Trial registries

Grey literature

Data collection and analysis

Selection of studies

Data extraction and management

Primary objective

Secondary objective

Assessment of risk of bias in included studies

Assessment of risk of bias in included study for the primary objective

Assessment of methodological limitations of included studies for the secondary objectives

Measures of treatment effect

Unit of analysis issues

Dealing with missing data

Assessment of heterogeneity

Assessment of reporting biases

Data synthesis

Subgroup analysis and investigation of heterogeneity

Sensitivity analysis

Summary of findings and assessment of the certainty of the evidence

Results

Description of studies

Results of the search

Included studies

Primary objective

Secondary objective

Interventions

Outcomes

Excluded studies

Risk of bias in included studies

Risk of bias in included studies for the primary objective

Methodological limitations of included studies for the secondary objectives

Effects of interventions

Primary objective

Comparison 1: Mobile stock notifications with enhanced management (EM) compared to standard care

Availability of commodities

Quality of stock management

Timeliness of stock management