Impact of public release of performance data on the behaviour of healthcare consumers and providers
Referencias
References to studies included in this review
References to studies excluded from this review
Additional references
References to other published versions of this review
Characteristics of studies
Characteristics of included studies [ordered by study ID]
Ir a:
| Methods | Design: ITS Country: USA Care setting: acute hospitals Duration: 1 July 2006 to 30 June 2012 Dataset: 5% nationally representative sample of Medicare beneficiaries Total participants: 315,092 hospitalisations Unit of analysis: individual hospitalisations; accounted for clustering within hospitals Data analysis: regression models | |
| Participants | Inclusion criteria: all patients enrolled with Medicare, i.e. predominantly those aged 65 years or older. Hospitals: more than 4,100 hospitals in the USA Participants: 315,092: 37,829 acute myocardial infarction (16.0%), 100,189 heart failure (42.5%), 17,907 diabetes (7.6%), 80,091 chronic obstructive pulmonary disease (33.9%). | |
| Interventions | Intervention: public reporting of risk‐standardised hospital readmission rates on a public website, Hospital Compare | |
| Outcomes | Main outcome
Secondary outcomes
| |
| Notes | Abbreviations: Interrupted Time Series (ITS) study | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Incomplete outcome data (attrition bias) | Low risk | Outcome data available for all patients |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Other bias | Low risk | No additional biases identified |
| Intervention is independent of other changes? (ITS) | Unclear risk | did not state whether there were other confounding events that might have changed performance over time |
| Shape of intervention effect pre‐specified? (ITS) | High risk | Shape of intervention effect not pre‐specified |
| Knowledge of the interventions adequately prevented during the study? (ITS) | Low risk | Individuals would not have been aware of the study, as this was performed retrospectively, using the Medicare data set |
| Intervention unlikely to affect / bias data collection? (ITS) | Low risk | Routinely collected administrative data, and so data collection was unlikely to be biased by the intervention |
| Methods | Design: cRT Country: USA (Iowa) Care setting: insurance plan beneficiaries in the community Duration: February to May 2000 Dataset: data provided by the Iowa Medicaid office Total participants: 13,077 Unit of allocation: household units Unit of analysis: individual Medicaid beneficiaries; accounted for clustering of beneficiaries within household units Sample size calculation: not reported; statistical significance was assessed at the 0.05 level Data analysis: multinomial logistic regression to model the outcomes (1) stayed in assigned HMO, (2) switched to another HMO, or (3) switched to MediPass. | |
| Participants | Inclusion criteria: all new cases (i.e. household units) newly eligible to participate in Iowa Medicaid Participants: 13,077 new beneficiaries in 7016 cases with 6515 beneficiaries in the control group and 6562 in the intervention group Health plans: two HMOs under contract with the Medicaid programme and 1 primary care case management plan (MediPass). One HMO scored more highly on the publicly reported performance measures than the other. | |
| Interventions | Intervention: standard enrolment materials and Consumer Assessment of Healthcare Providers and Systems (CAHPS) report delivered by personal mail Control: standard enrolment materials delivered by personal mail Duration: February to May 2000 Deliverer: the Iowa Medicaid office posted beneficiaries a packet health plan enrolment materials that included items, such as a plan enrolment form and the CAHPS report for the intervention group Funding: co‐operative agreement 5U18HS09204‐05; the Agency for Healthcare Research and Quality and the Center for Medicare and Medicare Services | |
| Outcomes | Main outcome
| |
| Notes | The star charts in the CAHPS report were based on each HMO's performance. The bar charts included 3 charts with ratings of the health plan, health care, and personal doctor. Five charts were included by the providers or health plan. Abbreviations: cluster randomised trial (cRT); health maintenance organization (HMO); Consumer Assessment of Healthcare Providers and Systems (CAHPs) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The new cases enrolled during the study period were randomly assigned to an experimental or control group. This random assignment was independent of case size, county of residence, and initial plan assignment. |
| Allocation concealment (selection bias) | Unclear risk | The new cases enrolled during the study period were randomly assigned to an experimental or control group. |
| Adequate blinding of participants, personnel and outcome assessors? | Unclear risk | did not state whether or not participants knew that they were part of a study and so had been allocated to an intervention or control group |
| Incomplete outcome data (attrition bias) | Low risk | Quote: 'Medicaid office supplied us with data files for the full sample of new beneficiaries' |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Baseline characteristics similar? | Unclear risk | Did not explicitly describe baseline characteristics, although attempted to take these into account when determining risk‐adjusted outcomes |
| Baseline outcomes similar? | Low risk | Did not explicitly describe baseline outcomes, but accounted for differences appropriately using multinomial logistic regression |
| Protection against contamination | Low risk | No specific safeguards against contamination, but reports were sent by post, so it was unlikely that the control group received the intervention |
| Other bias | Low risk | No additional biases identified |
| Methods | Design: cNRT (non‐randomised as participants allocated based on their Medicaid case ID number) Country: USA (New Jersey) Care setting: insurance plan beneficiaries in the community Duration: March to October 1998 Dataset: data provided by the New Jersey Medicaid office Total participants: 5217 Unit of allocation: household units Unit of analysis: individual Medicaid beneficiaries; did not account for clustering of beneficiaries within household units Sample size calculation: not reported; statistical significance was assessed at the 0.05 level Data analysis: multivariable logistic regression, using enrolment with the dominant Healthcare Maintenance Organisation (HMO), despite this being shown to perform poorly by the publicly released performance data | |
| Participants | Inclusion criteria: all new cases (i.e. household units) newly eligible to participate in Iowa Medicaid Participants: 5217 new beneficiaries with 2568 in the control group and 2649 in the intervention group Health plans: the Medicaid program has a form of mandatory (auto‐assignment) voluntary managed care programme, which includes one or more HMOs or (sometimes) a primary care case management plan. New enrollees have an option to switch programme around the time of enrolment | |
| Interventions | Intervention: standard enrolment materials and Consumer Assessment of Healthcare Providers and Systems (CAHPS) report delivered by personal mail Control: standard enrolment materials delivered by personal mail Duration: 25 March to 15 April 1998 Deliverer: the New Jersey Medicaid office published a 7‐page brochure (“Choosing an HMO”) that compared the Medicaid HMO consumer ratings and experiences reported in the CAHPS survey Funding: co‐operative agreement 5U18HS09204‐05; the Agency for Healthcare Research and Quality and the Center for Medicare and Medicare Services | |
| Outcomes | Main outcome
| |
| Notes | The star charts in CAHPS report were based on a HMO's performance compared to the average in every county of residence. The counts ranged from 20 to 29 stars. The resulting standardised CAHPS ratings ranged from ‐8.40 (well below the average) to 6.26 (well above the county average). Abbreviations: cluster non‐randomised trial (cNRT); Consumer Assessment of Healthcare Providers and Systems (CAHPs) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | High risk | Quote: 'Based on whether the last digit of the case ID was odd or even, half the cases were randomly assigned to an experimental group and half were assigned to a control group' |
| Allocation concealment (selection bias) | High risk | Allocation concealment was based on case ID number, therefore research investigators enrolling participants could possibly foresee assignment |
| Adequate blinding of participants, personnel and outcome assessors? | Unclear risk | Not stated whether or not participants knew that they were part of a study and so had been allocated to an intervention or control group |
| Incomplete outcome data (attrition bias) | Low risk | Quote: 'The analysis of the overall effects of CAHPS included the entire April 1998 sample of enrollees, and is therefore not subject to non‐response bias' |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Baseline characteristics similar? | Unclear risk | Did not explicitly describe baseline characteristics, although attempted to take these into account when determining risk‐adjusted outcomes |
| Baseline outcomes similar? | Low risk | Did not explicitly describe baseline outcomes, but accounted for differences appropriately using multinomial logistic regression |
| Protection against contamination | Low risk | No specific safeguards against contamination, but reports were sent by post and so it was unlikely that the control group received the intervention |
| Other bias | Low risk | No additional biases identified |
| Methods | Design: ITS (with non‐intervention control hospitals) | |
| Participants | Inclusion criteria: children's hospitals in US states that submitted data to the PHIS | |
| Interventions | Intervention: state‐based mandatory public reporting of healthcare‐associated infections | |
| Outcomes | Main outcomes
| |
| Notes | Abbreviations: Interrupted Time Series (ITS) study; Paediatric Health Information System (PHIS); central line‐associated blood stream infection (CLABSI) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Incomplete outcome data (attrition bias) | Low risk | Outcome data for all included hospitals, except for one that was excluded because of excessive missing data |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section are reported in tables, text, or both |
| Other bias | Unclear risk | No additional biases identified |
| Intervention is independent of other changes? (ITS) | Low risk | Not stated whether there were other confounding events that might have changed performance over time. However, this was unlikely overall, as each state implemented mandatory reporting at different stages and using different regulatory mechanisms |
| Shape of intervention effect pre‐specified? (ITS) | High risk | Shape of intervention effect not prespecified |
| Knowledge of the interventions adequately prevented during the study? (ITS) | Low risk | Individuals would not have been aware of the study as this was performed retrospectively, using a clinical registry |
| Intervention unlikely to affect / bias data collection? (ITS) | Low risk | Routinely collected clinical data, so data collection was unlikely to be biased by the intervention |
| Methods | Design: cRT Country: the Netherlands (Eindhoven) Care setting: primary care Duration: 2009 to 2010 Dataset: prospective data collection from GPs Total participants: 26 GPs (2:1 randomisation to intervention) Unit of allocation: individual GPs Unit of analysis: individual GPs; accounted for clustering of GPs within practices Sample size calculation: not reported; statistical significance was assessed at the 0.05 level Data analysis: multivariable logistic regression using a difference‐in‐difference approach | |
| Participants | Inclusion criteria: all GPs within the Eindhoven region Participant characteristics: male 41% (intervention) versus 44% (control), urban 35% (intervention) versus 33% (control) | |
| Interventions | Intervention: report cards sent by post to GPs that included a variety of quality indicators that depended on the specific condition (breast cancer, cataract surgery, hip or knee replacement) | |
| Outcomes | Main outcome
| |
| Notes | Abbreviations: cluster‐randomised trial (cRT); general practitioner (GP) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | Method of randomisation unclear |
| Allocation concealment (selection bias) | Unclear risk | No statement about allocation concealment |
| Adequate blinding of participants, personnel and outcome assessors? | High risk | No blinding of participants or personnel; the outcomes measured GP behaviour (i.e. referral patterns); individual GPs were not blinded to the group allocation |
| Incomplete outcome data (attrition bias) | Low risk | Data from all participating GPs included |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Baseline characteristics similar? | Low risk | Some baseline characteristics described (health professional sex and urban location), which suggested that the groups were balanced |
| Baseline outcomes similar? | Low risk | Baseline outcomes varied between hospitals, although multivariable logistic regression was used to adjust for baseline differences |
| Protection against contamination | Low risk | No specific safeguards against contamination, although it was unlikely that GPs shared hospital report cards amongst themselves when they knew these were the subject of a trial |
| Other bias | Low risk | No additional biases identified |
| Methods | Design: ITS Duration: 2003 to 2007 | |
| Participants | Inclusion criteria: all hospitals performing 100 or more deliveries per year Hospital regions: capital city (4.9%), metropolis (31.7%), satellite city (22.5%), city (24.5%), rural (16.3%) Hospital ownership: public (3.2%), non‐public (96.8%) Hospital deliveries (per year): > 700 (4.3%), 201 to 700 (26.4%), < 200 (69.4%) | |
| Interventions | Intervention: repeated public release of information (online, press releases) on hospital caesarean rates | |
| Outcomes | Main outcome
| |
| Notes | Abbreviations: Health Insurance Review & Assessment Service (HIRA); Autoregressive Integrated Moving Average (ARIMA) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Incomplete outcome data (attrition bias) | Low risk | Outcome data available for all patients |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Other bias | Low risk | No additional biases identified |
| Intervention is independent of other changes? (ITS) | Unclear risk | Not stated whether there were other confounding events that might have changed performance over time |
| Shape of intervention effect pre‐specified? (ITS) | Low risk | The authors pre‐specified that RPR would decrease and that cesarean section rates of institutions with higher cesarean section rates in the period before RPR would decrease further after RPR than those with lower starting rates |
| Knowledge of the interventions adequately prevented during the study? (ITS) | Low risk | Did not state explicitly that those responsible for data collection were informed that the publication of performance data was part of a study |
| Intervention unlikely to affect / bias data collection? (ITS) | Low risk | Routinely collected administrative data that were collected independently of the individuals at whom the public release of performance data were directed |
| Methods | Design: ITS Duration: January 2005 to November 2012 | |
| Participants | Inclusion criteria: all Medicare fee‐for‐service enrollees hospitalised with any of the 15 most common non‐surgical discharge diagnoses. Medicare is predominantly composed of patients aged 65 years or older. Hospital types: 6.8% major teaching hospital, 18.3% minor teaching hospital, 74.9% non‐teaching Hospital size: 42.7% small, 46.3% medium, 11.0% large Hospital ownership: 15% for‐profit, 62.8% non‐profit, 22.1% public Patients: 20,707,266 Patient characteristics: mean age 79 years, 41% male | |
| Interventions | Intervention: Public release of hospital performance data (using 30‐day mortality), published on a publicly accessible website. The intervention was the addition of 30‐day mortality to publicly accessible hospital performance data in 2008. In the pre‐intervention period, hospital performance data were available in the same format, but was limited to process metrics. | |
| Outcomes | Main outcome
| |
| Notes | Abbreviations: interrupted time series (ITS) study; Centers for Medicare & Medicaid Services (CMS) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Incomplete outcome data (attrition bias) | Low risk | Data from all participating hospitals included |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Other bias | Low risk | No additional biases identified |
| Intervention is independent of other changes? (ITS) | Low risk | Not stated whether there were other confounding events that might have changed performance over time. However, this was unlikely, given that this study identified few changes in outcome after the intervention |
| Shape of intervention effect pre‐specified? (ITS) | High risk | Shape of intervention effect not prespecified |
| Knowledge of the interventions adequately prevented during the study? (ITS) | Low risk | Individuals would not have been aware of the study, as this was performed using routinely collected administrative data |
| Intervention unlikely to affect / bias data collection? (ITS) | Low risk | Routinely collected administrative data, so data collection was unlikely to be biased by the intervention |
| Methods | Design: ITS (with non‐intervention control hospitals) Duration: 2006 to 2012 | |
| Participants | Inclusion criteria: all non‐VA acute hospitals enrolled in the NHSN were eligible to participate Hospital teaching hospital status: control (469 ICU days, 59.1%), intervention (844, 76.2%) Intensive care unit bed size > 30: control (45 ICU days, 5.7%), intervention (68, 6.1%) Number of patient days per year: control (mean 1384.1, standard deviation (SD) 2152.0), intervention (1855.4, SD 1447.6) Patient characteristics: no substantial case mix data provided | |
| Interventions | Intervention: mandatory public reporting of healthcare‐associated infections | |
| Outcomes | Main outcome
| |
| Notes | Abbreviations: interrupted time series (ITS study); intensive care unit (ICU); Centers for Disease Control and Prevention (CDC); National Healthcare Safety Network (NHSN), central line‐associated blood stream infection (CLABSI); Veterans Affairs (VA) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Incomplete outcome data (attrition bias) | Low risk | Outcome data for all eligible hospitals included |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Other bias | Low risk | No additional biases identified |
| Intervention is independent of other changes? (ITS) | Low risk | Not stated whether there were other confounding events that might have changed performance over time. However, this was unlikely overall, as each state implemented mandatory reporting at different stages, and using different regulatory mechanisms |
| Shape of intervention effect pre‐specified? (ITS) | High risk | Shape of intervention not prespecified |
| Knowledge of the interventions adequately prevented during the study? (ITS) | Low risk | Individuals would not have been aware of the study as this was performed retrospectively, using administrative data |
| Intervention unlikely to affect / bias data collection? (ITS) | Low risk | Routinely collected clinical data, so data collection was unlikely to be biased by the intervention |
| Methods | Design: CBA | |
| Participants | Inclusion criteria: all paediatric hospital discharges eligible for PDI2 (i.e. length of stay 2 or more days) in a state that was categorised as 'never reporters' (18 states), '2006 reporters' (2 states), or '2009 reporters' (7 states) Participants: 4,705,857 discharges, 2,580,621 of which were from 'never reporters', 179,322 from '2006 reporters', and 1,945,914 from '2009 reporters' Participant age: never reporters (mean 3.5, standard deviation (SD) 5.5), 2006 reporters (4.4, SD 6.0), 2009 reporters (3.6, SD 5.6). Participant sex: never reporters (male 54%, female 46%), 2006 reporters (54% male, 46% female), 2009 reporters (55% male, 45% female). | |
| Interventions | Intervention: mandatory public reporting of healthcare‐associated infections | |
| Outcomes | Main outcome
| |
| Notes | Abbreviations: controlled before‐after (CBA) study; Healthcare Cost and Utilization Project (HCUP); paediatric safety indicator (PDI12); Agency for Healthcare Research and Quality (AHRQ); International Statistical Classification of Diseases 9th Revision Clinical Modification (ICD‐9‐CM) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | High risk | CBA study, so did not use random sequence allocation |
| Allocation concealment (selection bias) | High risk | No allocation concealment as hospitals would have known whether or not their state mandated public reporting |
| Adequate blinding of participants, personnel and outcome assessors? | High risk | No blinding of participants, personnel, or outcome assessors, as all parties would have known whether or not their state mandated public reporting |
| Incomplete outcome data (attrition bias) | Low risk | Outcome data for all included hospitals |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Baseline characteristics similar? | Low risk | Baseline characteristics differed, but were sufficiently similar to undertake the study using appropriate analyses |
| Baseline outcomes similar? | Low risk | Baseline outcomes differed, but were sufficiently similar to undertake the study using appropriate analyses (2.4 PDI12 per 1000 discharges in the never‐reporting states, 2.6 in the 2006 reporter states, and 3.0 in the 3009 reporter states) |
| Protection against contamination | High risk | Unable to protect against contamination, as hospitals in states without mandatory reporting might have been influenced by states in which these laws were introduced |
| Other bias | Low risk | No additional biases identified |
| Methods | Design: ITS | |
| Participants | Inclusion criteria: adults admitted to acute non‐federal hospitals in California and New York for a target condition, i.e.: California ‐ target conditions
New York ‐ target conditions
Hospital characteristics: no substantial case mix data provided Participant characteristics: no substantial case mix data provided | |
| Interventions | Intervention: California (CHOP following legislation mandating the Office of Statewide Health Planning and Development to produce annual reports); New York (New York Cardiac Surgery Reporting System) | |
| Outcomes | Main outcome
| |
| Notes | Abbreviations: interrupted time series (ITS) study; California Patient Discharge Data Set (CPDDS); Statewide Planning and Research Cooperative System (SPARCS); Autoregressive Integrated Moving Average (ARIMA); acute myocardial infarction (AMI); coronary artery bypass grafting (CABG); California Hospital Outcomes Projects (CHOP) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Incomplete outcome data (attrition bias) | Low risk | Outcome data for all included hospitals |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both |
| Other bias | High risk | Main analysis based on the assumption of same trend before and after intervention; difference from predicted values was reported, rather than change in trend and level |
| Intervention is independent of other changes? (ITS) | Unclear risk | Not stated whether there were other confounding events that might have changed performance over time |
| Shape of intervention effect pre‐specified? (ITS) | Low risk | Quote: 'We therefore hypothesized that hospitals with lower than expected mortality or complication rates experience significant volume increases, and hospitals with higher than expected mortality or complication rates experience significant volume decreases in the year after publication of a report card' |
| Knowledge of the interventions adequately prevented during the study? (ITS) | Low risk | Individuals would not have been aware of the study, as this was performed retrospectively, using administrative data |
| Intervention unlikely to affect / bias data collection? (ITS) | Low risk | Routinely collected administrative data that were collected independently of the individuals at whom the public release of performance data were directed |
| Methods | Design: cRT | |
| Participants | Inclusion criteria: acute hospitals participating in Ontario, Canada that were identified from the Canadian Institute for Health Information hospital discharge administrative database 1999 to 2001 and treated more than 15 patients with acute myocardial infarction (AMI) annually Institution characteristics: 12% teaching hospitals in the intervention group versus 10% in the control group; 74% community hospitals in the intervention versus 79% in the control group; 14% small hospitals in the intervention versus 10% in the control group AMI patient characteristics: median age 69 (interquartile range 57 to 78) both groups; female 35.4% versus 36.7% CHF patient characteristics: median age 77 (interquartile range 70 to 84) versus 77 (69‐84); female 51.3% versus 49.2% | |
| Interventions | Intervention: report cards with baseline performance data publicly released online and at a press conference | |
| Outcomes | Main outcomes
Secondary outcomes
| |
| Notes | Abbreviations: cluster‐randomised trial (cRT); acute myocardial infarction (AMI); congestive heart failure (CHF); ST‐elevation MI (STEMI) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Method of randomisation not explicitly stated, but this was undertaken by a dedicated study statistician who used a stratified randomisation process |
| Allocation concealment (selection bias) | Low risk | Quote 'This random assignment was stratified by type of hospital and performed by a study statistician' |
| Adequate blinding of participants, personnel and outcome assessors? | High risk | Quote: 'It was not possible to blind the hospitals to their status' Quote: 'We could not blind the delayed feedback group to the media coverage and associated publicity surrounding the study results' Quote: 'Patient charts were abstracted by an experienced research nurse', but it is unclear whether or not the nurse was blinded to allocation |
| Incomplete outcome data (attrition bias) | High risk | One hospital withdrew from the baseline phase after randomisation, and 4 withdrew from the follow‐up phase, all due to resource constraints. No intention‐to‐treat analysis was performed. Additional exclusions of patients were not reported. |
| Selective reporting (reporting bias) | Low risk | A protocol was registered in advance of randomisation and all outcomes were reported in the final report, which also included a new outcome (all‐cause mortality). |
| Baseline characteristics similar? | Low risk | Baseline characteristics of patients and hospitals between the intervention and control groups were similar |
| Baseline outcomes similar? | Low risk | Baseline outcomes presented and varied between hospitals, although results were presented as absolute change, and so accounted for baseline differences |
| Protection against contamination | High risk | Quote: 'There was extensive media coverage following the release of the baseline performance for the early feedback hospitals' Quote: 'One unanticipated observation was that several hospitals in the delayed feedback group reported that they also initiated some quality improvement activities after becoming aware of the publicly released early feedback report cards, before receiving their own hospital‐ specific results'. |
| Other bias | Low risk | No additional biases identified |
| Methods | Design: cRT | |
| Participants | Inclusion criteria: primary care institutions selected from within Qian Jiang City Institution characteristics: 60 beds in the intervention group versus 66 in the control group; 28 versus 26 doctors, 50,199 versus 49,108 annual outpatient visits Patient characteristics: mean age 37.5 years, 49.5% male | |
| Interventions | Intervention: public display of prescription information (percentage of prescriptions requiring antibiotics, percentage requiring injections, and average patient expenditure) on outpatient department bulletin boards in participating institutions | |
| Outcomes | Main outcomes
| |
| Notes | Zhang 2016 represents a single study that was reported in five articles (Wang 2014; Yang 2014; Liu 2015; Liu 2016; Tang 2016) that individually satisfied our inclusion criteria. However, the senior author confirmed that these represented multiple analyses of a single cluster‐RT (Zhang 2018 [pers comm]). Therefore, we made the decision to present the cluster‐RT (as the original study design and higher level of evidence), rather than the designs (e.g. CBA and ITS) that were presented in the other articles. Abbreviations: cluster‐randomised trial (cRT); controlled before‐after (CBA) study; interrupted time series (ITS) study. | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: 'We flipped a coin to randomly assign one (primary care institution) into the intervention group and another into the control group'. |
| Allocation concealment (selection bias) | High risk | Healthcare providers could not be blinded to the allocation |
| Adequate blinding of participants, personnel and outcome assessors? | High risk | It was not possible to blind personnel, who must have been aware of the group to which their primary care institution had been allocated |
| Incomplete outcome data (attrition bias) | Low risk | Injection prescribing data retrieved from a comprehensive administrative database |
| Selective reporting (reporting bias) | Low risk | All outcomes and results outlined in the Method section were reported in tables, text, or both. Although a protocol for the cRT was published, this appeared eighteen months after the trial reports stated that the intervention began. |
| Baseline characteristics similar? | Low risk | Some baseline characteristics described (e.g. age and sex), which suggested that the groups were balanced |
| Baseline outcomes similar? | Low risk | Baseline outcomes presented and varied between hospitals. However, the hospitals were paired according to characteristics, and the results analysed using a difference‐in‐difference approach and regression models to account for residual baseline differences. |
| Protection against contamination | Unclear risk | No statement as to whether or not other events might have influenced performance over time |
| Other bias | Low risk | No additional biases identified |
Characteristics of excluded studies [ordered by study ID]
Ir a:
| Study | Reason for exclusion |
| Cross‐sectional study comparing outcomes between states with and without mandatory public reporting | |
| Study design, controlled before‐after design; no information reported from the 2 included registries. Not enough information was reported regarding the baseline data. | |
| Data reported on a private website, so they were not made available to the public | |
| Interrupted time series with insufficient data points | |
| Interrupted time series with insufficient data points |
Flowchart for study selection
Risk of bias graph: review authors' judgements about each risk of bias item, presented as percentages across all included studies. The blank spaces represent risk of bias criteria that were not applicable to the study design.
Risk of bias summary: review authors' judgements about each risk of bias item for each included study. The blank cells represent risk of bias criteria that were not applicable to the study design.
| People: Insurance plan beneficiaries, birthing mothers, GPs Settings (countries and clinical settings): United States, Canada, South Korea, Netherlands, China / Community, primary care and hospitals Intervention: Public release of performance data Comparison: No public reporting | |||
| Outcomes | Impact | No of clinical encounters | Certainty of the evidence |
| Changes in healthcare utilisation by consumers | Public release of performance data may make little or no difference to long‐term healthcare utilisation by consumers. However, two studies (one cNRT and one ITS) found that some population subgroups might be influenced by public release of performance data. | 18,294 insurance plan beneficiariesa (3: 1 cRT, 1 cNRT, 1 ITS) | ⊕⊕⊝⊝ |
| Changes in healthcare decisions taken by healthcare providers (professionals and organisations) | Public release of performance data may make little or no difference to decisions taken by healthcare professionals. Two studies (2 cRTs) found that some decisions might be affected by public release of performance data. One study (ITS) found that decisions might be influenced by the initial release of data, but that subsequent releases might have less impact. | 3,000,000 birthsb and 67 healthcare providers (4: 2 RTs, 2 ITS) | ⊕⊕⊝⊝ |
| Changes in the healthcare utilisation decisions of purchasers | No studies reported this outcome. | ‐ | ‐ |
| Changes in provider performance | Public release of performance data may make little or no difference to objective measures of provider performance. | 82 healthcare providers (1 cRT) | ⊕⊕⊝⊝ |
| Changes in patient outcome | Public release of performance data may slightly improve patient outcomes. | 315,092 hospitalisations and 7503 healthcare providers (5: 1 RT, 3 ITS, 1 CBA) | ⊕⊕⊝⊝ |
| Adverse effects | No studies reported this outcome. | ‐ | ‐ |
| Impact on equity | Public release of performance data may have a greater effect on provider choice among advantaged populations. | Unknown (1 ITS) | ⊕⊕⊝⊝ |
| EPOC adapted statements for GRADE Working Group grades of evidence † Substantially different = a large enough difference that it might affect a decision | |||
| a Number was based only on Farley 2002a and Farley 2002b studies, as the total number of cases analysed in Romano 2004 was unclear b Number of participants in Jang 2011 (3,000,000) estimated from data presented in Chung 2014 c Downgraded one level for inconsistency as effect shown by Zhang 2016, but not IkkersheJang 2011, Ikkersheim 2013, or Flett 201511 d Downgraded two levels for risk of bias, as there was attrition of participating hospitals, evidence of contamination of the intervention across intervention and control hospitals, and blinding was not possible given the nature of the intervention e Downgraded two levels for inconsistency, as there was marked disagreement between studies, with two showing improvements in patient outcome (LiuTu 2009; Liu 20179), and three showing no such improvements (DeVoRinke 2015; DeVore 2016; Joynt 201615) | |||
| cluster‐randomised trial (cRT); cluster‐non‐randomised trial (cNRT); controlled before‐after (CBA) study; interrupted time series (ITS) study; randomised trial (RT) | |||
| Study detailsa | Improvement by selection | Improvement by changes in care | Data available | ||||||
| Study | Design, setting, and participants | Intervention | Consumers | Providers | Purchasers | Provider performance | Patient outcome | Staff morale | |
| cRT; USA; 13,077 insurance plan beneficiaries | Consumer Assessment of Healthcare Providers and Systems (CAHPS) report | X | ‐ | ‐ | ‐ | ‐ | ‐ | X | |
| cNRT; USA; 5217 insurance plan beneficiaries | Consumer Assessment of Healthcare Providers and Systems (CAHPS) report | X | ‐ | ‐ | ‐ | ‐ | ‐ | X | |
| ITS; USA; ‐ | Report cards with risk‐adjusted patient outcomes produced by state agencies | X | ‐ | ‐ | ‐ | ‐ | ‐ | b | |
| ITS; USA; 21 hospitals | State‐based mandatory public reporting of healthcare‐associated infections | ‐ | X | ‐ | ‐ | ‐ | ‐ | X | |
| CBA; USA; 3207 hospitals | Mandatory public reporting of healthcare‐associated infections | ‐ | ‐ | ‐ | ‐ | X | ‐ | X | |
| ITS; USA; 315,092 hospitalisations | Online reporting of risk‐adjusted 30‐day re‐admission rates (Hospital Compare) | ‐ | ‐ | ‐ | ‐ | X | ‐ | b | |
| ITS; USA; 3970 hospitals | Online reporting of risk‐adjusted 30‐day mortality rates (Hospital Compare) | ‐ | ‐ | ‐ | ‐ | X | ‐ | X | |
| ITS; USA; 244 hospitals | Mandatory public reporting of healthcare‐associated infections | ‐ | ‐ | ‐ | ‐ | X | ‐ | ‐c | |
| cRT; Canada; 82 hospital organisations | Report cards with risk‐adjusted patient outcomes and a press conference | ‐ | ‐ | ‐ | X | X | ‐ | X | |
| ITS; South Korea; 3,000,000 live births | Repeated public release of information (online, press releases) on hospital caesarean rates | ‐ | X | ‐ | ‐ | ‐ | ‐ | X | |
| cRT; The Netherlands; 26 general practitioners | Report cards with risk‐adjusted patient outcomes sent to GPs for discussion with patients | ‐ | X | ‐ | ‐ | ‐ | ‐ | b | |
| cRT; China; 20 primary care providers | Public display of prescription information on outpatient department bulletin boards | ‐ | X | ‐ | ‐ | ‐ | ‐ | X | |
| controlled before‐after (CBA) study; cluster‐randomised trial (cRT); cluster‐non‐randomised trial (cNRT); Consumer Assessment of Healthcare Providers and Systems (CAHPs); general practitioners (GPs); interrupted time series (ITS) study Column headers: changes in healthcare utilisation by consumers (Consumers); changes in healthcare decisions taken by healthcare providers (professionals and organisations; (Providers)); changes in healthcare decisions of purchasers (Purchasers); changes in provider performance (Provider performance); changes in patient outcome (Patient outcome); changes in staff morale (Staff morale); impact on equity (Equity) Order of studies: listed in chronological order USA, then chronological order for other countries of study a Studies grouped by intervention, i.e. mode of public release of performance data b No change in slope and so re‐analysis of the ITS data was uninformative c Presented derived data (e.g. outputs of regression models) that were insufficient for re‐analysis | |||||||||
| Intervention | Outcome | Study | Type of study | Absolute post‐intervention difference | Absolute pre‐intervention difference | Post‐intervention level in control group | Relative effect |
| Dissemination of consumer reports directly to consumers | Assigned to high‐rated HMO (2 choices) | cRT | 1.5 | 0 | 15.9 | 0.0943 | |
| Assigned to low‐rated HMO (2 options) | 0.4 | 0 | 25 | 0.0160 | |||
| Assigned to high‐rated HMO (1 option) | 1.3 | 0 | 29.5 | 0.0441 | |||
| Assigned to low‐rated HMO (1 option) | 0.1 | 0 | 23.7 | 0.0042 | |||
| Proportion choosing plan | cNRT | 0.01 | 0 | 0.69 | 0.0145 | ||
| cluster‐randomised trial (cRT); cluster‐non‐randomised trial (cNRT); health maintenance organization (HMO) | |||||||
| Intervention | Outcome | Study | Type of study | Absolute post‐intervention difference | Absolute pre‐intervention difference | Post‐intervention level in control group | Relative effect | ||
| Public reporting of injection prescribing rates in outpatient areas | Average expenditure per prescription | cRT | 3.4 | 2.2 | 41.2 | 0.0291 | |||
| Percentage of prescriptions requiring antibiotics | 4.6 | 6.1 | 62.8 | ‐0.0249 | |||||
| Percentage of prescriptions requiring combined antibiotics | 2.1 | 4.1 | 18.6 | ‐0.1083 | |||||
| Percentage of prescriptions requiring injections | 9.0 | 13.2 | 64.9 | ‐0.0643 | |||||
| Average expenditure per prescription | 7.2 | 6.9 | 44.3 | 0.0070 | |||||
| Mandatory public reporting of healthcare‐associated infections | Pediatric quality indicator per 1000 eligible discharges | CBA | 0.6 | 0.5 | 1.0 | 0.1000 | |||
| Intervention | Outcome | Study | Type of study | Absolute level effect (95% CI) | Relative change at 3 months (95% CI) | Relative change at 6 months (95% CI) | Relative change at 9 months (95% CI) | Relative change at 12 months (95% CI) | Relative change at 24 months (95% CI) |
| Repeated public release of hospital caesarean section rates | Caeserean section rate | ITS | ‐0.52 (‐0.77 to ‐0.26) | ‐0.04 (‐1.23 to 1.18) | ‐1.49 (‐2.55 to ‐0.40) | ‐2.92 (‐4.50 to 1.30) | ‐4.34 (‐6.61 to ‐1.95) | ‐ | |
| Mandatory public reporting of healthcare‐associated infections | PICU blood cultures | ITS | 7.48 (1.09 to 13.87) | 6.21 (‐2.84 to 17.10) | 9.90 (‐0.45 to 22.64) | 13.87 (1.42 to 29.82) | 18.17 (2.90 to 38.77) | 22.87 (4.11 to 49.86) | |
| PICU antibiotics | 7.29 (4.46 to 10.12) | ‐0.11 (‐2.03 to 1.89) | 1.61 (‐0.45 to 3.75) | 3.36 (0.96 to 5.87) | 5.15 (2.26 to 8.20) | 6.98 (2.50 to 10.70) | |||
| NICU antibiotics | ‐5.79 (‐9.17 to ‐2.42) | 8.12 (4.11‐12.46) | 6.06 (2.08 to 10.35) | 4.05 (‐0.35 to 8.85) | 1.90 (‐3.17 to 7.53) | ‐0.36 (‐6.25 to 6.33) | |||
| NICU blood cultures | ‐1.14 (‐1.90 to ‐0.39) | 2.49 (‐0.51 to 5.67) | 1.06 (‐2.07 to 4.39) | ‐0.42 (‐3.93 to 3.36) | ‐1.95 (‐6.02 to 2.49) | ‐3.53 (‐8.26 to 1.72) | |||
| cluster‐randomised trial (cRT); controlled before‐after (CBA) study; 95% confidence interval (95% CI); interrupted time series (ITS) study; neonatal intensive care unit (NICU); paediatric intensive care unit (PICU) | |||||||||
| Intervention | Outcome | Study | Type of study | Absolute post‐intervention difference | Absolute pre‐intervention difference | Postintervention level in control group | Relative effect |
| Public release of a range of quality indicators | All AMI processes | cRT | 2.0 | 0.9 | 65.6 | 0.0168 | |
| Use of standard admission orders | 6.1 | 0.7 | 72.5 | 0.0745 | |||
| Left ventricular function assessment | 2.9 | 6.3 | 49.8 | ‐0.0683 | |||
| Lipid test < 24 hours arrival | 3.8 | 1.6 | 51.1 | 0.0431 | |||
| Fibrinolytics < 30 mins after arrival | 2.6 | 3.1 | 45.7 | ‐0.0109 | |||
| Fibrinolytics decided by ED physician | 2.0 | 4.4 | 84.3 | ‐0.0285 | |||
| Fibrinolytics prior to transfer to CCU | 3.8 | 2.9 | 95.7 | 0.0094 | |||
| Aspirin < 6 hours arrival | 5.5 | 3.1 | 82.6 | 0.0291 | |||
| B blockers < 12 hours arrival | 2.4 | 3.9 | 73.7 | ‐0.0204 | |||
| Aspirin at discharge | 0.9 | 0.0 | 84.0 | 0.0107 | |||
| B blockers at discharge | 0.6 | 0.0 | 85.6 | 0.0070 | |||
| ACEi, ARB for LV dysfunction | 4.7 | 3.4 | 81.7 | 0.0159 | |||
| Statin at discharge | 0.3 | 0.2 | 85.5 | 0.0012 | |||
| All CHF processes | 1.0 | 3.0 | 54.6 | ‐0.0366 | |||
| LVF assessment | 2.7 | 4.5 | 55.2 | ‐0.0326 | |||
| Daily weights recorded | 1.3 | 0.3 | 24.0 | 0.0417 | |||
| Counselling on > 1 aspect of CHF | 0.9 | 1.7 | 55.3 | ‐0.0145 | |||
| ACEi, ARB for LV dysfunction | 6.3 | 1.7 | 92.4 | 0.0498 | |||
| B blockers for LV dysfunction | 4.0 | 1.7 | 71.7 | 0.0321 | |||
| Warfarin for AF | 0.6 | 3.1 | 64.2 | ‐0.0389 | |||
| atrial fibrillation (AF); acute myocardial infarction (AMI); angiotensin‐converting enzyme inhibitor (ACEi); angiotensin‐2 receptor blockers (ARB); beta‐adrenergic blocking agents (B blockers); cluster‐randomised trial (cRT); coronary care unit (CCU); congestive heart failure (CHF); emergency department (ED); left ventricular (LV); left ventricular failure (LVF); minutes (mins) | |||||||
| Intervention | Outcome | Study | Type of study | Absolute postintervention difference | Absolute pre‐intervention difference | Postintervention level in control group | Relative effect | |
| Public release of a range of quality indicators | AMI 30‐day mortality | cRT | 2.4 | 0.5 | 9.8 | 0.1939 | ||
| AMI 1‐year mortality | 3.1 | 1 | 19.4 | 0.1082 | ||||
| STEMI 30‐day mortality | 3.1 | 0.4 | 8.3 | 0.3253 | ||||
| STEMI 1‐year mortality | 3.9 | 1.2 | 13.5 | 0.2000 | ||||
| NSTEMI 30‐day mortality | 2.3 | 0.3 | 10.5 | 0.1905 | ||||
| NSTEMI 1‐year mortality | 3 | 0.9 | 22.6 | 0.0929 | ||||
| CHF 30‐day mortality | 1 | 0.9 | 9.6 | 0.0104 | ||||
| CHF 1‐year mortality | 2.6 | 0.6 | 30.3 | 0.0660 | ||||
| CHF and LV dysfunction 30‐day mortality | 0.9 | 0.6 | 8.5 | 0.0353 | ||||
| CHF and LV dysfunction 1‐year mortality | 6.3 | 1.8 | 26.3 | 0.1711 | ||||
| Mandatory reporting of healthcare‐associated infections | Pediatric quality indicator per 1000 eligible discharges | CBA | 0.6 | 0.5 | 1 | 0.1000 | ||
| Intervention | Outcome | Study* | Type of study | Absolute level effect (95% CI) | Relative change at 4 months (95% CI) | Relative change at 8 months (95% CI) | Relative change at 12 months (95% CI) | Relative change at 24 months (95% CI) |
| Hospital quality process and outcome metrics reported on a public website | 30‐day risk‐adjusted mortality | ITS | 0.12 (0.03 to 0.21) | 1.57 (‐4.28 to 8.18) | ‐2.47 (‐8.20 to 4.03) | 3.71 (‐3.25 to 11.74) | 7.18 (‐1.91 to 18.13) | |
| Public reporting of risk‐standardised hospital re‐admission rates | 30‐day re‐admission (AMI) | ITS | 0.00 (0.00 to 0.00) | ‐2.04 (‐8.56 to 5.48) | ‐1.36 (‐7.92 to 6.20) | ‐0.69 (‐7.34 to 7.00) | 0.72 (‐6.32 to 8.90) | |
| 30‐day re‐admission (heart failure) | 0.00 (0.00 to 0.00) | ‐1.39 (‐4.17 to 1.56) | ‐1.84 (‐4.59 to 1.08) | ‐1.88 (‐4.68 to 1.10) | ‐2.78 (‐6.42 to 1.15) | |||
| 30‐day re‐admission (pneumonia) | 0.00 (0.00 to 0.00) | ‐4.44 (‐13.61 to 6.91) | ‐5.07 (‐14.17 to 6.20) | ‐5.69 (‐14.71 to 5.47) | ‐7.45 (‐18.10 to 6.37) | |||
| 30‐day re‐admission (COPD) | 0.00 (0.00 to 0.00) | ‐6.66 (‐11.42 to ‐1.37) | ‐0.76 (‐6.11 to 5.23) | ‐7.64 (‐12.31 to ‐2.44) | ‐9.06 (‐13.62 to ‐4.00) | |||
| 30‐day re‐admission (diabetes) | 0.00 (‐0.00 to 0.01) | ‐0.65 (‐13.66 to 16.96) | 0.00 (‐13.13 to 17.81) | 0.65 (‐12.44 to 18.35) | 1.98 (‐13.57 to 24.36) | |||
| 30‐day mortality (AMI) | 0.00 (0.00 to 0.00) | 34.38 (2.71 to 94.32) | 35.83 (2.79 to 100.17) | 37.38 (2.88 to 106.67) | 43.06 (3.20 to 133.08) | |||
| 30‐day mortality (heart failure) | 0.00 (0.00 to 0.00) | 6.04 (‐5.86 to 21.37) | 13.78 (‐0.56 to 32.94) | 9.98 (‐3.46 to 27.77) | 13.31 (‐0.54 to 31.64) | |||
| 30‐day mortality (pneumonia) | 0.00 (0.00 to 0.00) | ‐3.96 (‐23.10 to 27.85) | ‐3.72 (‐16.70 to 14.05) | 2.94 (‐18.04 to 19.00) | ‐3.84 (‐22.51 to 26.69) | |||
| 30‐day mortality (COPD) | 0.00 (0.00 to 0.00) | 20.89 (5.51 to 41.52) | 21.63 (5.68 to 43.24) | 20.99 (5.54 to 41.75) | 22.00 (5.77 to 44.13) | |||
| 30‐day mortality (diabetes) | 0.00 (0.00 to 0.00) | ‐14.73 (‐34.83 to 23.29) | ‐15.10 (‐35.48 to 24.12) | ‐14.78 (‐34.92 to 23.40) | ‐19.39 (‐42.65 to 35.66) | |||
| Acute Myocardial Infarction (AMI); ST‐Elevation Myocardial Infarction (STEMI); Non‐ST‐Elevation Myocardial Infarction (NSTEMI); Congestive Heart Failure (CHF); Left Ventricular (LV); Chronic Obstructive Pulmonary Disease (COPD); Cluster Randomised Trial (cRT); Controlled Before‐After (CBA) study; Interrupted Time Series (ITS) study; 95% Confidence Interval (95% CI) * Joynt 2016 and DeVore 2016 provided outcomes in quarters rather than months and so have been presented as 4‐ and 8‐months rather than the pre‐specified 3‐ and 6‐months. | ||||||||
