The effectiveness of the public release of performance data in changing consumer, healthcare professional or organisational behaviour.
Abstract
This is a protocol for a Cochrane Review (Intervention). The objectives are as follows:
To estimate the effects of public release of performance data on consumers of health care, providers of health care and purchasers of health care. We distinguish three target groups: consumers of healthcare (patients), providers of healthcare (health professionals), purchasers of health care.
For each target group, we will consider the following comparisons.
-
public release of performance data compared to control
-
different types of public release of performance data compared to each other
Background
It is becoming increasingly common to release information about the performance of hospitals, health professionals, and health care organisations into the public domain. This takes several forms ‐ 'consumer reports', 'report cards', 'provider profiles' or 'hospital performance tables'. There is no clear agreement about what this release of data is expected to achieve. Commentators have suggested several different ways in which the data might be used (Marshall 2000, Berwick 2003, Mason 2006). One is that the data will promote an efficient market mechanism in health care (Bentley 1998, Edgman‐Levitan 1996), by prompting consumers to choose access to high quality health care (Hannan 1994, Hibbard 1989, Schneider 1998, Lansky 1998). Some suggest that such data could be used as a marketing tool (Longo 1997). Others suggest that it will help to control costs (Berwick 1990, Sirio 1996), or affect the decisions made by those purchasing health care on behalf of consumers (Brook 1994, Hibbard 1997, Mukamel 1998). It has been proposed as a tool to regulate the health system (Hannan 1994, Rosenthal 1998), a method of ensuring accountability of provider organisations (Longo 1997, Topol 1994), or of making judgements about the performance of individual professionals (Kassirer 1994). Finally, some perceive public disclosure as a mechanism to promote quality improvement by informing purchasers or encouraging providers to focus on quality problems (Hannan 1994, Longo 1997, NHS Executive 2000, Rainwater 1998, Schneider 1996).
Much argument about the use of public disclosure focuses on the validity of the performance measures themselves (Giuffrida 1999) and the validity of implicit or explicit comparisons of performance (Parry 1998, Rixom 2002). There are concerns that failure to take sufficient account of the severity of illness may lead to hospitals or clinicians who treat higher risk patients being labelled as poor performers, or lead to providers choosing to treat lower risk patients (Schneider 1996). In health systems where providers charge at the point of use, 'better' performing providers may charge more (Mukamel 1998), thus restricting access to better care. Other adverse effects might be that consumers lose confidence in their providers without having the option to change to another provider, and demoralisation of health service staff, and reduced performance in aspects of care not being monitored.
The use of audit and feedback as a tool for improving professional practice and health care outcomes has been reviewed elsewhere (Jamtvedt 2006). However, those reviews did not cover the public disclosure of performance data with implicit or explicit comparisons to the performance of others. This review examines the evidence from well‐controlled investigations of the public disclosure of performance data. It is largely based on previous reviews published in 2000 (Marshall 2000) and 2008 (Fung 2008) that addressed the same question, but with an less extended systematic search and less restrictive criteria for study design, for example observational cohort and descriptive study.
Objectives
To estimate the effects of public release of performance data on consumers of health care, providers of health care and purchasers of health care. We distinguish three target groups: consumers of healthcare (patients), providers of healthcare (health professionals), purchasers of health care.
For each target group, we will consider the following comparisons.
-
public release of performance data compared to control
-
different types of public release of performance data compared to each other
Methods
Criteria for considering studies for this review
Types of studies
-
Randomised controlled trials (RCT), including cluster randomised trials.
-
Quasi randomised trials (QRT) i.e. using methods of allocation such as alternation or allocation by case note number.
-
Interrupted time series (ITS) studies with at least three data points before and three data points after the intervention.
-
Controlled before‐and‐after studies (CBA), whith at least two intervention sites and two control groups are chosen to be similar in respect of the main outcome measures at baseline.
Types of participants
Health care providers, including organisations e.g. hospitals, practices and individual health professionals, without any restriction by type of health care professional or provider.
Types of interventions
The active intervention should contain the following elements:
The preparation of performance data about any aspect of the performance of health care organizations or individuals within the same health care system, including process measures (e.g. waiting times) and/or health care outcomes (e.g. mortality) and/or structure measures (e.g. presence of waiting‐rooms) and/or consumer or patient experiences, like CAHPS (Consumer Assessment of Health care Providers and Systems) and/or expert‐ or peer‐assessed measures, like certification, accreditation, or quality ratings given by colleagues (Harris 2002). The summaries may or may not provide comparisons with similar providers or against quality standards and may or may not be adjusted for case mix. Summaries may be prepared and released by any organization (such as government, insurers, consumer organizations).
The release of performance data into the public domain in written or electronic form (including varying degrees of accessibility, such as a report in a publicly accessible library, or more active dissemination direct to consumers by newspapers, leaflets, personal mailings, broadcast media etc). The data may be presented numerically, graphically or pictorially.
The control intervention should consist of whatever usual practice was in that setting, which may include other interventions aimed at quality improvement, including internal use of the same performance data.
Types of outcome measures
Primary outcome measures will be organized according with two important aims of public release of performance data:
1. Improvement through selection:
-
changes in the health care utilisation decisions of consumer (public and patients)
-
changes in the health care utilisation decisions of healthcare professional
-
changes in the health care utilisation decisions of purchasers
2. Improvement through changes in care:
-
objective measures of provider performance, including those that were made public and others that were not
-
valid measures of staff morale or behavior ('valid' defined as having the development of the assessment tool reported in a peer reviewed journal)
Secondary outcome measures:
-
Awareness, attitude, views, knowledge in all target groups
-
Costs
Where possible, we will collect data on how outcome measures may vary with participants' characteristics.
Search methods for identification of studies
We will search the following electronic databases :
(a) The EPOC Register and the database of studies awaiting assessment (see SPECIALISED REGISTER under GROUP DETAILS)
(b) The Cochrane Central Register of Controlled Trials (CENTRAL) and the Database of Abstracts of Reviews of Effectiveness (DARE)
(c) MEDLINE OVID SP, EMBASE, CINAHL and PsychInfo.
Search strategies for electronic databases are being developed using the methodological component of the EPOC search strategy combined with selected MeSH terms and free text terms. The following terms will be used in the search in MEDLINE (This search strategy will be translated into the other databases using the appropriate controlled vocabulary as applicable):
-
random$.tw.
-
multicenter study.pt.
-
randomized controlled trial.pt.
-
randomized controlled trial.pt.
-
clinical trial.pt.
-
intervention studies/
-
experiment$.tw.
-
(time adj series).tw.
-
(pre test or pretest or (posttest or post test)).tw.
-
random allocation/
-
impact.tw.
-
intervention?.tw.
-
chang$.tw.
-
evaluation studies/
-
evaluat$.tw.
-
effect?.tw.
-
comparative studies/
-
compar$.tw.
-
1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 or 16 or 17 or 18
-
editorial.pt.
-
letter.pt.
-
comment.pt.
-
20 or 21 or 22
-
animals/
-
humans/
-
24 not 25
-
23 or 26
-
19 not 27
-
(public release of performance data and healthcare providers).mp. [mp=title, original title, abstract, name of substance word, subject heading word]
-
exp Primary Health Care/
-
exp Hospitals/
-
physicians/
-
health professionals.ab,ti.
-
health personnel/
-
health plans.ab,ti.
-
health plan.ab,ti.
-
insurance.ab,ti.
-
*Physicians's practice patterns/
-
*Group Practice/
-
*Institutional Practice/
-
*Private Practice/
-
*Family Practice/
-
*Physicians/
-
*Physicians, Family/
-
*Professional Practice/
-
*Nurses/
-
*Nurse Clinicians/
-
*Nurse practitioners/
-
*Pharmacists/
-
*Pharmacies/
-
*Pharmacy/
-
*Hospitals/
-
(physician$ or GP? or doctor? or general pract$ or prescriber? or group pract$ or institutional pract$ or partnership pract$ or family pract$ or general pract$ or office pract$ or private pract$ or primary pract$ or nurse or nurses).tw.
-
(pharmacist? or pharmacies or pharmacy).tw.
-
hospital?.tw.
-
physiotherapist.mp.
-
midwife.mp.
-
health care centre.mp.
-
dietician.mp.
-
health care provider.mp.
-
*Allied Health Personnel/
-
*Dental Clinics/
-
*Dentists/
-
*Outpatient Clinics, Hospital/
-
general pract$.tw.
-
psychologist.mp.
-
psychiatrist.mp.
-
35 or 36 or 37
-
or/38‐67
-
or/30‐35
-
69 or 70
-
68 or 71
-
quality assurance, health care/
-
*benchmarking/
-
*"process assessment (health care)"/
-
*"outcome assessment (healht care)"/
-
exp Quality Indicators, Health Care/
-
performance outcome.ab,ti.
-
(quality adj2 indicator?).tw.
-
(quality adj (criteria or criterion or standard? or norm)).tw.
-
(performance adj (indicator? or measure? or data or rating)).tw.
-
disclosure/
-
Information Services/
-
report card.ab,ti.
-
quality information.ab,ti.
-
public information.ab,ti.
-
consumer information.ab,ti.
-
patient information.ab,ti.
-
73 or 74 or 75 or 76 or 77 or 78 or 79 or 80 or 81 or 82 or 83 or 84 or 85 or 86 or 87 or 88
-
exp Consumer Satisfaction/
-
patient preferences.ab,ti.
-
public reporting.tw.
-
consumer reports.ab,ti.
-
decision making.ab,ti.
-
choice behaviour.ab,ti.
-
choice behavior.ab,ti.
-
exp "Patient Acceptance of Health Care"/
-
'provider profiling'.ab,ti.
-
90 or 91 or 92 or 93 or 94 or 95 or 96 or 97 or
100. 28 and 72 and 89 and 99
We will not restrict the searches by publication status, date or language. The following types of articles will be excluded: letters, editorials/commentaries, reviews, lectures.
Other sources:
Handsearches:
The following journals will be searched: Medical Care, Health Services Research, JAMA.
Contacted experts:
We will consult experts on public reporting and performance data and studies currently ongoing or which have not yet been published.
Additional searches:
Reference list of identified randomised clinical trials and review articles will be checked in order to find randomised trials not identified by the electronic or hand searches.
Data collection and analysis
SELECTION OF STUDIES
Titles and abstracts of studies from electronic databases will be screened by two reviewers independently (NK (100%), MF,SF,LHR, KD, ME (all 20%). After the first round, the authors examine the studies full text for inclusion, based on eligibility criteria, by two reviewers independently. We will make two sets of three reviewers (NK + SF + KD and MF+LHR + ME). Each group presents three researchers from different universities, prevent selecting bias. In all rounds of inclusion, disagreements will be resolved by discussion with a third reviewer.
DATA EXTRACTION AND QUALITY ASSESSMENT
Before we start with the data extraction we will have a group meeting and perform a pilot test with four studies. With this pilot we are able to see if the assessment is identical for all authors. We will extract data on the following aspects of each study:
a) target groups:
-
consumer
-
providers
-
purchasers
b) intervention
-
performance data (type of measures outcome, structural or process, whether and how adjusted for case mix, who prepared and released data)
-
method of dissemination (numerical, graphical or pictorial summary, how made available to potential users of data, with or without explicit comparisons with other providers and how comparisons done)
c) type of control intervention
-
usual setting
-
as for active intervention
-
other efforts at quality improvement
The two review groups will independently extract data relevant to quality, intervention and outcomes using the standard EPOC tools, as described in the EPOC module on The Cochrane Library. For dichotomous outcomes, we will extract the proportions before and after the intervention for intervention and control groups. For continuous data, we will collect a measure of the mean average and spread for both groups before and after intervention. For ITS studies, we will collect data for both the pre and post‐intervention phases.
Where studies include an economic analysis, we will assess the quality of this analysis using the BMJ guidelines (Drummond 1996).
ANALYSIS
It is unlikely that meta‐analysis will be possible due to the anticipated heterogeneity in interventions, settings and outcome measures.
For randomised and quasi randomised trials, we will present effect sizes for each study. For dichotomous measures, we will report relative risk and absolute risk differences. For continuous measures we will present the difference in post intervention means. If possible, we will also present standardized mean differences. We will attempt to re‐analyse cluster allocated studies with unit of analysis errors; where this is not possible, we will present the point estimates without confidence intervals or significance tests and note that the study has a potential unit of analysis error. For ITS studies we will present the change in level and change in slope. This is likely to necessitate some re‐analysis of individual studies as suggested by Ramsay and colleagues (Ramsay 2003).
We will use summary of findings tables for main comparisons in the review to interpret the results and draw conclusions about the effects (benefits, potential harms and costs) of different interventions including the size of effects and quality of the evidence for outcomes for which there is evidence.
QUALITY
For randomized controlled trials and controlled clinical trials we will use 'Risk of bias' tables (Higgins 2008) paying special attention to the following domains: sequence generation, allocation concealment, blinding, and incomplete outcome data. For the rest of elegible designs we will assess their quality using pre established criteria by the EPOC Review Group. Results for controlled trials and ITS studies will be presented separately. Each study will be classified according to its quality, as follows:
-
high quality (all seven EPOC criteria marked as 'done')
-
moderate quality (one or two EPOC criteria not marked as 'done')
-
low quality (more than two EPOC criteria not marked as 'done')
Results will be presented in the following framework, applied by NK and MF:
Comparison: Public release of performance data compared to control
Subgroups by description of active intervention, and within each subgroup:
Controlled trials, grouped as:
‐ high quality with known precision (presenting effect size with 95% confidence interval for each study)
‐ high quality with unknown precision (presenting effect size for each study)
‐ not high quality with known precision (presenting effect size with 95% confidence interval for each study)
‐ not high quality with unknown precision (presenting effect size for each study)
ITS studies, grouped as:
‐ high quality
‐ not high quality
Comparison: Different types of public release of performance data compared to each other
Subgroups by description of the two active interventions, then summarised in groups as above.
