Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children ‐ a review of clinical study reports
Abstract
This is a protocol for a Cochrane Review (Intervention). The objectives are as follows:
To review published and unpublished clinical study reports on effectiveness and harms of NIs for influenza in all age groups (and compare them with our published Review).
Background
The protocol will be posted for five months and in this time we will actively seek open feedback and criticism of the methods to be employed. All feedback will be logged and publicly posted (unless privacy is requested), and responded to. In the light of feedback, the protocol may be amended upon agreement of all review authors. Any amendments to the protocol will be detailed in full and along with the reasons why. Feedback can be submitted by email to the corresponding author and via a web‐based form on the web site where the protocol is posted.
Description of the condition
Influenza is mostly a mild, self‐limiting infection of the upper airways with local (including sniffles, nasal discharge, dry cough, sore throat) and systemic (fever, headache, aches and pains, malaise and tiredness) symptoms. Occasionally patients with influenza develop complications such as pneumonia, otitis media and dehydration, that may be due to effects of the influenza virus itself or associated secondary bacterial infections.
Influenza is not clinically distinguishable from influenza‐like illness (ILI) (Call 2005). Influenza in humans is caused by influenza A and B viruses. Currently, influenza A/H1N1 (2009), influenza A/H3N2, and influenza B cause most influenza infections worldwide (CDC 2010).
Treatment remains supportive rather than curative, despite the licensing of a class of antiviral drugs called adamantanes (amantadine and rimantadine) first applied to medicine in the 1960s. Following their use there was widespread viral resistance leading to effectiveness concerns (Bright 2006).
Description of the intervention
Neuraminidase inhibitors (NIs) comprise of inhaled zanamivir (Relenza, GlaxoSmithKline) powder, oral oseltamivir (Tamiflu, Gilead Sciences and Roche), parenteral Peramivir (BioCryst Ltd), inhaled Laninamivir (Daiichi Sankyo Co. Ltd, Sugaya 2010) and others still under development (Hayden 2009). The use of NIs has increased dramatically since the outbreak of A/H1N1 (2009) in April 2009, partly because of the rise in amantadine/rimantadine resistance and the lack of an effective vaccine which meant NIs became a widespread public health intervention for early containment and interruption of the virus. The World Health Organization (WHO) had previously encouraged member states to gain experience with NIs (WHO 2004).
How the intervention might work
Although NIs may reduce the ability of the virus to penetrate the mucus in the very early stage of infection (Bhatia 2007; Matrosovich 2004; Moscona 2005; Ohuchi 2006), their effectiveness is thought to lie in their ability to inhibit neuraminidase, which enables influenza viruses to exit host cells (Liu 1995; Moscona 2005). Oseltamivir phosphate (OP), Tamiflu, is the pro‐drug of oseltamivir carboxylate (OC), the effective form. OP dissociates in the gastrointestinal tract to form oseltamivir (OT) which is absorbed and metabolised into OC by hepatic carboxylesterase (h‐CE). OT may induce hypothermia (Ono 2008) possibly due to a central depressant action (Hama 2008). NIs may also inhibit human sialidase (Li 2007) thereby causing abnormal behaviour.
Any treatment that reduces the excretion of virus from infected people might be a useful public health measure to contain an epidemic. In addition to symptomatic treatment, prophylactic use for interrupting the spread of disease has informed pandemic planning over the past decade.
Why it is important to do this review
Most attention has been focused on oseltamivir because it is not just used as a prescription drug for patients suffering from influenza: on the recommendation of the WHO (WHO 2010) it has been purchased and supplied globally (Cohen 2009; Doshi 2009; Freemantle 2009; Godlee 2009). Governments spent billions of dollars stockpiling it as a public health measure. The WHO has recently also recommended it be added to the list of essential medicines (WHO 2010) and oseltamivir has been prescribed for the treatment of influenza worldwide after the outbreak of 2009 A/H1N1 influenza and the pandemic declaration by the WHO (WHO 2009). Oseltamivir has been prescribed far more than other NIs, most likely because of its ease of administration and storage.
There are some suggestions that NIs may not be as safe as previously assumed, with associations between oseltamivir use and neuropsychiatric adverse reactions, particularly, including sudden death (Hama 2008).
An earlier version of this Cochrane Review in adults, we found that NIs were effective in reducing symptoms and complications (Jefferson 2006). However, criticisms of that review led to doubts about their effectiveness against complications (Jefferson 2009a;Jefferson 2010a). Since then, doubts remain about the effectiveness and safety of the drug because its evaluation has been limited to manufacturer‐sponsored trials. There is clear evidence of publication bias (see below), and there is concern that some evaluations have not been available to scrutiny by the scientific community (Cohen 2009; Doshi 2009; Freemantle 2009; Godlee 2009).
In response to the most recent update of our Cochrane Review of NIs in healthy adults (Jefferson 2009a), oseltamivir's manufacturer pledged to make “full study reports” available for 10 treatment trials, of which eight have never been published (Smith 2009). This protocol explains the rationale behind our current efforts to re‐update our review in the light of this potential source of data and of regulatory documents, either openly sourced or obtained under the US Freedom of Information Act. This review is the amalgamation of two long‐standing Cochrane Reviews on the effects of NIs for influenza in healthy adults (Jefferson 2009a; Jefferson 2010a) and children (Matheson 2007; Shun‐Shin 2009). Publishing updates of the Cochrane Reviews of NIs in both children and adults generated intense interest from clinicians and the media during the influenza outbreak declared a pandemic by WHO in 2009. The Cochrane Review of NIs in healthy adults highlighted the presence of publication bias (Jefferson 2010a). Obtaining clinical study reports may allow us to clarify the effects by age because some trials report adult and paediatric outcomes.
As with most systematic reviews, our previous work included evidence identified by comprehensive searches of literature databases (such as PubMed) of published randomised, placebo controlled studies. This is designed to ensure that reviews are based on the highest quality, relevant evidence. In addition (and in line with common practice) we requested randomised controlled trials (RCTs) from authors of published trials (who may have undertaken other trials) and experts and manufacturers that had not been published. This harvested excerpts of eight unpublished and two published treatment trials with oseltamivir. However, we encountered discrepancies between the published trials and unpublished excerpts. Our attempts to reconcile these by contacting the pharmaceutical manufacturer and study authors failed (the latter were unable to provide us with the necessary data: some were not in possession of the data; others may have been restricted by confidentiality agreements). In addition, we ascertained that ghost writers had been involved (Cohen 2009), which means the named authors may not have been in full control of the trial publications. We have also identified several key differences in licensed indications for oseltamivir between regulatory systems (mainly between the US, Europe and Japan) and under‐reporting of harms (Doshi 2009). This undermined our confidence in published data and in the findings of our previous Cochrane Reviews.
To update the amalgamated reviews we are attempting to identify all relevant trials (that is, unpublished as well as published) and extract data from full clinical study reports (CSRs), a more detailed source of information than published journal articles. We know that this will be a more laborious process but we believe that the amalgamation of the two Cochrane Reviews (Jefferson 2010a; Matheson 2007) will make this process more efficient by sharing expertise and time in extracting and assessing data from these sources.
Examples of discrepancies and publication bias
The two most cited published trials of oseltamivir either do not mention serious adverse events (Nicholson 2000), or state that “... there were no drug‐related serious adverse events” (Treanor 2000). This finding has been repeated by bodies such as the UK National Health Service (NHS) (“No serious adverse events were noted in the major trials and no significant changes were noted in laboratory parameters”) (UKMIPG 2001). However, they are inconsistent with relevant information from CSRs Module 1 content released to us by Roche in January 2010. The CSRs Modules 1 report 10 serious adverse events (in nine participants) in the two trials, three of which were classified as possibly related to the study drug (oseltamivir). It has also emerged that 56% (2691/4813) of patient data from randomised, placebo controlled trials have never been published. Exclusion of unpublished data changed our previous findings regarding oseltamivir’s ability to reduce complications of influenza (Doshi 2009; Jefferson 2009a).
A modified approach
To resolve inconsistencies and under‐reporting, we are changing our approach by no longer including trial data as reported in papers published in biomedical journals. Instead, we are treating clinical study reports (CSRs) as our basic unit of analysis (the original and unabridged record of trials, short of individual patient data). CSRs are often sent to national drug regulators such as the Food and Drug Administration (FDA) and the European Medicines Agency (EMEA), which require more stringent standards for completeness and accuracy of reporting than biomedical journals.
Unfortunately, most CSRs go unpublished and are not readily available for wider scientific scrutiny, despite calls to make all relevant trial data public (Godlee 2009) and the known problems with reporting biases (McGauran 2010).
However, in the case of NIs, Roche, the manufacturer of oseltamivir, has pledged to make some of its full study reports available (Smith 2009) and expressed in email correspondence a willingness to consider making study reports for additional trials available as well. GSK have given a similarly positive response to our enquiries. We have also contacted BioCryst Ltd, makers of peramivir, and are in the process of contacting Daiichi Sankyo, makers of laninamivir, the newest NI, for similar information.
Therefore, in this review, we are modifying our approach. We will analyse unpublished reports only, which should enable us to address the remaining questions about the effects of NIs using the most complete data set short of individual patient data. We have requested the original CSRs from the manufacturers and will review additional, apparently unpublished trials we have since identified. At present, Roche has only provided us with partial CSRs: one module of the four to five contained in each CSR (Appendix 1) for 10 oseltamivir treatment trials. The other modules are likely to contain key information such as the protocols with the list of amendments and original reporting analysis plans. Regardless of success with our requests to obtain full CSRs, we intend updating our Review with available material and subsequently update it as and when additional data become available.
In addition to seeking CSRs, we will read and review regulatory documentation. Although no CSRs are obtainable from regulators, important information regarding trials which are either unpublished or supplementary details to CSRs of available trials are often contained in regulatory documents. Unlike us, regulators have access to the whole data set.
Implications
This modified approach to a Cochrane Review aims to provide patients, clinicians and policy‐makers with the most transparent and independent information possible about NIs for influenza.
Our Cochrane Review should contribute transparent and independent information to a European regulatory and pharmacovigilance legal framework which commentators declare to be weak (Cohen 2009; Godlee 2009). We believe that as NIs have become public health drugs, recommended and stockpiled globally, independent scrutiny of all the evidence relating to harms and effects on complications is necessary to provide a complete and unbiased view of their performance.
Implication for novel H1N1 influenza
In response to our December review (Jefferson 2009a; Jefferson 2010a), some have argued that its findings cannot be applied to A/H1N1 (2009), suggesting that it is a new virus and thus we need new evidence (JAID 2010; Maugh 2009; Nebehay 2009; NHS 2009; NHS 2010). We disagree. If the treatment and prophylaxis of novel A/H1N1 influenza were a new indication for which past clinical trials were inapplicable, the mass administration of oseltamivir over the past year would constitute off‐label use. However, there is little reason to believe this is the case. Novel A/H1N1 is a new strain of a subtype that has been circulating since 1977, but it also resembles A/H1N1 strain that has been circulating before 1957 (CDC 2009) or before the 1918 pandemic (Itoh 2009). Influenza subtype A/H1N1 was indeed circulating in the clinical trials we have included in our previous Reviews. In addition, oseltamivir and zanamivir were approved by regulators worldwide for the treatment and prevention of influenza types A and B, not specific subtypes or strains of influenza A and B. The expectation of regulatory approval is thus that the effects of these drugs demonstrated in clinical trials will apply to future strains of influenza A and B. Use of these drugs during the pandemic was not off‐label, but legal because of the assumption that the clinical trial evidence underpinning regulatory approval applied to novel A/H1N1. We intend to review the clinical trial evidence with the expectation that our results, similar to regulators, will apply to novel influenza A/H1N1 as well.
Wider implications
The modified approach in this Cochrane Review may provide a justification for widespread adoption of this type of method to systematic reviews of interventions. Our independent scrutiny of NI benefits and harms using all possible trial information may inform the debate on the adequacy of existing regulatory frameworks in the adoption of new drugs and whether other systematic reviews should move to this new more rigorous approach which focuses on trial programmes rather than single trials (Eyding 2010; Ioannadis 2010). We will discuss the implications of using published or unpublished clinical study reports in compiling systemic reviews. Although there is substantial evidence for the effects of reporting bias in estimates of effectiveness, less is known of its impact on the evidence of harms (Chou 2005). We intend to quantify the additional resources required to follow our novel approach. This entails a review of time and resources that made it possible to carry out this review. This may shed light as to the feasibility of other systematic reviews to proceed in a similar fashion. We intend to speculate on the generalisability of our approach to other disease areas in which a greater number of manufacturers and non‐commercial investigators are active (Eyding 2010; Ioannadis 2010)
Objectives
To review published and unpublished clinical study reports on effectiveness and harms of NIs for influenza in all age groups (and compare them with our published Review).
Methods
Criteria for considering studies for this review
Types of studies
We will include evidence from RCTs testing NIs effect for prophylaxis, post‐exposure prophylaxis (PEP) and treatment of influenza.
Types of participants
Previously healthy people (children and adults). 'Previously healthy' will be defined as including chronic illness (such as asthma, diabetes, hypertension) but excluding illnesses affecting the immune response (such as cancer, AIDS). We will use the same definitions as used in nearly all influenza RCTs and include only trials on people exposed to naturally occurring influenza with or without symptoms.
Types of interventions
NIs by any route compared with placebo or standard care during on and off treatment (on‐t and off‐t) periods.
Types of outcome measures
Primary outcomes
Primary outcome measures for treatment studies.
-
Symptom relief.
-
Hospitalisation and complications.
-
Harms.
Primary outcome measures for prophylaxis studies.
-
Influenza (both symptomatic and asymptomatic, and laboratory‐confirmed) and influenza‐like Illness (ILI).
-
Hospitalisation and complications.
-
Interruption of transmission (in its two components, reduction of viral spread from index cases and prevention of onset of influenza in contacts).
-
Harms.
Secondary outcomes
Secondary outcome measures for treatment studies.
-
Symptom relapse after finishing treatment.
-
Drug resistance.
-
Viral excretion.
-
Mortality.
Secondary outcome measures for prophylaxis studies.
-
Drug resistance.
-
Viral excretion.
-
Mortality.
We will examine listed secondary outcomes, although recognising that these may be less relevant, less reliably measured, or analysed with multiple statistical tests (leading to inflation of the overall significance level). Some trials we have reviewed so far had insufficient power to detect an effect on mortality.
We will pay particular attention to complications and adverse events, including 'compliharms', (outcomes which may be classified as either harms or complications), as this is where evidence is currently scarce or inconclusive (Jefferson 2009a; Shun‐Shin 2009). Our initial examination of some regulatory documents and some published versions of the studies, has identified that some symptoms and sequelae of influenza (such as pneumonia) are variously classified as a 'complication of influenza' or as an 'adverse event of the treatment' (Appendix 2). In post‐exposure prophylaxis (PEP) trials we will focus on evidence of interference with viral transmission.
Extracting 'compliharms' may be difficult in the Roche trials because adverse events are reported for all participants while complications are only reported for infected participants. Hence we may have to assume complications in non‐infected participants are equally likely in treatment and control groups if we have no access to those events (that is, the numerator will be an underestimate).
Search methods for identification of studies
A single, up‐to‐date and complete list of all clinical trials conducted on humans using a given drug is rarely available in the public domain. Such a list can be constructed using multiple, cross‐referencing methods. In addition, because the majority of clinical trials of a given drug are fully funded or sponsored by the drug’s manufacturer, manufacturers can be contacted to help ensure the accuracy and completeness of such a list.
To ensure the list does not include duplicate entries, it is important to assign each trial a Unique Trial ID. ‘Author’ is not a good choice of Unique Trial ID as different authors can be present across different versions of the same trial (that is, the authors of unpublished CSRs can be different from publications arising from the same clinical trial). Nor is ‘publication’ a good option for Unique Trial ID because not all studies are published. Some trials will have company specific codes and some will have public clinical trial registry numbers, or both or neither.
The majority of trials in our study are manufacturer funded (with corresponding manufacturer protocol IDs), and accordingly we have used the manufacturer protocol ID as our Unique Trial ID. Best efforts to ensure accuracy can still leave uncertainties that may require further correspondence for clarification (for example, the difference between WV15673 and WV15673D).
We are constructing a list beginning with clinical trials identified from previous Review updates. To this, we are adding additional trials in humans identified from multiple sources, such as manufacturer submissions to regulators, drug product information sheets, previous published reviews, Health Technology Assessment (HTA) documents, and public and manufacturers’ registers (Burch 2009; Cooper 2003; Jefferson 2006; Tappenden 2009; Turner 2003). These include reference lists of single or synthesis clinical trials, HTA documents, FDA medical reviews, a review by Kaiser (Kaiser 2003), the EMEA scientific discussion, material sent to us by Roche for our 2009 update, Roche’s and GSK’s submissions to UK National Institute of Clinical Excellence (NICE), Japanese regulatory new drug applications, registries such as ClinicalTrials.gov and www.roche‐trials.com, and other documents. We also plan to conduct traditional database searches (search strategy defined below) and searches of grey literature to identify previously unknown trials.
For each trial, we will attempt to gather the following details to enable decision‐making regarding whether the trials meet our inclusion criteria:
-
Unique Trial ID
-
Other IDs
-
Phase of study
-
Sponsor
-
Short description
-
Official Trial title
-
First authors (name and email)
-
Type of trial
-
Comparator
-
Outcomes assessed
-
Date of trial
-
Study period (days)
-
Population
-
Number of subjects planned
-
Number subject enrolled
-
Number subject completing
-
Trial status (for example, completed, ongoing, or early termination)
-
Publication status (a citation or understanding of why it was not published)
-
How identified (to record how the trial was discovered)
-
Notes
We will enter identified studies into a spreadsheet (part of Tool D ‐ see below). We will submit a draft list of clinical trials to manufacturers asking for their cooperation in checking the accuracy and completeness of its content. We plan to assign three categories to identified trials once we have our complete list:
-
definitely included;
-
definitely excluded; and
-
trials for which we need further information.
Where further information is required, it will be requested from the trials’ sponsor and/or first/corresponding author.
Electronic searches
We will update our searches of the electronic databases of published studies previously carried out for the Cochrane Reviews on NIs in children (Matheson 2007) and healthy adults (Jefferson 2010a). The purpose of the searches is to identify trials previously unknown to the review authors, but no published material for any trial will be analysed for this review. Rather the unpublished data from them will be analysed instead.
We will search the Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library latest issue) which includes the Acute Respiratory Infections Group's Specialised Register, the Database of Reviews of Effects (DARE) and the NHS Health Economics Database, MEDLINE (1966 to present) and EMBASE (1974 to present).
The following search strategy will be used to search MEDLINE and CENTRAL. The MEDLINE search will be combined with the Cochrane Highly Sensitive Search Strategy for identifying randomised trials in MEDLINE (Lefebvre 2011). The search strategy will be adapted for EMBASE. There will be no publication or language restrictions.
MEDLINE (Ovid)
-
Influenza, Human/
-
exp Influenzavirus A/
-
exp Influenzavirus B/
-
(influenza* or flu).tw.
-
or/1‐4
-
Oseltamivir/
-
Zanamivir/
-
Peramivir/
-
Laninamivir/
-
neuraminidase inhibitor*.tw.
-
(oseltamivir or zanamivir or tamiflu or relenza or peramivir or laninamivir or gs4071).tw,nm.
-
or /6‐11
-
5 and 12
Searching other resources
See Search methods for identification of studies section.
Data collection and analysis
Selection of studies
Four review authors (TJ, CH, MJ, RH) will independently read all data relating to the studies on the list constructed during our search and select studies fulfilling our inclusion criteria. One review author (PD) will compile the assessments into a single sheet for CDM. Disagreements will be resolved by discussion with another review author (CDM).
We will then request full internal CSRs (minus participant identification) for each trial that is definitely included.
Data extraction and management
We intend conducting a two stage exercise. In the first stage we shall assess the reliability and completeness of the identified trial data. Only reliable and reasonably complete data will be included in the second phase of the review, which is an analysis following standard Cochrane methods.
Stage 1
Two review authors will separately extract data from the same CSR for studies included in stage 1 of the review. The review authors will independently extract data from each of the sources where we have more than one type of study report on the same trial from different sources (for example, a trial report submitted to a regulatory body and a trial report from a pharmaceutical company) and then compare the results. We will record and tabulate disagreements between data extracted from the same source and between different sources. We will extract data using a modified CONSORT statement extraction template (Appendix 3).
The modified CONSORT reconstruction template aims to assemble a concise version of the CSR which will include all important methods as well as define and extract all relevant outcomes. The CONSORT template includes the features that would be expected to be found in a published trial report but in greater detail. It does not include introduction or discussion sections. The following will be extracted for each trial.
-
Background and objectives.
-
Methods: including trial design, important changes to methods after trial commencement (such as eligibility criteria), with reasons.
-
Participants: including eligibility criteria for participants and settings and locations where the data were collected.
-
Interventions: the interventions for each group with sufficient details to allow replication, including how and when they were actually administered.
-
Outcomes: pre‐specified primary and secondary outcome measures, including how and when they were assessed and changes to trial outcomes after the trial commenced, with reasons.
-
Sample size: how it was determined and explanation of any interim analyses and stopping guidelines.
-
Randomisation: including sequence generation and method used to generate the random allocation sequence.
-
Blinding: who was blinded after assignment to treatment groups.
-
Statistical methods: methods used to compare groups for primary and secondary outcomes and methods for additional analyses, such as subgroup analyses and adjusted analyses.
-
Results: participant flow, numbers of participants randomly assigned, losses and exclusions after randomisation, together with reasons. Baseline demographic and clinical characteristics for each group
-
Outcomes: primary and secondary outcome results for each group.
-
Ancillary analyses: results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre‐specified from exploratory.
-
Harms: all important harms or unintended effects in each group.
One review author will complete the reconstructed CONSORT template in full (Appendix 3) with the name and date of completion, a statement of conflict of interests. A second review authors will check the template. These reconstructed templates will be made available for scrutiny upon publication of the full review. We will copy extracted data, text, tables and figures directly from the relevant sections of the CSR into the appropriate section of the template. We will not change the text in any way apart from clarifying abbreviations, spellings etc, and aside from highlighting text. We will use three types of text highlighting in the document.
Yellow: will be used where text, figures or tables need to checked with further information (for example, if an adverse event is referred to in an appendices or a further CSR module).
Red: where text or comments have been inserted by one or both review authors but require an additional opinion due to concerns that there is the potential for discrepancies in the CSR.
Green: any text or tables of our own we have added to the template (for example, a reconstructed table of adverse events).
Two review authors have piloted the reconstruction method on Roche oseltamivir trial WV15671 with data from the CSR Module 1 from Roche and data submitted to UK NICE. The pilot reconstruction has been discussed amongst the whole group for clarification. Two review authors will judge the reliability and completeness of each reconstructed trial. A third review author will act as arbiter and will have the casting vote regarding the inclusion of the trial in Stage 2. Each reconstruction will be assessed using information from regulatory sources.
We have devised four types of repository extraction, management and cross‐referencing tools for the data and information retrieved and collated during the review due to the complexity of the work.
Tool A is a table of content (TOC). The TOC is a common resource, to be used as a formal directory or index, listing the location, by page number, where specific clinical trials are cited in primary documents the review team has access to and is included in the current review. The TOC primarily indexes regulatory documents (notably medical, pharmacological and statistical reviews written by the US FDA); it does not at present include EMEA reviews. The TOC is kept as a spreadsheet. The last sheet in the TOC lists CSRs the review team has access to, their provenance and degree of completeness (partial or complete). To identify information within and across documents, which are thousands of pages in length, we have used the trial ID to plot which trials are cited where in our database of regulatory and company data. Review authors assigned to a specific trial could do their own searches within regulatory material looking for mention of that trial, but there is a risk that without an index, they would miss relevant sections. This may occur for several reasons. Firstly, not all regulatory material may be searchable (by being transformed by 'optical character recognition' software); only somebody reading the documents serially would find relevant trial references. Secondly, reference to trials is inconsistent; trial WV15671 may be at times referred to as “WV 15671” or even “15671”. Again, without somebody reading through the documents serially, finding all references to a specific trial may be difficult even within searchable documents.
Tool B is the TOC evidence (TOCE). TOCE is a version of the TOC with annotations. It is based on the TOC but has an added brief description of the content of each regulatory and pharmaceutical files available to us.
Tool C is a narrative of our review, documenting the evolution of methods and detailed summary of the information contained in the FDA and other regulatory documents. This document is intended as a detailed record of the project with dates as well as a source of information on the topic.
The rationale for the creation of three different tools lies in the complexity of our undertaking. We are engaged in the assessment of the completeness and reliability of the data in trials X, Y, Z and sub trial programme Q (i.e. prophylaxis for NI R). To do this we have to use all information at our disposal. This includes information on the design, methods and results of trials ‐ but also the comments of those who had access to the full registration trial programme, notably regulatory agencies. We hope that their critiques will tell us things which are invisible to those not having access to full CSRs and the myriad attendant documents that go with trials. (The Roche submission to FDA for oseltamivir for the treatment of influenza in those aged 13 and above was over 300 volumes in length).
Finally, we have created a central reference resource holding details of correspondence, the definitive frame of studies with referencing to other study identifiers (published or semi‐published versions, company ID numbers), progress notes, lists of authors, relevant researcher, decision makers and pharmaceutical spokespeople quotes and other information fragments identified during the review (Tool D). We also have a central, password‐protected electronic repository where all tools and published and unpublished studies, reviews, HTA and regulatory or pharmaceutical documents are stored.
Assessment criteria. Assessment as to eligibility for trials to move to Stage 2 will be based on the following three criteria:
-
Completeness. CSR/unpublished reports includes both identifiable CONSORT‐specified methods to enable replication of the study. Identifiable CONSORT‐specified results (primary outcomes, tables, appendices)
-
Internal consistency. All parts of the same CSR/unpublished report are consistent.
-
Validated data trial details and outcome measures will be cross‐checked against regulatory documents, other versions of the same CSR/unpublished reports, other references. Consistency checks will be applied. where inconsistencies occur. They will be discussed or clarified with original trialists, manufacturers, or study guarantor/principal investigator(s).
A decision on whether a trial moves to Stage 2 will be reached by consensus. When consensus is not possible and cannot be reconciled, dissent will be reported.
Stage 2
This will be based on standard Cochrane methods for extracting, appraising and synthesising the evidence (with two review authors independently extracting data, with a third review author arbitrating). Data will be extracted onto standard forms, checked and recorded.
Assessment of risk of bias in included studies
Once stage 1 is completed, we will assess risk of bias using established criteria (Higgins 2011) in single studies. We will also test for the presence of potential biases across the entire trial programme of each NI using a multi‐level assessment framework formulated as a series of 30 null hypotheses (Additional tables). Previous studies comparing regulatory with published or internal company sources of evidence have reported a variety of different biases that affect medical knowledge (Chou 2005; MacLean 2003; McGauran 2010). We expect that our access to multiple sources for the same trial data set will aid us in assessing the presence, direction and impact of one or more of these reporting biases, some of which have already been discovered in the oseltamivir data set (i.e., publication bias). Due to the laborious and iterative nature of the assessment, we have prioritised the null hypotheses to be tested into first (Table 1) and second rank (Table 2) according to the potential impact of each type of bias on our conclusions and of knowledge of the effects of NIs. Methodological risk of bias assessment will be carried out using the Cochrane Handbook for Systematic Reviews of Interventions criteria (Higgins 2011).
| Null hypothesis | Definition | Potential Impact | Framework to test hypothesis |
| There is no under‐reporting (overview hypothesis) (Hopewell 2009; McGauran 2010) | Under‐reporting is an overall term including all types of bias when there is an association between results and what is presented to the target audience | Tailoring methods and results to the target audience may be misleading. The direction of the effect could change or the statistical significance of the effect could change or the magnitude of the effect could change from clinically worthwhile to not clinically worthwhile and vice versa | 1. Is there evidence of under‐reporting? |
| There is no difference between analysis plan in the protocol and final report (or the differences are listed and annotated) (McGauran 2010) | When protocol violations especially if not reported and justified, are not associated to study results | Post hoc analyses and changes of plan lead to manipulation of reporting and choice of what is and not reported | 1. List any discrepancies between what is pre‐specified in protocol and what was actually done |
| There is no difference between published and unpublished conclusions of the same study (McGauran 2010) | A specific bias relating to the selective reporting of data in association with target audience | Results have been tailored to the intended recipient audience | 1. Compare reporting of important outcomes (harms, complications) between published reports and other reports such as those to regulatory bodies e.g. FDA |
| Presentation of same data set is not associated with differences in spelling, incomplete, discrepant, contradictory, or duplicate entries (Doshi 2009; Golder 2010; Jefferson 2009a) | Different versions of the same data set are associated with discrepancies | Raises questions of whether these discrepancies are mistakes or deliberate? | 1. Document any differences or similarities in separate reports of important outcomes (harms, complications) based on the same studies |
| There is no evidence of publication bias (Hopewell 2009; McGauran 2010) | Publication status is not associated with size and direction of results | Negative or positive publication bias can have major impact on the interpretation of the data at all levels especially | 1. Are there studies that have not been published (yes/no)? |
| There is no evidence of outcome emphasis bias (McGauran 2010) | When over or under emphasis of outcomes is not associated with size or direction of results | Can lead to wrong conclusions because over emphasis on certain outcomes | 1. Are all of the pre‐specified outcomes in the study protocol reported? |
| There is no evidence of relative versus absolute measure bias (McGauran 2010) | When choice of effect estimates is not associated with size or direction of results | Can lead to wrong conclusions because of apparent under or overestimation of effects (e.g. in the use of relative instead of absolute measures of risk) | 1. Are both relative and absolute measures of effect size used to report the results? |
| There is no evidence of follow up bias (McGauran 2010) | When there is no evidence that length of follow up is related to size and direction of results | Can lead to wrong conclusions due to over or under emphasis of results | 1. Are reported results based on the complete follow up of each patient? |
| There is no evidence of data source bias (Chou 2005; McGauran 2010) | There is no difference between the evidence base presented to regulators (for approval for an indication) and that produced by or in possession of drug’s the manufacturer (Chou 2005) | Can lead to approved indications inconsistent with full data set | 1. Have regulators been presented with all data sets resulting from trials sponsored by the drug’s manufacturer? |
FDA: Food and Drug Admnistration
| Null hypothesis | Definition | Potential impact | Framework to test hypothesis |
| There is no difference by funder (Jefferson 2009b; McGauran 2010) | When results and tone of conclusions are associated with type of funder | Funder influences results, conclusions and study visibility | 1. Are there substantial numbers of comparable trials with different funding? |
| There is no evidence of authorship musical chairs bias (Cohen 2009; Doshi 2009; Jefferson 2009a; MacLean 2003) | When different authors for the same data set are presented to different target audiences | Raises an accountability question: who is responsible for the study? | 1. Are the names of the people responsible for the unpublished report the same as those of the published reports? |
| There is no evidence of time lag bias (McGauran 2010) | When result reporting time frame is not associated with size or direction of results | Can lead to wrong conclusions | 1. Are there significant differences in on‐t and off‐t treatment data? |
| There is no evidence of location bias (Higgins 2011) | The publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results | Can lead to wrong conclusions in a specific setting or mislead generalisation to another context | 1. Is there an association between publishing trials in journals with similar ease of access and data basing and size or direction of results? |
| There is no evidence of disclosure pressure bias (McGauran 2010) | When external stimuli to publish or not are not associated with size or direction of results | Can lead to wrong conclusions because of blocks on what is reported or not | 1. Why were some data and/or studies not published? |
| There is no evidence of off‐label bias (McGauran 2010) | When reporting is not associated with a higher or lower probability of unregistered indications use or recommendations thereof | Can lead to wrong conclusions because of reporting of data which leads to off label use or is a product of off label use | 1. Is there any difference in the on label indications and dosage between published and unpublished clinical study reports? |
| There is no evidence of commercial confidentiality bias (McGauran 2010) | When commercial confidentiality rules do not impact on presentation of results | Can lead to wrong conclusions because IPR or commercial confidentiality prevent full disclosure of results | 1. Is there evidence of commercial confidentiality being invoked for the decision to publish or otherwise. |
| There is no evidence of inclusion of previously unpublished data bias (Golder 2010; McGauran 2010) | When there is no evidence of inclusion of heterogeneous unpublished data of variable quality and sometimes difficult to interpret either because of swamping or absence of methods chapters | Can lead to wrong conclusions because of the inclusion of biased data not clearly identified as such | 1. Is there any evidence of published review studies (particularly meta‐analyses) containing previously unpublished data? |
| There is no evidence of blank cheque bias | When there is no evidence that third‐party independent researchers agree to having a trial’s sponsor fill in their data extraction sheets for unpublished data | Can lead to wrong conclusions because of the impossibility of independently assessing data. If the practice is not declared, it can mislead readers, giving conclusions a spurious impression of robustness | 1. Are there unpublished data included in the third‐party data set or meta‐analysis that were gained without independent verification? |
| There is no evidence of competition bias (McGauran 2010) | When there is no evidence that any type of reporting bias is related to market competition, leading to a better positioning of the drug | Can lead to wrong conclusions because what you see may be due to market pressures | 1. Do the types of bias detected (outcome emphasis, time lag. etc) favour NIs versus other drugs or interventions in particular ways? |
| There is no evidence of language bias (Higgins 2011) | When there is no evidence that reporting is associated with language of target audience | Can lead to wrong conclusions because what you see may be due to the type of market being targeted | 1. Is there evidence of presentation of unpublished (e.g. slide shows, product inserts) or published evidence in a particular language? |
| There is no evidence of differences in methodological quality (McGauran 2010) | When there is no evidence of difference on methodological quality by source and outcome | Can lead to wrong conclusions because methodological quality affects estimates of effect, so if quality is not in fact equivalent, then differences ascribed to drug performance may be false | 1. Is there difference in methodological quality between published and unpublished data? |
| There is no evidence of differences in sample size bias (McGauran 2010) | When there is no evidence of the presence of differences in sample in association with size and direction of results | Same potential impact as methodological quality, but with respect to sample size | 1. Are there significant differences in sample sizes between published and unpublished material? |
| There is no evidence of multi‐centre status bias (McGauran 2010) | When there is no evidence that there the presence of many or few centres is associated with size and direction of results | Can lead to wrong conclusions because what you see may be due to selection of centres and may not be generalisable | 1. Are the methods used different from centre to centre? |
| There is no evidence of citation bias | When there is no evidence that citation of a selected study is associated with size and direction of results | Pressure is placed on authors of reports of study to provide an unbalanced interpretation or perspective by selecting citations or misreporting their content | 1. Are the references in the published studies comprehensive? |
| There is no association between affiliation of authors and positive research conclusions (McGauran 2010) | When there is no evidence that differences in affiliation/employer of authors may be associated with differences size and direction of results or conclusions drawn | This form of bias is particularly dangerous when readers’ understanding or policy are based solely on the abstracts or conclusions of studies | Are there differences in study conclusions associated with affiliation of authors? |
| There is no evidence of publication constraints (McGauran 2010) | When there is no evidence that obstacles to publication are associated with size and direction of results | What you see has been filtered on the basis of its results | 1. If unpublished studies exist, why were they not published? |
| There is no evidence of study design bias | When there is no evidence that there may be differences in design to emphasise size and direction of selected results | Can be misleading as design affects results and generalisability and the choice of design is influenced by considerations other than study objective and ethics | 1. Is there any relationship between study design and study conclusions? |
on‐t: on time frame
off‐t: off time frame
IPR: intellectual property rights
Measures of treatment effect
All analyses will use the intent‐to‐treat (ITT) population as it is the most methodologically rigorous and clinically relevant. Our intention is to express the results of our analyses both as absolute and relative measures and estimate the likelihood and impact of bias on our conclusions. Assessment of bias on this scale will be done for the whole trial programme rather than for each trial, as recently recommended by Ioannidis (Ioannadis 2010) and outlined above. We will use the tri‐dimensional dose‐relatedness, timing and patient susceptibility (DoTS) methodology to assess likelihood of harms causality (Aronson 2003).
Based on pharmacokinetic, toxicokinetic data and the comprehensive nature of oseltamivir we will review harms evidence in two time frames. In time frame 1 we will assess intermediate reactions (occurring after some delay) to oseltamivir carboxylate/zanamivir such as infection, renal toxicity, hyperglycaemia/diabetes haemorrhage and general weakness (Aronson 2003). These may be closely linked to drug efficacy. In time frame 2 we will assess the first dose/early reactions to unchanged oseltamivir. This is the review for pure adverse effects. First dose reactions occur after the first dose (or on the first day) of a course of treatment and not necessarily thereafter. Reactions occurring early in treatment may abate with continuing treatment as the participants develop tolerance (Aronson 2003), or as a result of interaction with the infection (Hama 2008). In addition we will analyse harms by on time frame periods and off time frame periods.
We will report risk ratios (RR) or hazard ratios, absolute risks (risk differences and numbers need to treat), treatment harms (numbers needed to harm) and treatment effects to ensure clear interpretation of the drugs’ benefits and harms.
We will present estimates of effect as data for three broad age groups: children, adults, and the elderly, where relative and absolute benefits and harms may differ. We will present data separately for prophylaxis, post exposure prophylaxis (PEP) and treatment studies.
Unit of analysis issues
We expect the majority of trials to randomise at the patient level and events to only occur at most once in a participant. However, in the case of multiple events per participant, we will consider analyses that compare incidence rates in the two groups such as Poisson and Negative binomial regression. In the case of cluster randomised trials (for example, households rather than individuals are randomised) we will include data from analysis either at the household level or at the participant level if the effect of clustering has been taken account of appropriately (for example, by using hierarchical modelling techniques).
Dealing with missing data
We have developed a comprehensive strategy for dealing with data which we know are missing at the trial level, i.e. unpublished trials (see Search methods for identification of studies section).
At the participant level (i.e. within a trial) we will consider not including a trial if the amount of missing data is large (for example, > 20%) or if missing data is in different proportions in the treatment and control groups or if all participants are not accounted for in the reporting of the analysis. We will also consider the impact of including or excluding trials with potentially unreliable results due to missing data issues in a sensitivity analysis.
Assessment of heterogeneity
We will use the I2 statistic to measure the level of statistical heterogeneity for each outcome (Higgins 2011) and test for heterogeneity using Cochrane's Q chi2 test.
Assessment of reporting biases
This will be based on the empirical framework in Table 1. Biases will be assessed depending on available data and order of priority. For example, the results of the testing of the overarching hypothesis (presence of reporting bias) will take place on the basis of the data currently available and will be revised each time new data are obtained. Equally, the second null hypothesis (testing for analysis plan differences between the study protocol and the full study) will be tested if the modules containing the protocols have been released to us. We aim to eventually assess the remaining potential biases in both tables and compare clinical study reports from unpublished data with those published (that is, within current Reviews).
Data synthesis
When there is substantial heterogeneity (I2 statistic > 50%) we will consider possible explanations for this and consider not combining results. We will use a sensitivity analysis when necessary to investigate the contribution of individual trials to any heterogeneity. Whether or not heterogeneity is detected, we will perform a random‐effects meta‐analysis. Random‐effects methods will be used to compare the dichotomised outcomes (RR and absolute risk reduction (ARR) for efficacy and safety). More specifically, the DerSimonian & Laird method will be used. Note that in the case of zero cells (for example, no events in one group) 0.5 will be added to each cell of the 2 × 2 table for any such study.
We will convert medians of treatment groups into (log) hazard ratios (estimating the variance of these) (Parmar 1998) to enable meta‐analysis of time to event outcomes. Due to concerns that time to event outcomes may violate proportional hazards assumptions, we will also consider analysis of this outcome as a continuous outcome noting that the data are likely to be skewed. We will use the inverse‐variance random‐effects method for this analysis.
Subgroup analysis and investigation of heterogeneity
We will attempt to carry out a subgroup analysis by type of NIs and age group, specifically comparing effects of NIs in children (up to the age of 12 years), adolescents (13 to 17 years), adults (18 to 64 years) and the elderly (65 years and onwards) in which relative and absolute benefits or harms may differ. To avoid selective reporting we will conduct analyses to be consistent with what is pre‐specified in the trial study reports. Any additional analyses will be reported as "post‐hoc".
Sensitivity analysis
We will carry out a sensitivity analysis of methods, comparing our results obtained using fixed‐effect and random‐effects models. The Mantel‐Haenszel method will be used for fixed‐effect analysis except in the case of sparse data, in which case Peto’s method will be used (as recommended in the Cochrane Handbook for Systematic Reviews of Interventions). We also intend carrying out a sensitivity analysis of our results by publication status (published trials versus regulatory submissions versus CSRs) to assess the extent to which publication status and source of data affect results, by funder and study quality.
Finally, we will compile and populate data tables showing data and conclusions from different studies and different regulator sources. We aim to reconcile any differences by contacting manufacturers and regulators, and formulating a series of questions based on any documented discrepancies.
| Null hypothesis | Definition | Potential Impact | Framework to test hypothesis |
| There is no under‐reporting (overview hypothesis) (Hopewell 2009; McGauran 2010) | Under‐reporting is an overall term including all types of bias when there is an association between results and what is presented to the target audience | Tailoring methods and results to the target audience may be misleading. The direction of the effect could change or the statistical significance of the effect could change or the magnitude of the effect could change from clinically worthwhile to not clinically worthwhile and vice versa | 1. Is there evidence of under‐reporting? |
| There is no difference between analysis plan in the protocol and final report (or the differences are listed and annotated) (McGauran 2010) | When protocol violations especially if not reported and justified, are not associated to study results | Post hoc analyses and changes of plan lead to manipulation of reporting and choice of what is and not reported | 1. List any discrepancies between what is pre‐specified in protocol and what was actually done |
| There is no difference between published and unpublished conclusions of the same study (McGauran 2010) | A specific bias relating to the selective reporting of data in association with target audience | Results have been tailored to the intended recipient audience | 1. Compare reporting of important outcomes (harms, complications) between published reports and other reports such as those to regulatory bodies e.g. FDA |
| Presentation of same data set is not associated with differences in spelling, incomplete, discrepant, contradictory, or duplicate entries (Doshi 2009; Golder 2010; Jefferson 2009a) | Different versions of the same data set are associated with discrepancies | Raises questions of whether these discrepancies are mistakes or deliberate? | 1. Document any differences or similarities in separate reports of important outcomes (harms, complications) based on the same studies |
| There is no evidence of publication bias (Hopewell 2009; McGauran 2010) | Publication status is not associated with size and direction of results | Negative or positive publication bias can have major impact on the interpretation of the data at all levels especially | 1. Are there studies that have not been published (yes/no)? |
| There is no evidence of outcome emphasis bias (McGauran 2010) | When over or under emphasis of outcomes is not associated with size or direction of results | Can lead to wrong conclusions because over emphasis on certain outcomes | 1. Are all of the pre‐specified outcomes in the study protocol reported? |
| There is no evidence of relative versus absolute measure bias (McGauran 2010) | When choice of effect estimates is not associated with size or direction of results | Can lead to wrong conclusions because of apparent under or overestimation of effects (e.g. in the use of relative instead of absolute measures of risk) | 1. Are both relative and absolute measures of effect size used to report the results? |
| There is no evidence of follow up bias (McGauran 2010) | When there is no evidence that length of follow up is related to size and direction of results | Can lead to wrong conclusions due to over or under emphasis of results | 1. Are reported results based on the complete follow up of each patient? |
| There is no evidence of data source bias (Chou 2005; McGauran 2010) | There is no difference between the evidence base presented to regulators (for approval for an indication) and that produced by or in possession of drug’s the manufacturer (Chou 2005) | Can lead to approved indications inconsistent with full data set | 1. Have regulators been presented with all data sets resulting from trials sponsored by the drug’s manufacturer? |
| FDA: Food and Drug Admnistration | |||
| Null hypothesis | Definition | Potential impact | Framework to test hypothesis |
| There is no difference by funder (Jefferson 2009b; McGauran 2010) | When results and tone of conclusions are associated with type of funder | Funder influences results, conclusions and study visibility | 1. Are there substantial numbers of comparable trials with different funding? |
| There is no evidence of authorship musical chairs bias (Cohen 2009; Doshi 2009; Jefferson 2009a; MacLean 2003) | When different authors for the same data set are presented to different target audiences | Raises an accountability question: who is responsible for the study? | 1. Are the names of the people responsible for the unpublished report the same as those of the published reports? |
| There is no evidence of time lag bias (McGauran 2010) | When result reporting time frame is not associated with size or direction of results | Can lead to wrong conclusions | 1. Are there significant differences in on‐t and off‐t treatment data? |
| There is no evidence of location bias (Higgins 2011) | The publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results | Can lead to wrong conclusions in a specific setting or mislead generalisation to another context | 1. Is there an association between publishing trials in journals with similar ease of access and data basing and size or direction of results? |
| There is no evidence of disclosure pressure bias (McGauran 2010) | When external stimuli to publish or not are not associated with size or direction of results | Can lead to wrong conclusions because of blocks on what is reported or not | 1. Why were some data and/or studies not published? |
| There is no evidence of off‐label bias (McGauran 2010) | When reporting is not associated with a higher or lower probability of unregistered indications use or recommendations thereof | Can lead to wrong conclusions because of reporting of data which leads to off label use or is a product of off label use | 1. Is there any difference in the on label indications and dosage between published and unpublished clinical study reports? |
| There is no evidence of commercial confidentiality bias (McGauran 2010) | When commercial confidentiality rules do not impact on presentation of results | Can lead to wrong conclusions because IPR or commercial confidentiality prevent full disclosure of results | 1. Is there evidence of commercial confidentiality being invoked for the decision to publish or otherwise. |
| There is no evidence of inclusion of previously unpublished data bias (Golder 2010; McGauran 2010) | When there is no evidence of inclusion of heterogeneous unpublished data of variable quality and sometimes difficult to interpret either because of swamping or absence of methods chapters | Can lead to wrong conclusions because of the inclusion of biased data not clearly identified as such | 1. Is there any evidence of published review studies (particularly meta‐analyses) containing previously unpublished data? |
| There is no evidence of blank cheque bias | When there is no evidence that third‐party independent researchers agree to having a trial’s sponsor fill in their data extraction sheets for unpublished data | Can lead to wrong conclusions because of the impossibility of independently assessing data. If the practice is not declared, it can mislead readers, giving conclusions a spurious impression of robustness | 1. Are there unpublished data included in the third‐party data set or meta‐analysis that were gained without independent verification? |
| There is no evidence of competition bias (McGauran 2010) | When there is no evidence that any type of reporting bias is related to market competition, leading to a better positioning of the drug | Can lead to wrong conclusions because what you see may be due to market pressures | 1. Do the types of bias detected (outcome emphasis, time lag. etc) favour NIs versus other drugs or interventions in particular ways? |
| There is no evidence of language bias (Higgins 2011) | When there is no evidence that reporting is associated with language of target audience | Can lead to wrong conclusions because what you see may be due to the type of market being targeted | 1. Is there evidence of presentation of unpublished (e.g. slide shows, product inserts) or published evidence in a particular language? |
| There is no evidence of differences in methodological quality (McGauran 2010) | When there is no evidence of difference on methodological quality by source and outcome | Can lead to wrong conclusions because methodological quality affects estimates of effect, so if quality is not in fact equivalent, then differences ascribed to drug performance may be false | 1. Is there difference in methodological quality between published and unpublished data? |
| There is no evidence of differences in sample size bias (McGauran 2010) | When there is no evidence of the presence of differences in sample in association with size and direction of results | Same potential impact as methodological quality, but with respect to sample size | 1. Are there significant differences in sample sizes between published and unpublished material? |
| There is no evidence of multi‐centre status bias (McGauran 2010) | When there is no evidence that there the presence of many or few centres is associated with size and direction of results | Can lead to wrong conclusions because what you see may be due to selection of centres and may not be generalisable | 1. Are the methods used different from centre to centre? |
| There is no evidence of citation bias | When there is no evidence that citation of a selected study is associated with size and direction of results | Pressure is placed on authors of reports of study to provide an unbalanced interpretation or perspective by selecting citations or misreporting their content | 1. Are the references in the published studies comprehensive? |
| There is no association between affiliation of authors and positive research conclusions (McGauran 2010) | When there is no evidence that differences in affiliation/employer of authors may be associated with differences size and direction of results or conclusions drawn | This form of bias is particularly dangerous when readers’ understanding or policy are based solely on the abstracts or conclusions of studies | Are there differences in study conclusions associated with affiliation of authors? |
| There is no evidence of publication constraints (McGauran 2010) | When there is no evidence that obstacles to publication are associated with size and direction of results | What you see has been filtered on the basis of its results | 1. If unpublished studies exist, why were they not published? |
| There is no evidence of study design bias | When there is no evidence that there may be differences in design to emphasise size and direction of selected results | Can be misleading as design affects results and generalisability and the choice of design is influenced by considerations other than study objective and ethics | 1. Is there any relationship between study design and study conclusions? |
| on‐t: on time frame | |||
