Improving surgical practice ‐ a systematic review of effective education strategies to improve surgical technical skills in operative procedures
Abstract
This is a protocol for a Cochrane Review (Intervention). The objectives are as follows:
a)To determine the effectiveness of educational interventions to improve surgeon technical performance and patient health outcomes.
b)To examine whether there is any difference in the effectiveness of training programs for different characteristics such as level of training, surgical specialty, surgical techniques (open and minimally invasive surgery) and clinical setting.
Background
The problem of translating research into practice improvement has become a major focus area in healthcare over the past several years. As a result, many evidence based practice guidelines have appeared in the medical literature proclaiming success in improving physician performance through the transfer of knowledge. A wide variety of methods for changing physician practice behaviors have been identified including at least six methods identified by Eisenberg 1986: education, feedback, participation of physicians in efforts to induce change, administrative regulations, financial incentives and sanctions (Eisenberg 1986). More recently, Davis 1995 and Oxman 1995 compiled a similar list, excluding strategies that were considered to be solely administrative or coercive. The authors concluded that effective change strategies for medical professionals may include reminders, patients mediated interventions, outreach visits and opinion leaders; however feedback and educational materials were found to be less effective and formal continuing medical education conferences and activities lacking enabling or practice‐reinforcing strategies had relatively little impact. The generally modest effects these methods have achieved have led to widespread uncertainty as to the best methods to improve physician performance (Fingerhut 2005). Furthermore, the majority of individual studies and reviews of professional behaviour change focus on improving clinical decision making. However within surgery, further attention needs to be paid to skill acquisition to ensure that surgeons achieve technical competence.
The credentialing of surgeons for new and innovative procedures has been a matter of great concern (Kopelman 1997). There seems to be no consensus about the best way to teach these new technical skills; however it is recognized that skills training may require, at the very least, attendance at short courses. One of the first skills workshops for practicing surgeons was held at the Royal College of Surgeons of England in 1977 (Apley 1980). Since then, intercollegiate basic surgical skills course is now a mandatory part of surgical training in the United Kingdom. In the United States, the need for psychomotor skills training in orthopedic surgery was recognized in the mid‐1960. In 1975, an 18 hour course was held at the University of Washington's School of Medicine on motor skills for orthopedic residents (Lippert 1975). In Canada, courses in surgical techniques were held as early as 1962 (Couves 1970). A variety of similar models for teaching technical surgical skills for laparoscopic techniques have also been developed, including a study by Rosser 1998 which concluded that basic skills relevant to laparoscopic performance 'can be acquired with a high level of competence in a brief course unrelated to prior surgical experience, sex or age'.
With the recent advent of several new surgical techniques and technologies, accelerated changes in the healthcare system, increasing economic and ethical constraints, there is a need for defining a more structured approach to the attainment of surgical technical skills. Of equal interest is the identification of individual and organizational barriers in the teaching of technical surgical skills to surgeons and surgical trainees. The most effective knowledge transfer strategy could be combined with additional interventions and incentives designed to optimize the utilization of research recommendations. This paper will review the effectiveness of different approaches aimed at improving surgeons' technical performance and their patients' health outcomes.
Objectives
a)To determine the effectiveness of educational interventions to improve surgeon technical performance and patient health outcomes.
b)To examine whether there is any difference in the effectiveness of training programs for different characteristics such as level of training, surgical specialty, surgical techniques (open and minimally invasive surgery) and clinical setting.
Methods
Criteria for considering studies for this review
Types of studies
Randomized controlled trials (RCTs) or well designed quasi‐experimental studies (quasi‐RCTs), controlled before and after studies (CBAs). No studies will be excluded on the basis of language.
Types of participants
Any surgeon, surgical trainee, surgical resident or surgical clinical fellow provided with an educational program aimed at improving their surgical technical skills in operative procedures. Medical students will not be included.
Types of interventions
Mentoring, training workshops, planned preceptorships physical and virtual simulators training or any other educational program aimed at improving technical surgical skills in operative procedures. The program may have been delivered at a teaching or community hospital setting, clinic, CME or professional association. Web‐based educational programs platforms including webcasting will also be included in the review. Techniques for both open surgery and minimally invasive surgery will be considered for inclusion.
Types of outcome measures
Outcomes will be divided into those which are measured on a simulated patient and those which are directly related to patient care. Outcome measures on simulated patients will include time to complete procedure, error rates, economy of movement and other measures of technical skills. Outcome measures directly related to patient care will include any objective measure of professional performance related to surgical skills such as operative times, time to completion of task, error rates, conversion rates, and patient health outcomes such as length of stay, complication rates, morbidity rates and survival rates. Outcomes measures will include all the effects of the intervention‐ both the benefits and the harms. Measures of surgeons' knowledge, attitudes or satisfaction will be included in this review as they may provide useful secondary information. However, studies only reporting knowledge, attitudes or satisfaction with no objective measure of professional performance or patient outcomes will be excluded.
Search methods for identification of studies
The proposed electronic databases to be included in the search will be searched for the period up to May 2006:
a)The Cochrane Central Register of Controlled Trials and the Database of Reviews of Effectiveness.
b)The EPOC Register and the database awaiting assessment.
c)MEDLINE, Web of science‐ISI, CINAHL, PsycINFO, ERIC, Embase and Google Scholar.
Search Strategies for electronic databases have been developed using the methodological component of the EPOC search strategy combined with selected MeSH terms and free text terms related to training programs for surgical technical skills.
The data terms that will be used in the MEDLINE search strategy are provided below. This search strategy will be translated into the other databases using the appropriate controlled vocabulary as applicable.
1. (surgeon or surgeons).tw.
2. specialties, surgical/
3. "Internship and Residency"/
4. (Surgery or surgical).tw.
5. 3 and 4
6. (surg$ resident or surg$ trainee$).tw.
7. ((technical or surgical) adj2 skill?).tw.
8. exp surgery/ed
9. 1 or 2 or 5 or 6 or 7 or 8
Reference lists of all papers will be screened for relevant articles and authors of relevant papers will be contacted regarding any further published or unpublished work.
Data collection and analysis
Screening
FH will screen the titles and abstracts and hard copies of potentially relevant studies will be retrieved. Two reviewers (FH, AM) will independently screen potentially eligible studies against the inclusion criteria. Any disagreements will be resolved by discussion or consensus with JG and RB.
Data extraction
FH and AM will undertake data abstraction from the full text of each eligible paper using a modified version of the EPOC data collection checklists. Any disagreements will be resolved by discussion and by consensus with JG and RB.
Quality
We will not exclude any studies on the basis of reviewer's perceived quality. Tow reviewers will assess the quality based on the criteria provided by the EPOC module (see ADDITIONAL INFORMATION, ASSESSMENT OF METHODOLOGICAL QUALITY under GROUP DETAILS). Any disagreements in quality ratings will be resolved by involvement of a third reviewer.
Reporting
We will report the data in natural units. Where baseline results are available from RCTs, and CBAs, we will report pre‐intervention means and proportions for study and control participants and unadjusted and adjusted absolute change with confidence intervals (95%) from baseline.
Data analysis
Primary analysis
Primary analyses will be based upon consideration of dichotomous process measures. Where studies report more than one measure for each endpoint, the primary measure will be abstracted or the median measure identified on the risk difference scale. For example, if the comparison reports five dichotomous process of care variables and none of them were denoted the primary variable then we would rank the effect sizes for the five variables and take the median value. The results for all comparisons will be presented using a standard method of presentation where possible. For comparisons of RCTs, CBAs we will report (separately for each study design):
a) Median effect size across included studies
b) Inter‐quartile ranges of effect sizes across included studies
c) Range of effect sizes across included studies
Comparisons that randomize or allocate clusters (professionals or health care organizations) but do not account for clustering during analysis have 'potential unit of analysis errors' resulting in artificially extreme p‐values and over narrow confidence intervals (Ukoumunne 1999). We will attempt to reanalyse studies with potential unit of analysis errors where possible. If a comparison is re‐analyzed then the p‐value will be quoted and annotated with 'reanalyzed'. If this is not possible, we will report only the point estimate.
Secondary analysis
Secondary analyses will explore consistency of primary analyses with other types of endpoints (for example continuous process of care measures; dichotomous outcome of care measures and continuous outcome of care measures). Standardised effect sizes will be calculated for continuous measures by dividing the difference in mean scores between the intervention and comparison group in each study by an estimate of the (pooled) standard deviation. This results in a "scale free" estimate of the effect for each study, which can be interpreted and pooled across studies regardless of the original scale of measurement used in each study (Laird 1990).
Heterogeneity analysis
To assess heterogeneity, study characteristics will be summarized in a table and bubble plots specific for each study design. A specific quality table to assess heterogeneity will be produced whenever possible for each study design. Forest plots will be used to assess and display potential heterogeneity; funnel plots will be used to explore the possibility of publication bias. The underlying causes of heterogeneity will be explored and when possible a stratified analysis will be performed. If a sufficient number of studies are identified, a meta‐regression analysis will be performed to examine if the heterogeneity between studies can be explored by one or more factors across all studies.
Sensitivity analysis
We will perform sensitivity analysis to explore the robustness of any conclusions reached from the meta‐analysis. We will specifically examine reviews for which the results are substantially different from other studies.
Ongoing Studies
We will descirbe any on‐going studies identified, where available, detailing the primary author, research question(s), methods and outcome measures together with an estimate of the reporting date.
