EMSOP Alexandria Present 2002

Prehospital Outcomes Research : 

Prehospital Outcomes Research Establishing the Scope and Methodological Approach

PREHOSPITAL CARE: 

PREHOSPITAL CARE Increasing scrutiny Questioning the value: Interventions Transport Persistent concern--Lack of proof: Effectiveness Cost-effectiveness Methodologically sound Outcomes Research long overdue

EMS Outcomes Research and Classical Health Research: 

EMS Outcomes Research and Classical Health Research Four classical health research disciplines: Basic Research Clinical Research Epidemiological Research Health Systems/Services Research Recent development: Outcomes and Effectiveness Research (OR/OER) Understanding the development of OR

THE DEVELOPMENT OF OUTCOMES RESEARCH: 

THE DEVELOPMENT OF OUTCOMES RESEARCH Convergence of multiple forces: 70’s/80’s: Dramatic healthcare inflation Overwhelming cost  political urgency Huge variation in cost NOT correlated with outcomes More/more expensive: Little association with improvement

THE CONTINENTAL SHIFT: 

THE CONTINENTAL SHIFT Overwhelming cost Societal/corporate outcry Political urgency Growing awareness: Non-association between cost/outcome

MID-80’s: A MASSIVE CHANGE: 

MID-80’s: A MASSIVE CHANGE Approach to Hospital Reimbursement “Prospective Payment System” “Diagnosis Related Groups” (DRG’s) Not reimbursed based upon care Lump sum: Diagnosis

COINCIDENTAL RESEARCH POLITICS: 

COINCIDENTAL RESEARCH POLITICS Traditional approach to medical research: Great strides in EFFICACY research Often did NOT translate  Practice “Real world” outcomes not changed Outcry for a new discipline: Able to identify EFFECTIVENESS

ANOTHER COINCIDENCE: 

ANOTHER COINCIDENCE Societal/political concern: DRG’s  “Quicker but sicker” D/C based upon days, not clinical status Congressional hearings: Unintended consequences of incentives

WE WANT RESEARCH!!!: 

WE WANT RESEARCH!!! Politicians wanted a new research discipline Methods of correlating: Variations in practice Cost Outcome Why? To protect citizens against under-treatment To measure what they got for the federal dollar

AHEAD OF THEIR TIME: 

AHEAD OF THEIR TIME YET ANOTHER COINCIDENCE: 70’s: Pioneering work in Outcomes Research Concept: Correlate CARE with outcomes Correlate COST with outcomes Approach: Mining large databases Status: Meaningful but obscure

A MATCH MADE IN HEAVEN: 

A MATCH MADE IN HEAVEN 1986: HCFA—”Have we got data!!!” Mortality Readmission rates Adverse outcomes Result: We’ve got the data…you’ve got the method YeeHaa!!!

THE REAL HISTORY OF OER: 

THE REAL HISTORY OF OER The Outcomes “MOVEMENT”: Societal/political/regulatory/financial urgency Outcomes RESEARCH A discipline conceived before the federal juggernaut Result: Political tidal wave swept OR to the forefront Federal agenda to fund the new discipline

DHHS-1987: 

DHHS-1987 Federally sponsored meetings Agenda: Can OR use Medicare Databases? Variations in care  Outcomes Quality monitoring Quality Improvement Help identify wasteful therapies/procedures Improve health with decreased cost

THE RESULT: 

THE RESULT Major federal initiative to find “what works”: 1989: Agency for Health Care Policy and Research (AHCPR)

CONCEPTUAL FOUNDATION FOR OR: 

CONCEPTUAL FOUNDATION FOR OR Dramatic assumption: “Guidance for optimal medical practice can be gleaned from analysis of data routinely gathered in the process of delivering and paying for patient care.” (AHCPR Publication No. 99-R044)

EARLY YEARS OF OR: 

EARLY YEARS OF OR Narrow definition for OR Mining large administrative databases: Data from routine patient care Emphasis on: Identifying cost Identifying effective care Meta-analysis: “Pooling” studies to make a larger “study”

RESPONSE OF TRADITIONAL RESEARCHERS: 

RESPONSE OF TRADITIONAL RESEARCHERS ABSOLUTELY BONKERS!!!

YEAH…RIGHT!!!: 

YEAH…RIGHT!!! The assumption: Identifying what impacts patient outcomes can be determined by retrospectively looking at large patient populations in administratively developed databases The response: Heated debate NIH vs. AHCPR Philosophy Funding Politicians vs. politicians Researchers vs. researchers (L. Pauling)

THE MATURING OF OR: 

THE MATURING OF OR The early years: Mining and Meta-analysis Now: No single definition accepted Quality analysis/Quality Improvement becoming prominent Emphasis on using multiple methodologies Emphasis on measuring MANY outcomes The Six D’s Agency for Healthcare Research and Quality Codifying terminology, methodology

LEVELS OF IMPACT OF RESEARCH RESULTS: 

LEVELS OF IMPACT OF RESEARCH RESULTS Level 1 impact: Research findings that do not lead to a direct change in policy or practice. New research tools (severity indicators) New methodologies (Episode of Care Model) Instruments assisting clinical decision-making Identifying important outcomes questions Identifying changes needed in current practice

LEVELS OF IMPACT OF RESEARCH RESULTS: 

LEVELS OF IMPACT OF RESEARCH RESULTS Level 2 impact: Research results that are translated directly into policy or program changes/development Legislation Bureaucracy Healthcare payment Healthcare planning Clinical guidelines (e.g. ACEP)

LEVELS OF IMPACT OF RESEARCH RESULTS: 

LEVELS OF IMPACT OF RESEARCH RESULTS Level 3 impact: Research that leads to an actual alteration in clinical care provided. Findings leading to treatment changes Findings leading to alterations in patient behavior Level 4 impact: Research that leads to actual improvement in patient outcomes Basic Research  Efficacy  Improved outcome in the “real world”

AHCPR/AHRC REPORT CARD: 

AHCPR/AHRC REPORT CARD The first decade: Vast majority of OR: Only level 1 impact

APPROACH TO EMS OR: 

APPROACH TO EMS OR Still proponents: Narrow definition EMSOP’s opinion: Won’t work for EMS OR EMSOP’s approach: Integration among ALL appropriate research disciplines

EMSOP’s DEFINITON OF OR: 

EMSOP’s DEFINITON OF OR OER evaluates the impact of healthcare (including discrete interventions such as particular drugs, medical devices, and procedures, as well as broader programmatic or system interventions) on the outcomes of patients and populations. OER may include evaluation of economic impact linked to outcomes. OER emphasizes evaluations of care delivered in general, real-world settings; multi-disciplinary teams; and a wide range of outcomes. OER may entail any in a range of primary data collection methods and secondary (or “synthetic”) methods that combine data from primary studies. Mendelson-1998

INTEGRATION IN FUTURE EMS RESEACH: 

INTEGRATION IN FUTURE EMS RESEACH Utilize the strengths of classical OR Mining databases Meta-analysis Integrate with: Traditional clinical research methods Epidemiology Systems research

A BALANCED APPROACH TO EMS OR: 

A BALANCED APPROACH TO EMS OR The Outcomes Approach: Inexpensive source of knowledge about effectiveness Utilizing data collected routinely in EMS systems Using pooled information from published studies However, there is a paucity of: Rigorous systems research Clinical trials ESPECIALLY prospective, controlled trials

ENAMORED WITH RCT’s: 

ENAMORED WITH RCT’s Oversimplification to emphasize : Strengths of RCT’s Weaknesses of OR Why? Uncommon: RCT’s  Clinical practice Very rare: RCT’s  Clinical practice  Improved outcome proven Very common: Efficacious interventions never make it into practice Efficacious interventions  Proven INEFFECTIVE (Defibrillation in NY City)

PREHOSPITAL SETTING: 

PREHOSPITAL SETTING Great risk of efficacious interventions being ineffective Even if funding available for RCT’s, studies of effectiveness must occur in varied sizes and types of EMS systems. EMSOP urges a balanced view: Recognize strengths and limitations of each methodology Must accelerate knowledge transfer to the field

USING OR METHODS IN EMS RESEARCH: 

USING OR METHODS IN EMS RESEARCH In one sense…a perfect match EMS: The epitome of potential efficacy/effectiveness mismatch Can’t assume ANYTHING So…outcomes from “real world practice” MUST be studied Many large databases exist in many types and sizes of systems But…

WARNING #1: 

WARNING #1 Current databases grossly inadequate Vestigial Locally developed No consistent terminology No consistent definitions No national data Incomplete Inaccurate

If we’re going MINING…: 

If we’re going MINING… Challenge: Robust, usable databases Long history  Little progress Lacking: Accuracy Precision Comprehensiveness: Validated Risk Adjustment Measures Completeness: BP  Visual Analog Scale for SOB??? Mandatory: Innovative methods for obtaining RA data (automated?)

WARNING #2: 

WARNING #2 Poorly done database mining is a DISASTER Has led to MANY wrong conclusions MANY complaints by traditional disciplines are TRUE Remember the limitations: Mining databases can identify associations It is very efficient for hypothesis generation It cannot PROVE cause and effect A TRAGIC example

LONG HAIR PROTECTS FROM FATAL INJURY: 

LONG HAIR PROTECTS FROM FATAL INJURY Ron Maio, MD University of Michigan Herb Garrison, MD East Carolina University Journal of Irreproducible Results

LONG HAIR PROTECTS FROM FATAL INJURY: 

LONG HAIR PROTECTS FROM FATAL INJURY Hypothesis: A large database can be used to prove that increased hair length reduces injury severity and mortality. Participants: Patients entered into the trauma registries of two university-based Level I Trauma Centers Interventions: None.

METHODS: 

METHODS Prospective evaluation of the impact of hair length on injury severity and survival Inclusion: Mechanistic or physiologic criteria for Level I Trauma (ACS) Exclusion: Age <10 or bizarre hair cut IRB Dis-approval was obtained

METHODS: 

METHODS Measurement: Mean hair length Maio-Garrison Method: Ten lengths of hair Randomly yanked from the scalp Hairs without an intact follicle excluded Bad samples: Keep yanking Risk Adjuster: Injury Severity Score Final Outcome measure: Mortality

RESULTS: 

RESULTS Mean hair length: 18.27 centimeters. Unexplained Distribution: Bimodal Attempted to correlate hair length to: Ethnicity Income Ethanol levels Hair color Pre-event Quality of Life No correlations

RESULTS: 

RESULTS 500 patients entered “Short-hairs” “Long-hairs” Mean ISS similar 9.7 vs 9.3 (p = .73) Survival similar 94.3% vs 95.6% (p = .89)

MAJOR TRAUMA RISK: 

MAJOR TRAUMA RISK 350 (70%) were short hairs Risk of Short-hairs and Long-hairs being involved in major trauma National Association of Hair Salon Owners Study areas: Percentage of Short-hairs and Long-hairs in the general population nearly identical. Odds Ratio of Short-hairs suffering a major traumatic event: 2.33

CONCLUSIONS: 

CONCLUSIONS When involved in injury events, long-hairs and short-hairs have similar severity of injury and mortality Being short-haired is an incredibly high risk factor for being involved in major trauma (OR 2.33).

RECOMMENDATIONS: 

RECOMMENDATIONS Educational programs beginning in preschool to teach children the dangers of having short hair. Immediate post-injury education to discourage future hair cuts and thereby decrease recidivism Immediate legal sanctions against all hair cutting establishments and manufacturers of hair cutting instruments.

RECOMMENDATIONS: 

RECOMMENDATIONS A sin tax for getting a hair cut. Immediate establishment of an NIH study section to identify interventions that increase the rate of hair growth (e.g. minoxidil) NHTSA should promulgate regulations ensuring that these hair growth factors are placed in all public water supplies

RECOMMENDATIONS: 

RECOMMENDATIONS Passage of laws making it illegal to operate motor vehicles, motor cycles or bicycles with short hair Passage of laws making it illegal to wear bicycle or motorcycle helmets Law enforcement officers: Short-hairs vs. Long-hairs (“shovers”)

ASSOCIATION DOES NOT IMPLY CAUSE: 

ASSOCIATION DOES NOT IMPLY CAUSE What did they miss? The association between GENDER and HAIR LENGTH Embellished…but not much There is EVER the tendency to use associations to “prove” causality This has happened in REAL publications

MINING DATABASES: Beware of Selection Bias: 

MINING DATABASES: Beware of Selection Bias Analysis of Suicide rates in a county in the southern U.S. Study county: High percentage of Catholics in population Control county: High percentage of Protestants Suicide rate: 3X higher in “Catholic” county Conclusion: Being Catholic leads to higher likelihood of suicide

SUBSEQUENT REANALYSIS: 

SUBSEQUENT REANALYSIS What was the problem with the conclusion? No one actually identified the religion of suicide victims 70% of all suicides: Jumping from a high bridge The bridge was in the “Catholic” county Nearly half of the jumpers were residents of the “Protestant” county

WARNING #3: 

WARNING #3 Learn from the history of health research Large chasms between the disciplines Journals, conferences, associations sequestered and non-interactive Minimal crossover Terminology Expertise Information Findings

THE COST OF “RESEARCH SILOS”: 

THE COST OF “RESEARCH SILOS” Few researchers able to identify the best methodology for answering many questions: “Methodologic Tunnel Vision” Only use methods from the researcher’s own domain Problem: Method from another domain may be far superior for a given question

INTEGRATED APPROACH TO EMS RESEARCH: 

INTEGRATED APPROACH TO EMS RESEARCH All EMS care is rendered within the framework of a complex, interactive system THEREFORE Outcomes Research, Systems Research, Basic Research, Clinical Research, and Epidemiology…will always be interdependent…each being informed by the other

EMSOP’S APPROACH TO EMS RESEARCH: 

EMSOP’S APPROACH TO EMS RESEARCH Identifying effectiveness of an intervention: Requirements: Meaningful Conceptual Framework Approach that leads to broadly applicable conclusions Robust and pertinent specific methodology

EXAMPLE OF INTEGRATION: 

EXAMPLE OF INTEGRATION A way to study effectiveness in EMS: Conceptual Framework: Episode of Care Model Origin: Outcomes Research Assuring broad applicability: Multiple System Approach  Numerous system types and sizes Origin: Systems Research Specific Methodology: RCT Origin: Classical Clinical Research

The Episode of Care Model: 

The Episode of Care Model Prehospital Care ED Care Emergent Subspecialty Care Inpatient Care Follow-up Care Precipitating Event Long-term outcomes RA = Risk Adjustment Measures T=Therapeutic Intervention(s) OUT=Outcome Measure(s) RA T OUT RA T OUT RA T OUT RA T OUT RA T OUT Identifying the impact from each "unit of service"

IMPLICATIONS OF THE INTEGRATED APPROACH: 

IMPLICATIONS OF THE INTEGRATED APPROACH 1) No discipline “owns” a methodology RCT: Can identify efficacy AND effectiveness New asthma drug EFFICACIOUS in a pulmonary clinic Drug EFFECTIVE in numerous types of EMS systems Before-After Controlled Trial Traditional Clinical Research: Each INDIVIDUAL is their own control for testing an intervention Systems Research: A group of SYSTEMS being studied serve as their own control for testing an intervention

Prospective, Before-after System Trial: 

Prospective, Before-after System Trial Phase I Phase I(R) Phase II (Baseline) (Run-in) (Experimental Phase) Prospective risk adjustment and outcome measurements Begin training for new intervention Intervention given to all appropriate patients Analysis

EXAMPLE: OPALS: 

EXAMPLE: OPALS Prospective: 20 EMS systems Population: Patients presenting with SOB Phase I: BLS only Phase II: Systems add ALS Intubation Medications Results: Decreased MORTALITY Implications--Second EMS condition to be PROVEN effective with: Sound methodological approach Inclusion of MANY system types and sizes Broad applicability

IMPLICATIONS OF THE INTEGRATED APPROACH: 

IMPLICATIONS OF THE INTEGRATED APPROACH 2) Various methods can cross the “Six D’s” --Example: --A drug found to be effective at decreasing morbidity in the prehospital environment --Economic analysis: Not COST- effective in small EMS systems

CHALLENGES FOR THE FUTURE: 

CHALLENGES FOR THE FUTURE Get serious about GOOD EMS databases Methodology: Terminology Definitions Completeness of data Quality of data Develop and validate RA measures Available from many types/sizes of system Implied: Well-funded career researchers NATIONAL database

CHALLENGES FOR THE FUTURE: 

CHALLENGES FOR THE FUTURE Prevent the errors of the past Understand the limitations of OR Association/Cause Selection Bias GOOD Risk Adjustment No more research SILOS Researchers well versed across disciplines Sacrifice TURF for TRUTH

CHALLENGES FOR THE FUTURE: 

CHALLENGES FOR THE FUTURE Recognize the tendency to stagnate at level 1 research Level 1 work is mandatory, but not sufficient Don’t be impressed until OUTCOMES are affected

CHALLENGES FOR THE FUTURE: 

CHALLENGES FOR THE FUTURE Be TIRELESS in pursuing funding Science gets a freebie (PCN) about once a…CENTURY Level 4 OR in EMS will NOT occur without continuous, major funding The history of MAJOR research funding—Two prerequisites: An important health/societal issue TIRELESS lobbying by stakeholders

CHALLENGES FOR THE FUTURE (THE KEY): 

CHALLENGES FOR THE FUTURE (THE KEY) Rejection of the “Mount Everest” approach to prehospital care Why do we do what we do…???? No more acceptance of interventions just because they work in rats Why is this so important? If EMS PROFESSIONALS don’t REQUIRE good research…no one will PAY for good research

QUESTIONS: 

QUESTIONS ??????