Automated surveillance methods for notifiablediseases: theory, practice, and evaluation: Automated surveillance methods for notifiable diseases: theory, practice, and evaluation Matthew Samore, MD Utah Center of Excellence in Public Health Informatics Epicenter for Prevention of Healthcare Associated Infection Salt Lake VA Health Care System University of Utah


Objectives: Objectives Define situation awareness and situation comprehension in the context of public health surveillance Application of an inference engine to electronic health records for case identification Evaluation of impact on public health surveillance work processes


Outline: Outline Surveillance theory Surveillance practice Surveillance evaluation


Surveillance Theory: Surveillance Theory


Paradigms and Models : Paradigms and Models The Panopticon Archetype: “Total information awareness” Signal detection theory Trade-off between hit rate and false alarm rate Situation awareness Detection, diagnosis, prediction


What is situation awareness?: What is situation awareness? Comprehension of a dynamic environment


Understanding comprehension: Understanding comprehension A “phenomenon that emerges from an orchestra of cognitive processes” Perception Surface level “Eventbase” Semantic information extracted from perceptual input Situation model Integration of semantic information with prior knowledge Durso, et. al: Handbook of Applied Cognition


A tragic error: A tragic error “On a beautiful night in October 1978, in the Chesapeake Bay, two vessels sighted one another visually and on radar. On one of them, the Cost Guard cutter training vessel Cuyahoga, the captain saw the other ship up ahead as a small object on the radar, and visually he saw two lights, indicating that it was proceeding in the same direction as his own ship. He thought it possibly was a fishing vessel. Since the two ships drew together so rapidly, the captain decided that it must be a very slow fishing boat that he was about to overtake.


Slide9: At the last moment the captain of the Cuyahoga realized that in overtaking the supposed fishing boat, which he assumed was on a near-parallel course, he would cut off that boat’s ability to turn as both of them approached the Potomac River. So he ordered a turn to the port. This brought him directly in the path of the oncoming freighter, which hit the cutter. Eleven coastguardsmen perished.


The captain’s mental model: The captain’s mental model


Cell phone use during driving: Cell phone use during driving How it may affect comprehension of the dynamic environment of the road Fail to check rearview mirror (attention) At 4 way stop, may fail integrate spatial and temporal information to determine who has right of way and should enter intersection first May recognize that driver in oncoming vehicle is distracted but fail to infer the appropriate situation model, that the car will not stop


Situation awareness and public health : Situation awareness and public health Relevance The health of a population is indeed dynamic Time horizons cover a wide range The concept captures the data hungry nature of epidemiology and public health Current health status Current health care capacity Early detection


Limitations: : Limitations: Vaguely expressed as goal of surveillance Decisions or actions not specified More importantly: Description of the situation model is lacking


More effective use of the concept of situation awareness: More effective use of the concept of situation awareness Probe deeper Interpretation of current state Projected states Measurement Links theory to practice


Describing the current situation: Describing the current situation As given to computer scientist Yarden Livnat by Mary Hill, epidemiologist at Salt Lake Valley HD “There is significant flu in the valley” “Fever and coughs are rising” “12 people hospitalized throughout the valley” “20% are children” “Most cases are rapid A” “School absenteeism is high in middle/high schools” “Btw, the hunting season started yesterday”


Mental maps of epidemiologists: Mental maps of epidemiologists GI outbreak scenario Cluster of individuals with acute diarrhea who had eaten at a fast food restaurant Preliminary analysis: Approach to problem space varies substantially across public health personnel


Surveillance Practice: Surveillance Practice


Why syndromic surveillance (e.g., RODS) was perceived to be useful during Winter Olympic games (and other special events): Why syndromic surveillance (e.g., RODS) was perceived to be useful during Winter Olympic games (and other special events) At not very useful at other times


Thesis: Thesis Utility of syndromic surveillance depends on situation model Under normal conditions, its value is highly limited Current level of threat Suspicion of departure from normality


Public Health Practice: Public Health Practice Based Research


Core Principles: Core Principles Partnerships and synergies Active participation of health departments at local and state levels Public health leadership or co-leadership of projects Vision of public health research laboratory Interdisciplinary collaboration at local and national levels


Utah Department of HealthOffice of Public Health Informatics: Utah Department of Health Office of Public Health Informatics Directed by Wu Xu, PhD Goals Coordinate statewide eHealth initiatives Coordinate integration of projects Create a laboratory for applied public health research Develop theory and methods that advance information and population science in the context of public health practice


How CoE Projects Support Public Health: How CoE Projects Support Public Health Case Identification Case Investigation & Management Analysis Actions based on Results Data Dissemination RT-CEND Linkage projects DSIDE INTERACT Outbreak response, advisories, other actions Rx Drug Deaths Readmission/Mortality


Leveraging the electronic health record: Leveraging the electronic health record Real-Time Clinical Electronic Notifiable Disease Surveillance Electronic health record-based syndromic surveillance Text processing Leads Utah Department of Health Lisa Wyman, Melissa Stevens Dimond, David Jackson, Corona Nigatuvai, Robert Rolfs University of Utah Catherine Staes, Deepthi Rajeev Intermountain Healthcare Scott Evans, Per Gesteland VA Brett South, Matthew Samore, Adi Gundlapalli, Sylvain DeLisle


Data visualization and decision support: Data visualization and decision support Pathogen-specific surveillance (GermWatch) Heterogenic data visualization Public health decision support Interactive simulation Key investigators include: Per Gesteland, Carrie Byington, Andy Pavia, Adi Gundlapalli, Yarden Livnat, Frank Drews, Laverne Snow, Chris Barrett, Stephen Eubank, Madhav Marathe, Jim Koopman, Yong Yang, Robert Rolfs


RT-CEND Project: RT-CEND Project Health care system: Rule-based detection of notifiable diseases Message sending Electronic case transmission Health department Message receiving Integration into workflow at local and state level


Slide28: Electronic Medical Record Laboratory Data Data Driver Decision Support Engine Medical Logic Modules Code Tables 2 3 4 Alert File 5 1


Slide29: Alert File Time Driver Reportable Disease Monitor Electronic Medical Record ICP Daily Printout


Counts of notifiable diseases alerted: Counts of notifiable diseases alerted


Slide31: EMR Daily report NETSS faxed Manual entry Local Health Dept Intermountain HC State Health Dept CURRENT data flow Email or fax Alert file NETSS Manual entry


Slide32: EMR Daily report NETSS faxed Simplified view of NEW data flow Email or fax Alert file Study/ NEDSS Data views, reports, extracts NETSS Manual entry Manual entry HL7message Local Health Dept State Health Dept Intermountain HC


Electronic case transmission: Electronic case transmission Work led by Deepthi Rajeev, Catherine Staes, Scott Evans Compiled required data fields Modeled the HL7 message structure for reporting from healthcare systems to local & state health departments Evaluated existing messaging models, for instance PHIN implementation Allows transmission of multiple lab tests based on one or multiple specimens in a single message Message structure implemented


Planned work on health care system side: Planned work on health care system side Automated HIPAA documentation Chart review validation


Message receipt &workflow integration: Message receipt & workflow integration Appropriate data flow between state and local health departments Utah Department of Health Lisa Wyman, Corona Nigatuvai, David Jackson, Melissa Stevens Dimond, Robert Rolfs Salt Lake Valley Health Department Heath Harris, Mary Hill, Ilene Risk Davis County Health Department Brian Hatch, Nicole Stone


NEDSS implementation in Utah: NEDSS implementation in Utah Current status: Vendor: Collaborative Software Initiative Funded by grant from Novell Open source software model Agile, rapid development method underway


Enhancing situation comprehension: Enhancing situation comprehension Leveraging the electronic health record to support public health investigation Start with possible event of public health interest Use pre-defined and ad hoc queries on a health care system data warehouse to support evaluation Collect additional epidemiologic data: how are cases linked? Who, what, where, when


How “good” are electronic health record-based case criteria?: How “good” are electronic health record-based case criteria? May be better or worse than conventional criteria Conventional case criteria constitute a reference standard but are never a true gold standard Wit h respect to disease occurrence in a target population Sensitivity is virtually always less than 100% because of incomplete clinical evaluation and testing Lack of true gold standard with respect to


Surveillance Evaluation: Surveillance Evaluation


Conceptual framework: Conceptual framework Six core activities Detection Registration Confirmation Reporting Analyses Feedback. Public health action Acute (epidemic-type) responses Planned responses Mcnabb, et. al. , BMC Public Health, 2002


Evaluation of public health information system implementation: Evaluation of public health information system implementation Formative evaluation Semi-structured interviews Survey administration Information technology acceptance model Fit to workflow Observation of efficiency of work processes Measurement of timeliness, completeness, accuracy Assessment of the application development process


Time Line: Time Line


Synthesis: Synthesis


Tier 1: Event Detection: Tier 1: Event Detection Trade-offs Reliability versus validity of case criteria “False alarm rate” versus “hit rate” Our project activities Text processing Rash syndromes Rule development Central nervous system infections


Tier 2: Cognitive processes: Tier 2: Cognitive processes Our related research activities Center of Excellence in Public Health Informatics Public health decision-making Simulation-based decision support


Tier 3: Broader Public Health Goals: Tier 3: Broader Public Health Goals Our related research activities Informatics Information exchange Data linkage Epidemiology Epidemic models Valid ecologic inference


Slide47: Syndromes Sensors Pathogen- specific Notifiable diseases Local/state health dept- centered activities Evaluation of existing systems (ELR, RODS, others) New system implementation BioSense NEDSS RT-CEND Tier 4: System Performance