Metzenbaum Shelley

No Longer Flying Blind: 

No Longer Flying Blind Progress and Promising New Paths in Performance Measurement Environmental Compliance Consortium, www.complianceconsortium.org Facility Reporting Project www.facilityreporting.org Shelley Metzenbaum, April 2004, smetzenbaum@aics.net

Two Topics: 

Two Topics Facility Reporting Project (FRP): measurement for facilities Internal assessment and diagnostics, external analysis and accountability: performance track, performance contracts, EMSs, benchmarking, diagnostics, int’l mgt. www.facilityreporting.org Environmental Compliance Consortium: measurement for government Assessment and diagnostic measurement avail now for compliance assistance programs, enforcement, compliance info www.complianceconsortium.org

I. Facility Performance Reporting: 

I. Facility Performance Reporting Promising New Path: the Facility Reporting Project

Emerging/Promising Practices Among Regulated Parties: 

Emerging/Promising Practices Among Regulated Parties Attitude transition from opposition to management to strategic opportunity (Hoffman) Reputational concerns: better community relations (e.g., Shell) Leading to: EMS adoption ISO certification Balanced Scorecard with “beyond finance” measurement

Government Response to Regulated Community Change: 

Government Response to Regulated Community Change Performance Track, Green Tier, performance contracts, EMS as permit alternative, EMS mandate in compliance agreements, sectoral agreements But… Little public reporting from these mechanisms Hard to assess/learn from non-standardized info

The Facility Reporting Project (FRP): 

The Facility Reporting Project (FRP) The FRP is a multi-stakeholder initiative developing a: Generally accepted, Credible, Comparable, and Consistent facility-level environmental and social sustainability reporting framework.

Guiding principles: 

Guiding principles The framework will be: Legitimized by a multistakeholder and broadly consultative development process Informed by best current knowledge and practice Useful to the public, the private sector & gov’t A product that improves and evolves as experience is gained with facility-level reporting US-based to simplify initial implementation Upward compatible with the GRI

What We Have Accomplished: Benchmarking/Baseline Analysis: 

What We Have Accomplished: Benchmarking/Baseline Analysis www.facilityreporting.org

Implementation Steps: 

Implementation Steps Benchmark Research on the “state of facility reporting” Constitute Multi-Stakeholder Steering Panel Develop reporting framework via a multistakeholder process Communications program/stakeholder engagement; facility beta-testing; revisions Institutionalize Formal endorsements, institutional transition, ongoing evaluation and improvement

Status: 

Status Discussion draft out to “Fast Feedback” group (done) Comments discussed at late October Steering Panel meeting (done) Global Reporting Initiative MOA (in process) Public Exposure draft (May 15, 04) Combined hearing re FRP and MSWG Extl Value EMS (summer 04) Revise and pilot test (Fall 04)

II. Government Performance Reporting: 

II. Government Performance Reporting Progress: Advancing Positive Outcomes

What Questions Do We Want Govt Measurement to Answer?: 

What Questions Do We Want Govt Measurement to Answer? What problems need attention? Did an intervention work – raise compliance levels and improve environmental quality? Did a compliance assistance program raise compliance levels, lessen env. impacts, increase env. quality? Which interventions/actions work best and warrant replication? Which don’t work, so should be curtailed? Informed public, so know if govt has right priorities and reasonably trying to address them

Regulatory Objectives: 

Regulatory Objectives Making Good Things Happen Human and Environmental Health Compliance Adoption of Good Practices Awareness of Legal Obligations Preventing Bad Things from Happening Harmful incidents Accidents Bad Practices Pre-cursor events

Fresh & Frequent Ambient Data Rich CA Measure – Diagnostic: When and where problems occur/whether intervention worked: 

Fresh & Frequent Ambient Data Rich CA Measure – Diagnostic: When and where problems occur/whether intervention worked Charles River – MA/EPA R1

Discharge/Emissions Trends Valuable CA/P2 Measure when:: 

Discharge/Emissions Trends Valuable CA/P2 Measure when: Date of CA also tracked and shown (for small project) Date of Compliance Initiative Tons/year CO Emissions

Compliance Rates Valuable CA/P2 Measure (Program) if:: 

Compliance Rates Valuable CA/P2 Measure (Program) if: Also Show Inspection Data Fresh & Frequent Date of Initiative

Slide17: 

(10) Leak Checks (18-19) Controls (36i) Emergency Procedures Diagnostics: MA ERP Self-Audit Data 1997 & 1998 Self-Reported, Categorized Compliance Trends Valuable MA ERP

“Critical Compliance Indicators” Valuable CA Measure when:: 

“Critical Compliance Indicators” Valuable CA Measure when: Fresh and Frequent (enough) Statistically valid or large percentage of total NH, CO

Survey Data Valuable CA Measure: 

Survey Data Valuable CA Measure Awareness of regulatory obligations Awareness of other enforcement actions (pre-req. for deterrence effect) Awareness of best practices not mandated by law Relatively low cost, very informative Oregon

II. Government Performance Reporting: 

II. Government Performance Reporting (b) An Additional Path Forward: Preventing Negative Outcomes

Measure Unwanted Incidents to Manage Them Down: 

Measure Unwanted Incidents to Manage Them Down

Lessons from Other Fields – Big/Common to Small/Rare: Fire: 

Lessons from Other Fields – Big/Common to Small/Rare: Fire Every major fire studied for key characteristics (cost of damage, injuries, cause of fire, building materials, etc.) Data base created and studied Cooperation with insurance industry Patterns identified Control methods tested Model codes created Model codes adopted Firefighters now primarily EMTs

Lessons from Other Fields: (Reduce Level of Harm)Transportation Accidents: 

Lessons from Other Fields: (Reduce Level of Harm)Transportation Accidents Percentage of Fatalities Related to Speeding (top) and Alcohol (bottom), 1999 - 2000 Fatality Rates/ 100 mm 50 % 30 %

Lessons from Other Fields: Flight, Medical, Food Safety: 

Lessons from Other Fields: Flight, Medical, Food Safety Pilots set up NASA near-miss tracking program Identifies problematic patterns, while it Protects those being measured with anonymity. VA replicated system to reduce medical errors. Post-hoc plane accident analysis to look for possible causes Tarmac incursion program to prevent pre-cursor events FDA centralized adverse event reporting system

PA Complaint Analysis: An Adverse Event Reporting System: 

PA Complaint Analysis: An Adverse Event Reporting System

What were they looking for?: 

What were they looking for? Common Occurrences: Organize and Analyze Data to Find It Anomalies in Data Patterns: They Trigger Focused Follow-Up Questions Leading to Problem and Solution Identification Contributing (explanatory, pre-cursor) factors for future tracking and prevention Correlations

Possible Applications in Environmental Protection: 

Possible Applications in Environmental Protection Accidents (Wharton, TX A&M, LEPCs) Study/track frequency of past accidents Study/track characteristics with post-hoc analysis Excedences Non-compliance characteristics Practices Professional certification Complaints (adverse event reporting system) Other?

Effective Measurement Systems emphasize:: 

Effective Measurement Systems emphasize: Learning/diagnostics over accountability Performance over punishment Create “Safe” Measurement Systems for those Being Measured, so will not implode system Pilots: Personal Stake Third-party reporting without attribution Veteran’s Admin hospital staff Workers

Caveats:: 

Caveats: Don’t Get Tied up in the Quest for Perfect Information Tie data accuracy level to intended use Diagnostics < Evaluation < Rule-writing and enforcement (a suggested hierarchy): Need Better Diagnostic Measurement Systems Don’t worry about claiming credit, It will get in your way. If performance linked to rewards, create teams and reward teams. Better yet, don’t create direct links between rewards and performance measures. Use reputational motivation.

Accountability for what?: 

Accountability for what? Relentless attention to outcomes Mastery of measurement information Cogent strategies Intelligent experimentation Continual analysis of experience to guide future action

Time to Stop Flying Blind : 

Time to Stop Flying Blind No reason to run a compliance assurance program without outcome measures – outcomes attained and harms prevented Without measurement, like throwing darts without being able to see if hit the target. Why bother?