GAO to send

Goals, Measures, Grants, Federal Role, Budget Office Role: 

Goals, Measures, Grants, Federal Role, Budget Office Role Shelley H. Metzenbaum smetzenbaum@verizon.net Performance-Focused Grant Management October 2005

Overview: 

Overview Why, How, Which Goals and Measures? Roles: OMB/Agency Budget Office/Federal Program Managers/States Accountability and What It Means?

The Key Questions:: 

The Key Questions: What is the objective? Not…”What is organization trying to do?” But… “What is organization trying to accomplish?” What works and is worth replicating? What does not work that needs fixing?

Basic Building Blocks : 

Basic Building Blocks Goals Measures Analysis Feedback Incentives

Why Use Goals & Measures?: 

Why Use Goals & Measures? Increase Accountability Improve Outcomes Strengthen Democracy

Accountability – What are you accountable for?: 

Accountability – What are you accountable for? Meeting a target? Making progress? More progress than other guy? Working honestly? Working hard? Working smart?

Accountability - What Do People Say? : 

Accountability - What Do People Say? “Better accountability!” Performance Budgeting, Program Justification Performance-based Pay Problem arises: “Someone needs to be held accountable!” Translation: Someone’s got to be punished.

Accountability – For What?: 

Accountability – For What? Doing the Right Thing! What citizens and elected leaders want: Outcome Emphasis; Right Outcomes Right target levels Trade-offs among competing targets Doing it Well -- Intelligently, Diligently, Honestly, Effective: Outcomes and Interim Outcomes Cost-Effective: Outcomes/Inputs Efficient: Outputs/Inputs; Timeliness

Accountability Problems:Slams & Scams, Blame & Claim: 

Accountability Problems: Slams & Scams, Blame & Claim Slams and Blame lead to: Measurement Manipulation – scams and shams Timid Targets – outcome avoidance, forgetting to stretch Omission of Key Outcome Indicators Attribution (Who Gets Credit?) Problem – claims and claim avoidance. (You are key to avoiding this problem!)

So how can government use goals and measures to - drive improved performance,  - increase accountability, and  - strengthen democracy without triggering dysfunctional reactions?: 

So how can government use goals and measures to - drive improved performance, - increase accountability, and - strengthen democracy without triggering dysfunctional reactions?

Goals: 

Goals

Goals:Specific, Challenging Goals Drive Improved Performance: 

Goals: Specific, Challenging Goals Drive Improved Performance Focus Energize Encourage Persistence Stimulate Discovery So help grant recipients set specific, challenging targets for Block Grant $ aligned with agency BG goal.

Specific Goals Clarify What Govt Will Do and What It Won’t: 

Specific Goals Clarify What Govt Will Do and What It Won’t Clean Charles 2005 -- Where: Lower Charles River When: 10 years How much: Swimmable Who: All Sources Invite public attention Public can respond

Goal Specification: 

Goal Specification Time Place Population Quantity Fed BG Role? Helping states and grantees understand how to set specific, challenging goals.

Why Use G & M:  Improve Outcomes  : 

Why Use G & M: Improve Outcomes EPA & Charles River – 19 to 59 % swimmable Next-day mail delivery rates – 79 to 94% NYC dirty sanitation sections - over 30% to zero

Goals Strengthen Democracy (Trade-Offs) : 

Goals Strengthen Democracy (Trade-Offs) New York - police crime rate and police abuse IRS - fairness or friendliness Forest Service - commerce, recreation, or habitat

How – Goals Enlist and Engage: 

How – Goals Enlist and Engage Shared objectives (DOT safety across agencies and levels of government) Communicate with citizens (Educ, Health, Voc.Ed.: inform citizen choices)

Key Characteristics of Effective Goals/Targets: 

Key Characteristics of Effective Goals/Targets Specific, challenging, but realistic Simple, resonant, understandable by public Outcome-focused, public Cascade down/roll up so org units and other govt bodies understand links Useless unless used

Implications for OMB and Agency Budget Offices: 

Implications for OMB and Agency Budget Offices PART gets it right, if interpreted right Help programs/states/locals understand: how goals work goals that are useful as described Use goals/targets to converse with Congress – “catch ball” (SSA)

Measures &AnalysisFor Detection, Diagnostics, Dissemination: 

Measures & Analysis For Detection, Diagnostics, Dissemination

Measures Motivate, Communicate, Illuminate: 

Measures Motivate, Communicate, Illuminate Motivate (Coast Guard, MD recycling, Healthy People 2010) Learn Find Problems Find Successes Inform Choices (home purchases, registry office/inspection station)

How Does Measurement Support Learning and Better Outcomes?: 

How Does Measurement Support Learning and Better Outcomes? DETECTION: Anomalies, direction change, speed change, patterns, etc. Measurement to trigger & focus follow-up, not always to assess perf. DIAGNOSTICS: Why? Reason for variation? ASSESSMENT (post action): Did it seem to work? Replicate, expand, or stop and fix?

Measurement to Support Learning to Achieve  Better Outcomes: 

Measurement to Support Learning to Achieve Better Outcomes DISSEMINATION: Share lessons re what worked and what did not. ASSUMPTION CONFIRMATION: Periodic loop-back to confirm If federal agencies don’t do this, who will?

Measures Support Evolving Notion of Accountability: 

Measures Support Evolving Notion of Accountability Ethical: Self-Dealing/Fraud (track $ flows to whom for what – accounting & fiscal audits) Fiscal: Allowed Spending/Low Waste (Efficient) Performance: Improve Social Outcomes/Reduce risk (Effective/Cost-effective)

DETECTION /DISSEMINATION: Find and Share Progress: 

DETECTION /DISSEMINATION: Find and Share Progress Recognize Progress ! Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes

DETECTION : Direction Change to Sense Problems: 

DETECTION : Direction Change to Sense Problems Why the Downturn ? Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes

Slide27: 

(10) Leak Checks (18-19) Controls (36i) Emergency Procedures Diagnostics: MA ERP Self-Audit Data 1997 & 1998 DETECT, DIAGNOSE, ASSESS: Self-Reported Practices Trends (97/98) MA ERP Lower than Other Areas: Problem? Changes over Time: Progress?

DETECT: Not Always Quantitative: 

DETECT: Not Always Quantitative

ASSESSMENT: Not Always QuantitativeConsolidated Tailings Pile (CO): 

ASSESSMENT: Not Always Quantitative Consolidated Tailings Pile (CO) Prior to 1989 After Reclamation 1999

DIAGNOSTICS/ASSESSMENT: Slice & Dice by Categories to Identify Cause of Problems, Progress: 

DIAGNOSTICS/ASSESSMENT: Slice & Dice by Categories to Identify Cause of Problems, Progress

DIAGNOSTICS: Complaint Relevance?: 

DIAGNOSTICS: Complaint Relevance? Source: PA DEP

DETECTION, DIAGNOSTICS, ASSESSMENT: Interim Outcome Indicators Ideas: 

DETECTION, DIAGNOSTICS, ASSESSMENT: Interim Outcome Indicators Ideas Attention Comprehension Beliefs/Attitudes Motivation Information Behavior Measure Awareness (OR) Measure Understanding Measure Attitudes Measure Motivation Measure Behavioral Changes (ERP)

Feedback -- Federal Value-Adding RoleReturn Data to Data Suppliers with More Value than When Received Gather, Organize, Analyze, Disseminate : 

Feedback -- Federal Value-Adding Role Return Data to Data Suppliers with More Value than When Received Gather, Organize, Analyze, Disseminate

Slide34: 

NHTSA: Learn by Assembling, Sharing Information (Y/N measurement)

Slide35: 

NHTSA: Organizing (Not Ranking) Triggers Better Follow-Up Questions

NHTSA: Learning with Controlled Experiments: 

http://www.nhtsa.dot.gov/people/injury/airbags/clickit_ticket/clickitcomposite/clickit_composite.pdf NHTSA: Learning with Controlled Experiments

Implications for OMB and Agency Budget Offices: 

Implications for OMB and Agency Budget Offices Help programs understand how measures work Help programs pick measures that are useful for: Detection Diagnostics (followed by treatment, program intervention) Assessment Dissemination Assumption Confirmation

Implications for OMB and Agency Budget Offices: 

Implications for OMB and Agency Budget Offices Use measures to inform Congress What is problem, national and local Implications: local measurement Where is progress? Implications: local and program measurement Most important! Use measures to help grantees/data suppliers: Find what works and is worth replicating Find what doesn’t, that needs fixing

Issues to Consider: 

Issues to Consider Data Accuracy: Match to need. Lower for Detection; Higher for Rules and Enforcement. Project and Measurement Scale can vary: Test marketing is good. Encourage it. Allow extrapolation. So is epidemiological approach. Efficiency and cost-effectiveness can also work at project level.

Incentives: 

Incentives

Bratton Accountability Principle: 

Bratton Accountability Principle “Nobody ever got in trouble because crime numbers on their watch went up. Trouble arose if the commanders didn’t know why the numbers were up and didn’t have a plan to address the problems.” OMB and Budget Office Expectations: Outcome-Focused Goal, Measurement Mastery, Strategy Cogency, Action, Interactive Inquiry

Accountability for What? Reward What?: 

Accountability for What? Reward What? Outcome Emphasis Measurement Mastery Strategy Cogency/Implementation Interactive Inquiry and Analysis Ongoing Search for Problems and Progress (Scanning and Experiments) Assumption Confirmation Continuous Learning/Lesson Application

Extrinsic Incentives for What?: 

Extrinsic Incentives for What?

Whom to Reward?: 

Whom to Reward?

Slide45: 

PART gets it right, if interpreted right… Performance over Punishment!

Accountability – For What?: 

Accountability – For What? Doing the Right Thing! Doing it Well -- Intelligently, Diligently, Honestly

Federal Role: 

Federal Role Assure Outcome-Focused Goal Setting Value-Adding Information Mgt. Gather, Organize, Analyze, Experiment, Disseminate Basement Border Guard (penalties) Direct Delivery Back Stop

Remember: : 

Remember: Learning, then Accountability Performance, not Punishment ! Useful Useful Useful!