Evaluation ED

Evaluation, the Government Performance and Results Act, and the ED 524B Reporting FormFiscal Year (FY) 2005ERCM Grantee MeetingJanuary 11, 2005: 

Evaluation, the Government Performance and Results Act, and the ED 524B Reporting Form Fiscal Year (FY) 2005 ERCM Grantee Meeting January 11, 2005 U.S. Department of Education Office of Safe and Drug-Free Schools Tara Hill Sara Strizzi Michelle Sinkgraven

Session Objectives: 

Session Objectives Review the importance of evaluation Discuss key evaluation components Discuss creating performance measures and an evaluation plan Highlight the Emergency Response and Crisis Management (ERCM) Government Performance and Results Act (GPRA) indicators for FY 2005 grantees Explain how to collect data and report on the GPRA indicators on the ED 524B form

Why do we Evaluate ERCM Projects?: 

Why do we Evaluate ERCM Projects? Because Evaluation.... Is a way to gather school-based and community data to help improve crisis plans Uncovers new information or consequences that were not anticipated Involves multiple stakeholders in the process and creates buy-in for crisis planning

What Evaluation DOES...: 

What Evaluation DOES... Determines if a project is accomplishing the performance objectives Supports decision-making in the district and community Communicates to stakeholders the benefits/purpose of the plan Builds a knowledge base about what does and does not work in an ERCM plan Helps the Department of Education report on progress

What Evaluation Does NOT Do...: 

What Evaluation Does NOT Do... Attempt to judge a program subjectively Aim to represent a false picture of program success for administrators or funders

Creating an ERCM Evaluation Plan: Key Components: 

Creating an ERCM Evaluation Plan: Key Components Key Questions to Consider Who will conduct your evaluation? Who are the key stakeholders? Have you captured the appropriate data at the beginning of your project? What are your key project objectives?

Reviewing Project Objectives: 

Reviewing Project Objectives ERCM Grant Objectives Government Performance and Results Act Indicators Project-Specific Objectives and Performance Measures Drawn from needs assessment As reflected in grant applications Based on individual school and district issues

Project-Specific Objectives & Performance Measures: 

Project-Specific Objectives & Performance Measures Project Objective: A specific, measurable statement about what the project will achieve. Hint: Start the objective with the word "To" followed by a verb. For example, "To increase the number of bus drivers trained in CPR from 5 bus drivers to 25 bus drivers.

Project-Specific Objectives & Performance Measures, Cont.: 

Project-Specific Objectives & Performance Measures, Cont. Performance Measure/Indicator: A value or characteristic that can be used to determine the extent to which the objective has been achieved. For example, the # of bus drivers trained in CPR between October 1, 2006-March 31, 2007.

Government Performance and Results Act (GPRA) : 

Government Performance and Results Act (GPRA) What is GPRA? Purpose of the GPRA ED's Response to GPRA GPRA Measures for the ERCM Program Measure 1 Measure 2 Measure 3

What is GPRA?: 

What is GPRA? The Government Performance and Results Act of 1993 requires all federal agencies to manage activities with attention to outcomes. Agencies must: State intended accomplishments Identify the resources required Periodically report to Congress

Purpose of the GPRA: 

Purpose of the GPRA Continuous monitoring of program outcomes results in: Improved accountability for expenditures using public funds Informed Congressional decision making through use of objective information Government focus on results

US Department of Education's Response to GPRA: 

US Department of Education's Response to GPRA Development of a strategic plan for 2002-2007 Plan reflects the Department's goals and priorities Department goals: Goal 1: Create a Culture of Achievement Goal 2: Improve Student Achievement Goal 3: Develop Safe Schools and Strong Character Goal 4: Transform Education into an Evidence-based Field Goal 5: Enhance the Quality of and Access to Postsecondary and Adult Education Goal 6: Establish Management Excellence

What are the three Performance Measures for ERCM?: 

What are the three Performance Measures for ERCM? Measure 1: Demonstration of increased number of hazards addressed by the improved school emergency response plan as compared to the baseline plan; Measure 2: Demonstration of improved response time and quality of response to practice drills and simulated crises; and   Measure 3: A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by the district and community partners beyond the period of Federal financial assistance.

Measure 1: 

Measure 1 Demonstration of increased number of hazards addressed by the improved school emergency response plan as compared to the baseline plan.

Measure 1: Increased Number of Hazards: 

Measure 1: Increased Number of Hazards Definitions: Hazards: Broadly defined as potential threats to the school / school district whose date or time of occurrence cannot be predicted. Examples include: Terrorist attack Tornado Earthquake Chemical spill Unlawful entry by unlawful person Criminal activity Addressed: Written plan is in place that responds to hazards Baseline plan: ERCM plan in place prior to grant award date

Data Needed for Measure 1: 

Data Needed for Measure 1 Number of hazards addressed by the improved school emergency response plan Number of hazards addressed by the baseline plan Measure can include information on number of hazards at the district level and at the school level

Completing the 524B: 

Completing the 524B Sample scenario: ABC School District had a basic ERCM plan in place prior to the grant that addressed fires, tornadoes, and unlawful entry. At the end of the grant, the district addressed three additional hazards: terrorist attack, bomb threat, and active shooter.

Completing the 524B: 

Completing the 524B Box 1.a. Performance Measure: Number of hazards addressed by the improved school emergency response plan as compared to the baseline plan. Measure Type: GPRA Quantitative Data: Step 1: Enter 999 in all data fields Step 2: Provide numerical answers to each of the following in the "Explanation of Progress" section: Number of hazards addressed in original (baseline) plan Number of hazards addressed in final plan Number of additional hazards addressed by final plan Formula: # of hazards in final plan - # of hazards in baseline plan

Completing the 524B Form, Continued: 

Completing the 524B Form, Continued Additional information to include in the "Explanation of Progress" section: Provide a narrative that explains what the numbers mean and how they were calculated. Be specific and list additional hazards that have been addressed. Describe any barriers that prevented you from meeting your goal, if applicable.

FAQs: 

FAQs Can I count facilities improvements as hazards that my school/school district has addressed? What if our district did not include funds to address hazards that were uncovered in our vulnerability assessment? Do I need to report school-by-school data on the 524B? My application is for a consortium of school districts. How should I report data for this measure—do I aggregate or report on a district-by-district basis?

Measure 2: 

Measure 2 Demonstration of improved response time and quality of response to practice drills and simulated crises.

Measure 2: Improved Response Time and Quality of Response : 

Measure 2: Improved Response Time and Quality of Response Definitions: Response Time: The number of minutes and seconds it takes to respond to a practice drill or simulated crisis. The type of response depends on the type of drill staged. Drills include, but are not limited to: Fire drill Lockdown Community-wide emergency response drill Hazardous weather drill Active shooter Shelter-in-place Evacuation

Measure 2: Improved Response Time and Quality of Response: 

Measure 2: Improved Response Time and Quality of Response Definitions, continued: Demonstration of improved response time: A quantifiable decrease in the amount of time it took to respond to a practice drill or simulated crisis that was staged at the beginning of the performance period and the end of the performance period. Quality of response: To respond to this portion of the measure, attach a copy of the after-action report for each drill and clearly state what has been done to make improvements in quality.

Data Needed for Measure 2: 

Data Needed for Measure 2 Response time: Response time to drill conducted at the beginning of the performance period. Response time (to same type of drill) at the end of the performance period. Quality of response: A thorough narrative description of the drills and a plan for addressing any shortcomings. Attach an after-action report for each drill.

Completing the 524B: 

Completing the 524B Sample scenario: ABC School District conducted a tornado drill. An after-action report highlighted confusion about designated safe areas. In response to the report, students were briefed and shelter-in-place signs were hung around school. A few months later, a second tornado drill was conducted. The after-action report attributed the 3 minute, 16 second reduced response time to the changes implemented from the lessons learned in the first (baseline) drill.

Completing the 524B: 

Completing the 524B Box 1.a. Performance Measure: Demonstration of improved response time and quality of response to practice drills and simulated crises. Measure Type: GPRA Quantitative Data: Step 1: Enter 999 in all data fields Step 2: Provide numerical answers to each of the following in the "Explanation of Progress" section: Actual time for drill #1 (baseline drill) Goal time for drill #2 Actual time for drill #2 Difference between drill time #1 and drill time #2

Completing the 524B, continued: 

Completing the 524B, continued Additional information to include in the "Explanation of Progress" Section: Provide a narrative for the data provided, explaining what the numbers mean and how they were calculated Be specific, include information about what went right and wrong and how times were calculated Provide the rationale for the drill #2 goal and when it was established (I.e. upon completion of drill) Describe any barriers that prevented you from meeting the goal for drill #2, if applicable Include a qualitative analysis of each drill, such as an after-action report

FAQs: 

FAQs Do I need to compare similar types of drills for reporting purposes? Does my first (baseline) drill need to occur during the grant performance period or can we use data from a prior drill or simulation? What if our grant application specifies running a variety of drills only once? Can I report on community-wide crisis drills?

Measure 3: 

Measure 3 A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by the district and community partners beyond the period of Federal financial assistance.

Measure 3: Plan for and Commitment to Sustainability and Continuous Improvement: 

Measure 3: Plan for and Commitment to Sustainability and Continuous Improvement Definitions: Plans: The Emergency Response and Crisis Management (ERCM) plan developed under this grant.

Data Needed for Measure 3: 

Data Needed for Measure 3 Provide the answer to the following question: "Do we have a plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance?" A simple "yes" or "no" is not sufficient. A copy of the plan should be provided with your final report. In addition, an objective analysis of whether or not the plan meets this measure should be provided.

Completing the 524B: 

Completing the 524B Box 1.a. Performance Measure: A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance. Measure Type: GPRA Quantitative Data: Target, Raw Number: YES Actual, Raw Number: YES, if your district has met this standard. If not, NO.

Completing the 524B: 

Completing the 524B Explanation of Progress: Provide a detailed answer to the question, "Do we have a plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance?" Provide proof of your plan. This could include, but is not limited to, the following: Copies of updated Partner Agreements Timelines/agenda for upcoming community partner meetings/drills after the grant period is over Your district's completed crisis plan

FAQs: 

FAQs Do I need to submit a copy of my district's completed crisis plan with my final report? What are some ways a district might demonstrate sustainability? Do I need to submit signed copies of all MOU's again at the conclusion of the grant period?

QUESTIONS???: 

QUESTIONS???