program evaluation planning

PLANNING A PROGRAM EVALUATION : 

PLANNING A PROGRAM EVALUATION NOREBEL ARAZO BALAGULAN PhD – EPM MUST, CdeO City

PROGRAM EVALUATION : 

PROGRAM EVALUATION the systematic investigation of a project or program to determine its worth or merit the thoughtful process of focusing on questions and topics of concern, collecting appropriate information, and then analyzing and interpreting the information for a specific use and purpose 9/4/2010 2 Norebel A. Balagulan - PhD-EPM

PROGRAM EVALUATION : 

PROGRAM EVALUATION a systematic method for collecting, analyzing, and using information to answer basic questions about projects, policies and programs 9/4/2010 3 Norebel A. Balagulan - PhD-EPM

PROGRAM EVALUATION : 

PROGRAM EVALUATION the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy 9/4/2010 4 Norebel A. Balagulan - PhD-EPM

Program Evaluation Plan : 

Program Evaluation Plan a blueprint or a map for an evaluation. details the design and the methods that will be used to conduct the evaluation and analyze the findings. must be a well thought out plan 9/4/2010 5 Norebel A. Balagulan - PhD-EPM

Warning: : 

Warning: You should not implement an evaluation until you have completed an evaluation plan. 9/4/2010 6 Norebel A. Balagulan - PhD-EPM

Approaches to Program Evaluation Planning : 

Approaches to Program Evaluation Planning * Participatory Approach * Traditional Approach * Visual Approach 9/4/2010 7 Norebel A. Balagulan - PhD-EPM

Steps in Planning a Program Evaluation(Participatory Approach) : 

Steps in Planning a Program Evaluation(Participatory Approach) I. Pre-Planning Meetings (Evaluation Coordinating Group) * Define evaluation goal and objectives - What are you going to evaluate? - What is the purpose of the evaluation? 9/4/2010 8 Norebel A. Balagulan - PhD-EPM

Steps ... : 

Steps ... I. Pre-Planning... * Define evaluation goal and objectives - Who will use the evaluation? - How will they use it? 9/4/2010 9 Norebel A. Balagulan - PhD-EPM

Steps in Planning a Program Evaluation(Traditional Approach) : 

Steps in Planning a Program Evaluation(Traditional Approach) I. Pre-Planning Meetings... * Identify evaluation team members - Who are qualified to evaluate? - Are these individuals involved in the different levels of program implementation? 9/4/2010 10 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... I. Pre-Planning Meetings... * Plan logistical and administrative arrangements - When is the evaluation needed? - What resources do you need – time, money, people? 9/4/2010 11 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... I. Pre-Planning Meetings... * Develop visual framework of the program - chart – defines the scope of the evaluation - includes: program goal, objectives, activities, concepts 9/4/2010 12 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... I. Pre-Planning Meetings... * Orient evaluation planning workshop facilitators - discuss the topics which every member shall handle - review the things to be done 9/4/2010 13 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... II. Evaluation Planning Workshop * Organize Project stakeholders into an effective team - develop spirit of camaraderie - develop a sense of team membership and mission - exercise on team building 9/4/2010 14 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... II. Evaluation Planning... * Define evaluation question - What questions will the evaluation seek to answer? - Does the evaluation respond to the concerns of the program manager and stakeholders? 9/4/2010 15 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... II. Evaluation Planning ... * Identify data collection techniques - What sources of information will you use? - What data collection method(s) will you use? - What collection procedures will you use? 9/4/2010 16 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... II. Evaluation Planning ... * Develop data collection instruments - interview guide - observation guide 9/4/2010 17 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... II. Evaluation Planning ... * Finalize sample of data collection sites and interviewees - How many persons are going to be interviewed? - When is the collection of data? 9/4/2010 18 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... III. Fieldwork: Preparation, Data Collection and Analysis * Prepare fieldwork teams - logistics and expectations of the evaluation team members - in-depth interviewing skills of the team members (KSA) 9/4/2010 19 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... III. Fieldwork ... * Conduct interviews and observations - data collection schedule - conducting group interviews - mini-feedback session for the team 9/4/2010 20 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... III. Fieldwork ... * Analyze information collected - How will the data be analyzed? - How will the information be interpreted? 9/4/2010 21 Norebel A. Balagulan - PhD-EPM

Steps... : 

Steps... III. Fieldwork ... * Summarize fieldwork findings - Each evaluation question – team’s integrated evaluation findings - summary statements – concise and clearly worded 9/4/2010 22 Norebel A. Balagulan - PhD-EPM

Steps ... : 

Steps ... IV. Workshop to formulate lessons learned * Formulate lessons learned for each evaluation question - What worked well? (effective or positive aspects) - What needs improvement? (problematic aspects) 9/4/2010 23 Norebel A. Balagulan - PhD-EPM

Steps ... : 

Steps ... IV. Workshop ... * Team assessment of the evaluation process - What were the problems met? - Sharing of insights - Suggestions 9/4/2010 24 Norebel A. Balagulan - PhD-EPM

Steps ... : 

Steps ... V. Summarize evaluation results * Summarize evaluation findings and lessons learned - give concise description of what transpired - organized around the major elements in the program framework 9/4/2010 25 Norebel A. Balagulan - PhD-EPM

Steps ... : 

Steps ... VI. Development of an Action Plan * Development of an Action Plan based on evaluation findings - should include the main program objectives, components, activities and strategies to modify or integrate 9/4/2010 26 Norebel A. Balagulan - PhD-EPM

Steps ... : 

Steps ... VII. Finalization, dissemination and discussion of the evaluation result * Write evaluation result - focus on reporting the findings and recommendations - should include a 1-2 pages executive summary 9/4/2010 27 Norebel A. Balagulan - PhD-EPM

Steps ... : 

Steps ... VII. Finalization ... * Distribute and discuss evaluation results with program collaborators/ stakeholders - prepare different versions of the result for different readers 9/4/2010 28 Norebel A. Balagulan - PhD-EPM

Steps in Planning a Program Evaluation( Visual Approach ) : 

Steps in Planning a Program Evaluation( Visual Approach ) Build a visual program logic (outcomes model). Include justifying evidence. Visually map indicators onto the model. 9/4/2010 29 Norebel A. Balagulan - PhD-EPM

Steps in Planning a Program Evaluation( Visual Approach ) : 

Steps in Planning a Program Evaluation( Visual Approach ) Visually map evaluation questions onto the model. Set up your evaluation projects based on the evaluation questions they answer. Include evaluation management information. 9/4/2010 30 Norebel A. Balagulan - PhD-EPM

Steps in Planning a Program Evaluation( Visual Approach ) : 

Steps in Planning a Program Evaluation( Visual Approach ) 7. Use the evaluation plan to discuss evaluation planning with decision-makers. 8. Use the same visual evaluation plan to control evaluation implementation. 9. Report on, and present the evaluation plan. 10. Share templates of your evaluation plans. 9/4/2010 31 Norebel A. Balagulan - PhD-EPM

Slide 32: 

9/4/2010 32 Norebel A. Balagulan - PhD-EPM

Steps in Program Evaluation Plan( Traditional Approach) : 

33 Steps in Program Evaluation Plan( Traditional Approach) Step 1: Engage Stakeholders Step 2: Describe the Program Step 3: Focus the Evaluation Design 9/4/2010 Norebel A. Balagulan - PhD-EPM

Steps in Program Evaluation Plan( Traditional Approach) : 

34 Steps in Program Evaluation Plan( Traditional Approach) Step 4: Gather Credible Evidence Step 5: Justify Conclusion Step 6: Ensure Use and Share Lessons Learned 9/4/2010 Norebel A. Balagulan - PhD-EPM

Slide 35: 

9/4/2010 Norebel A. Balagulan - PhD-EPM 35 CHECKLIST FOR PROGRAM EVALUATION PLANNING Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC Researched by: NOREBEL A. BALAGULAN Name of Organization: Name of Program: Purpose of Evaluation?What do you want to be able to decide as a result of the evaluation? For example: __ Understand, verify or increase impact of products or services on customers/clients (eg, outcomes evaluation) __ Improve delivery mechanisms to be more efficient and less costly (eg, process evaluation) __ Verify that we're doing what we think we're doing (eg, process evaluation) __ Clarify program goals, processes and outcomes for management planning __ Public relations __ Program comparisons, eg., to decide which should be retained __ Fully examine and describe effective programs for duplication elsewhere __ Other reason(s)

Slide 36: 

9/4/2010 Norebel A. Balagulan - PhD-EPM 36 Audience(s) for the Evaluation?Who are the audiences for the information from the evaluation, for example: __ Clients/customers __ Funders/Investors __ Board members __ Management __ Staff/employees __ Other(s) _________________________________________________ What Kinds of Information Are Needed?What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences, for example, information to understand:   __ The process of the product or service delivery (its inputs, activities and outputs) __ The customers/clients who experience the product or service __ Strengths and weaknesses of the product orservice __ Benefits to customers/clients (outcomes) __ How the product or service failed and why, etc. __ Other type(s) of information?

Slide 37: 

9/4/2010 Norebel A. Balagulan - PhD-EPM 37 Type of Evaluation?Based on the purpose of the evaluation and the kinds of information needed, what types of evaluation is being planned? __ Goal-based? __ Process-based? __ Outcomes-based? __ Other(s)? ___________________________________________________ Where Should Information Be Collected From? __ Staff/employees __ Clients/customers __ Program documentation __ Funders/Investors __ Other(s) ________________________________________________

Slide 38: 

9/4/2010 Norebel A. Balagulan - PhD-EPM 38 How Can Information Be Collected in Reasonable and Realistic Fashion? __ questionnaires __ interviews __ documentation __ observing clients/customers __ observing staff/employees __ conducting focus groups among___________________________ __ other(s) When is the Information Needed?_______________________________________________________ _______________________________________________________ What Resources Are Available to Collect the Information? _______________________________________________________ _______________________________________________________

Slide 39: 

SAMPLE TEMPLATE FOR PROGRAM EVALUATION PLAN 9/4/2010 39 Norebel A. Balagulan - PhD-EPM

Slide 40: 

9/4/2010 40 Norebel A. Balagulan - PhD-EPM

Slide 41: 

9/4/2010 41 Norebel A. Balagulan - PhD-EPM

“Research seeks to prove, Evaluation seeks to improve...” - M. Q. Patton : 

“Research seeks to prove, Evaluation seeks to improve...” - M. Q. Patton 9/4/2010 42 Norebel A. Balagulan - PhD-EPM

If the Goal of Evaluation is… : 

43 If the Goal of Evaluation is… … to improve a program Then no evaluation is good unless findings are used to make a difference

References: : 

References: http://www.the-aps.org/education/promote/promote.html http://www.netc.org/planning/eval/ http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/chapter_2_pmguide.html 9/4/2010 44 Norebel A. Balagulan - PhD-EPM

References: : 

References: http://www.cdc.gov/healthyyouth/evaluation/resources.html http://en.wikipedia.org/wiki/Program_evaluation  http://gsociology.icaap.org/methods/evaluationbeginnersguide.pdf http://www.programevaluation.org/outcomeplanner.htm 9/4/2010 45 Norebel A. Balagulan - PhD-EPM