Monitoring and Evaluation lecture By Sayed Aziz Jamalzai

Understanding Evaluation:

1 Understanding Evaluation 1. Achievement Seeing what has been achieved 2. Measuring Progress In accordance with the objectives of program 3. Improving Monitoring For better management 4. Identifying Strengths & Weaknesses To strengthen the program and remove weaknesses 5. Seeing If Effort Was Effective What difference has the program made? 6. Cost Benefit Were the costs reasonable? 7. Collecting Information for better plan and management of program activities 8. Sharing Experience To prevent others making similar mistakes, or to encourage them to sue similar methods 9. Improving Effectiveness To have more impact 10. Allow for bitter planning More in the line with the needs of people, especially at community level Why Evaluate?

Evaluation is carried out mainly as a way of looking at:

2 Evaluation is carried out mainly as a way of looking at Program Activities Human Resources Material Resources Information Facts and Figures In order to Monitor Progress and Effectiveness Consider Costs and Efficiency Show Where Changes were Needed And Help to Plan More Effectively for The Future

Some other reasons for evaluating:

3 Some other reasons for evaluating What Evaluation can (and cannot) do? (Evaluation is not the medicine to cure all ills) Looking at success and failure From Experience: The Brazil Case

Slide 4:

4 Quantitative Evaluations: Number of people involved in the program, Number of products or services produced, Amount of material resources available or required, Financial cost of running a program or the extent of an area cultivated. Qualitative Evaluations: People’s behavior, abilities, qualities, attitudes, values, and motivations; Evaluation looks at quantity and quality

Who Can Evaluate:

5 Who Can Evaluate Can take a fresh look to the program No personally involved so its easier to be objective Is not a part of normal power structure Gains noting from program but may gain prestige from evaluations Trained in evaluation methods, may have experience evaluations As an outsider may not understand the program and people May cause anxiety as program staff and participant are not sure his or her motives Knows the program only too well Finds hard to be objective Is a part of power and authority structure My be motivated by hopes of personal gain May not be specially trained except little training Familiar with people and understand the program can interpret social behavior and attitude Known the program, no anxiety of threat Internal Evaluator External Evaluator

Participating in Evaluation:

6 Participating in Evaluation 1. The Studying Specimens’ type of evaluation People are only expected to play a small part in evaluation Examined, Questioned, etc 2. Refusing to Share Result’ type of evaluation People take a small part At the end people receive chosen feedback

3. The looking up the expertise type of evaluation:

7 3. The looking up the expertise type of evaluation People take part in the decision to evaluate and selecting the aims and methods of the evaluation, take part in feedback, if change recommended the do, 4. The real partnership in development type of evaluation People take part in deciding, when and how to evaluate, in selecting the methods, collecting and analyzing fact and information.

1. Who is Evaluation for?:

8 1. Who is Evaluation for? Sometimes asked by line agencies. Sometimes the organizations assess the Program Gaining of more knowledge Before evaluation it is essential to know what different expectations people have,

2. When should it be done?:

9 2. When should it be done?

2. When should it be done? :

10 2. When should it be done? Decision about when to evaluate will depend; If program got long term objectives (2 years) If the program got short term objective (2–3 month vaccination) end of program What kind of monitoring method already used Are external evaluators to be involved? What about the climate and season? What about the people’s time? What about the program staff time? What about ministries and outside agencies?

3. Where should it be done and Why?:

11 3. Where should it be done and Why?

4. How long will it take?:

12 4. How long will it take? Some factors affecting: The length of time the program has been in operations (Are there many records to look or only few ) The number of people involved? Weather the program is spread over wide area? People’s devoting time (External and internal evaluator) Preparation and testing of evaluation materials (questionnaires, survey forms) Looking for result (speedy look of ministry or agency for result)

5. How much will it cost?:

13 5. How much will it cost? Amount of money available for such activity? Material and resource involved? The objective and scope of the evaluation? Participant will work extra without being paid extra? 6. Who does it belong to?

Planning and Organizing Resources :

14 Planning and Organizing Resources Chapter Objectives: How to choose the required evaluation methods How to use existing records and written materials How to make use of the available resources

Taking the Decision to Evaluate :

15 Taking the Decision to Evaluate Joint by Program & Participants GOOD COMMUNICATION AND MUTUAL AGREEMENT LEAD TO THE BEST DECISIONS

Why do you need to Plan Evaluation?:

16 Why do you need to Plan Evaluation? Importance Planning evaluation is organized method to reach the evaluation objective and when. Selection of priorities and objectives. Indication for kind of methods needs to be used. Decide what you need to do in details. Decide the role of each person in evaluation. Indication for the time frame. Cost. See how one part of the evaluation relates to the other.

Making a Plan to Suit Your Needs :

17 Making a Plan to Suit Your Needs Design = Plan Factors which affect the selection of plan; The objective of your evaluation The scope of the evaluation The participant and resources those are available Understanding of participant Achievements and problems

Evaluating a Moving Program :

18 Evaluating a Moving Program

Evaluating a Moving Program :

19 Evaluating a Moving Program Program consist of People (staff and participants) Structure and Material (building, equipment, machines) Activities (patterns of management, organization)

Evaluation Objectives & Program Objectives :

20 Evaluation Objectives & Program Objectives In evaluation it is helpful if program objectives are very specific, clear. How do you decide which objectives to evaluate? Whether the evaluation is being used mainly to look at program efficiency and progress or impact? The expectation and needs of policy – makers and funders. (Quick result for policy making) Availability of resources.

What is Indicators?:

21 What is Indicators? To show progress towards objectives

Where indicators are not clear :

22 Where indicators are not clear Without having clear and measurable objectives If objectives are not clear try to find it before evaluation What they are doing & why? Listing of activities Indicators format: Percentage Rate Ratio

How to Make the Objectives Clear:

23 How to Make the Objectives Clear

Choosing Appropriate Evaluation Methods :

24 Choosing Appropriate Evaluation Methods Effect of program on people Program structure and organization Effect of program activities People (commonly-used methods) Physical measurements: height, weight, medical test Verbal questions: questionnaires, interviews, meetings Written questions: test of knowledge

Choosing Appropriate Evaluation Methods:

25 Choosing Appropriate Evaluation Methods Analysis of existing information: reports, records Observations: photographs and drawings

Choosing Appropriate Evaluation Methods:

26 Choosing Appropriate Evaluation Methods Program structure and organization Written information: reports, budgets, past evaluation result, minutes of meeting Verbal or written questions: questionnaires Interviews Observation Discussions Record of personal work

Slide 27:

27 Program activities Measurement of various kinds: area covered, level of product Physical tests Verbal or written questions: questionnaires, Interviews Observation and pictures Written information

Using Reliable Evaluation Methods :

28 Using Reliable Evaluation Methods Method which can be trusted and give good result Compatible result producer (same) Qualitative & Quantitative Evaluation Methods Ruler Vs Questionnaire

What Valid & Invalid Mean :

29 What Valid & Invalid Mean Valid = well worked, reasonable, true. Invalid = not reliable or untrue Ten ways to spoil evaluation and make it invalid Assuming that changes are caused only by the existence of program (learning) Ignoring influences outside of a program like (change in government, policy, power groups, and opportunities. Forgetting same evaluation method The bad choice of groups for comparison

Ten ways to spoil evaluation and make it invalid:

30 Ten ways to spoil evaluation and make it invalid Loss of interest by program staff or participants Repeating tests, interview, questionnaires, where participant still remember Ignoring of interested people Claiming that result of a small – scale evaluation also apply to a wide group of people or area Planning evaluation badly from the start Using unreliable evaluation methods

Making an Evaluation Plan :

31 Making an Evaluation Plan Methods of planning and organization they use or wish to try. Who will be involved? Relationship of many derivative plans

Evaluation Plan:

32 Evaluation Plan Introduction Where is the program situated? What are the characteristics of the country, climate? What are the characteristics of the region, patterns of living, agriculture, employment, education, income, conditions of health?

Evaluation Plan:

33 Evaluation Plan History of the program When, why, where and how did the program begin? How has the program progressed since then? What have been the most important events affecting its development? How many people have been involved as staff, participants, students, patients, groups of farmers, mothers, etc.

Evaluation Plan:

34 Evaluation Plan Program Goals: Who chose the original program goals? Are they still same today? Which are the most important program goals at present and why? Do all those involved in the program have the same idea about its goals? What kind of basic information did you have when the program started, and how was it collected and used? Which of the program goals are very specific and when do you plan to reach them? Program Effect:

Planning the day-to- day details :

35 Planning the day-to- day details

Slide 36:

36

Collecting Written Material :

37 Collecting Written Material

Check list for collecting written material:

38 Check list for collecting written material The original program proposal Letters & papers relating to the program proposal Reports of past evaluations or surveys by other people A map of target location List of present program objectives A chart showing staff positions & function in program Report of past program meetings Regular progress reports (annual and monthly) Other written program records The program’s budget and past financial reports The evaluation budget Past evaluation of a similar program

Resources: What have you got :

39 Resources: What have you got Effective use of resources To keep the cost of evaluation low Only buy extra materials that are essential Have an evaluation you can afford

Resources: What else may you need :

40 Resources: What else may you need Paper Carbon paper Colored card or thick paper Pen, pencil, erasers, chalk, ruler, Larger items : Type writer Photocopier Calculator Camera and film Type recorder, batteries