How to succeed with Theory of Change in the CGIAR Research Programs

Views:
 
     
 

Presentation Description

Seminar presented at WorldFish HQ, Malaysia, 19-January 2016. The new CGIAR Research Program (CRP) proposals will be evaluated in part on their respective theories of change. Drs Boru Douthwaite and Rodrigo Paz-Ybarnegaray explain what is being asked of CRPs by the ISPC and the Consortium Office and identify issues that FISH will likely find challenging. They offer advice and point to learning from AAS that WorldFish and FISH may find useful.

Comments

Presentation Transcript

How to succeed with Theory of Change in the new CRPs:

How to succeed with Theory of Change in the new CRPs Boru Douthwaite and Rodrigo Paz 19 January 2016

Impact pathways, theory of change and the new CRPs:

Impact pathways, theory of change and the new CRPs IPs/ToC a central part of full proposals; key part of results framework Testing ToC part of core research agenda The assessment criteria for IPs/ToC at both CRP and Flagship level: Plausibility and feasibility of IPs/ToC Alignment with the SRF sub-IDOs and IDOs” ( From “Final Guidance for Full Proposals”, p. 27)

Slide3:

CRP IPs and ToC Eg . x% of 190 million ha of degraded land restored

Structure of presentation:

Structure of presentation What is ToC ? How does ISPC see ToC? How does the Consortium Office see ToC? Key issues for FISH and what to do about them? What has AAS has learned about ToC that might help?

What is theory of change?:

What is theory of change? Instructions: Discuss in twos or threes Write one or two cards to answer each question Blue = what is theory of change? Pink = what is it useful for ?

Group definition of ToC (summary only - no audio):

Group definition of ToC (summary only - no audio) What is ToC? Uses of ToC Tool for outlining explicitly how we think our research will lead to impact/change Mental model / logical framework linking activities to outputs to outcomes and impact that makes causal assumptions explicit Securing funding (donors often require Toc) Consensus building; Planning Make thinking / assumptions explicit; exploring if intended outcomes are realistic Communicating major assumptions underlying program impact; communicating role of research Structuring learning during program implementation Makes impact more likely

A theory of change describes how a project has or will achieve outcomes and impact. It describes the causal assumptions:

Vision Plan ACTIVITIES OUTPUTS OUTCOMES IMPACT INPUTS Time A theory of change describes how a project has or will achieve outcomes and impact. It describes the causal assumptions

Slide8:

OUTPUT OUTCOME INPUTS ACTIVITY INPUTS ACTIVITY INPUTS ACTIVITY INPUTS OUTPUT OUTPUT ACTIVITY OUTPUT OUTPUT OUTCOME OUTCOME OUTCOME OUTCOME OUTCOME OUTCOME Vision Plan

Characteristics of ToC (that makes it hard to tie down):

Characteristics of ToC (that makes it hard to tie down) Relates to an intent (project, program, …) Simple or complicated Single or nested Forward-looking or backward-looking An output (diagram, narrative ) And a process / a way of thinking

How does the ISPC see Theory of Change?:

How does the ISPC see Theory of Change? The ISPC will review proposals’ ToC Based on thinking in ISPC White Paper (pers . comm. Maggie Gill, 2016) http://ispc.cgiar.org/sites/default/files/ISPC_WhitePaper_TOCsIPs.pdf

ISPC Definitions:

ISPC Definitions Logical Framework: Links inputs and activities to outputs, outcomes and impact Doesn’t provide insights into causality Usually presented in a table Impact pathways: A more detailed logical framework, in particular showing links between outcomes and impact Usually presented graphically Theory of change Identifies underlying causal assumptions; explains the arrows; Makes change mechanisms explicit

BMZ logical framework matrix:

BMZ logical framework matrix Intervention logic narrative summary Objectively verifiable indicators Means of verifications Major assumptions IDO contributions Overall objective Project purpose Outputs

A FISH Impact pathway:

A FISH Impact pathway

Slide15:

Theory of Change for the RinD approach Causal narrative + Grounded enough to be able to formulate testable hypotheses!

IP and ToC made simple:

IP and ToC made simple IP and ToC is called “Program Theory ” or simply “Theory of Change” in the field of evaluation Impact pathways are the boxes and arrows – “they don’t provide insights into causality” ToC is the testable causal logic behind the boxes and arrows Thorough ToC often requires unpacking black boxes Requires unpacking black boxes IPs are better for skating on causal thin ice

Key points from ISPC White Paper:

Key points from ISPC White Paper ToC a way of handling complexity Each CRP should have an overarching IPs and ToC that lays out key c ausal mechanisms/hypothese s that the CRP will trigger Major CRP components should have own IPs and ToC Testing causal hypotheses is part of the research agenda, to build the evidence base for understanding how research triggers outcomes ToC to be regularly revisited during life cycle of program ToC should express an ‘optimum level of complexity’ - not to simple to overlook feedback loops; not too complex to cause paralysis ToC to be evaluated on degree of thoroughness and realism

Key guidance from CO:

Key guidance from CO Show how CRP contributes to SRF; basis for RBM Develop overall ToC and IPs for CRP Nested ToC for each FSP and gender ToC linked to RBM through CRP Performance Indicator Matrix that provides CRP targets ToC is the justification for the targets set MEL to strengthen and validate ToC “as part of the core research agenda” (p. 30) carried out by researchers; support learning and adaptive management For 2% of CRP budget

Key issues:

Key issues

Slide20:

Optimum level of complexity of ToC Scale / timeframe to impact For communication; for the proposal To be causally thorough Credibility ga p Optimum level of complexity for IP/ToC for germplasm improvement program

Slide21:

Optimum level of complexity of ToC Scale / timeframe to impact Credibility gap To be causally thorough For communication; for the proposal Optimum level of complexity for ToC for a NRM program Credibility is in the eye of the beholder

Slide22:

“The theory of change and impact pathway for RTB is clear, focused, logical and plausible” ISPC 2015

Advice from John Mayne (who did some of the ToC review for ISPC):

Advice from John Mayne (who did some of the ToC review for ISPC) FISH ToC material was not bad. Better than many others Get level of detail right which is more of an art than a science.   Understand CRP level ToC as an impact pathway (IP) showing the main pathways to impact through the FPs. The FP ToCs are more detailed, should show and explain causal links Tell a story Mind the gap - often a big jump to the sub-IDOs and IDOs (the credibility gap).

Performance Indicator Matrix:

Performance Indicator Matrix Core of the RBM framework S ummarizes and budgets the outcomes the CRP proposes to deliver: Part CRP’s contract and provide the basis for: assessing a CRPs value for money, monitoring progress during implementation; evaluation , as well as impact assessment. Page 26

Issue 1: Bridging credibility gap:

Issue 1: Bridging credibility gap Developing ToCs that are realistic and thorough while at the same time justify the impact targets (that are aspirational and non-auditable) Suggestions: Keep ToC simple (to communicate / for space ) See CRP ToC as IPs; Explain causality in FP ToC Show you understand a credibility gap exists Describe how ToC will be built and revisited as part of MEL system (see: AAS ToC publication ) Keep impact targets realistic Tell a story

Issue 2: Designing and implementing MEL :

Issue 2: Designing and implementing MEL Designing monitoring, evaluation and learning (MEL) system that: tests causal hypotheses, builds evidence base, is used to and adjust program implementation on basis of learning with 2% budget Suggestion Learn from AAS

What can FISH learn from AAS?:

What can FISH learn from AAS? Food for thought – Tuesday 19/Jan/2016

Key notes:

K ey notes M&E system of AAS – a quick overview Developing ToC - the critical importance of engaging stakeholders (Who needs to be part of a ToC development process?) Beyond developing: nesting and testing ToC – this is the ‘new’ thing and in practice is not easy Emerging outcomes as indicators - Outcome Evidencing

M&E system of AAS: objectives:

M&E system of AAS: objectives Five objectives guided the design of the AAS M&E system: to fulfill accountability expectations, monitor outcomes, contribute to learning and adaptive management, contribute to research on how change happens, and be credible, useful, and cost-effective.

Slide33:

M&E system of AAS: components 1 2 3 4

Developing, nesting and testing ToC in AAS:

D eveloping, nesting and testing ToC in AAS

Slide35:

Develop ToC: Multiple levels (e.g. Barotse Hub) Hub-level ToC Community-level ToC Initiative-level ToC Collective, multi-stakeholder endeavor

Participatory development of a ToC: Nutrition and food safety initiative – Barotse Hub:

Participatory development of a ToC: Nutrition and food safety initiative – Barotse Hub Mechanisms of change (assumptions)

What de we test? Mechanisms of change (hypothesis to be tested):

What de we test? Mechanisms of change (hypothesis to be tested) Intervention (research activities and outputs) Outcome Mechanisms of change We make explicit how and why change is going to come about, and what is needed for that change to come about

Testing: Stages of the Outcome Evidencing process:

Testing: Stages of the Outcome Evidencing process Stage 3. Carry out the evaluation – substantiate the case of change and contribution   Stage 2. Collectively make sense of outcomes and outcome trajectories and change mechanisms (ToC looking back), and design the evaluation in a workshop – build a case of change and contribution Stage 1. Prospect and describe outcomes – annual call to stakeholders

Test: outcome evidencing supporting learning as part of program M&E:

Test: outcome evidencing supporting learning as part of program M&E 6. Evaluation – Substantiate the case Rapid single-loop learning 4. Workshop – build a case ‘Reflect’ 3. Prospect for outcomes ‘Observe’ 1. Develop a ToC ‘Plan’ 5. Rework the ToC ‘Plan again’ M&E for Learning 2. Implement ‘Act’ Slower double-loop learning

Key learning:

Key learning In AAS we spent three years developing an M&E for Learning system that very well matches what is being asked by the ISPC and in the CRP-2 guideline M&E systems that support learning need to be flexible and evolve over time (staging and timing is critical). On a real-time basis an M&E system should help us identify and make sense of what is happening so to change and adapt rapidly A ToC is most helpful if it is revised regularly and critically And this is the research the ISPC calls for

Resources:

Resources https://sites.google.com/a/cgxchange.org/aasresources/home

Links:

Links Journal articles Apgar, M., Douthwaite, B., Allen, W., Paz- Ybarnegaray , R., (submitted). Getting beneath the surface in program planning, monitoring and evaluation: Learning from use of participatory action research and theory of change in the CGIAR Research Program on Aquatic Agricultural Systems . Action Research Journal Special Issue: Development, Aid and Social Transformation. Douthwaite , B., Mayne , J., McDougall, C., (submitted). Evaluating Complex Interventions: A Theory-Driven Realist-Informed Approach . Evaluation Paz- Ybarnegaray , R., Douthwaite, B., (submitted). Outcome evidencing: a method for enabling and evaluating program interventions in complex systems   Douthwaite, B., Alvarez, B.S., Cook, S., Davies, R., George, P., Howell, J., Mackay, R. and Rubiano , J. (2008). Participatory Impact Pathways Analysis: A Practical Application of Program Theory in Research-for-Development. Canadian Journal of Program Evaluation. 22(2) pp. 127–159 Douthwaite, B., T. Kuby , E. van de Fliert and S. Schulz. (2003). Impact Pathway Evaluation: An approach for achieving and attributing impact in complex systems. Agricultural Systems 78 pp. 243-265

What are the issues for FISH? How can AAS help?:

What are the issues for FISH? How can AAS help?

authorStream Live Help