2007 Annual Session Michael Quinn Patton

Deepening: 

Deepening Extension’s Knowledge Base

Seeing ThroughanEvaluation Lens: 

Seeing Through an Evaluation Lens

Slide3: 

In the beginning…

Extension: 

Extension as a knowledge-based learning organization

Slide6: 

LEARNING FRAMEWORK Action Beliefs Knowledge

Premises: 

Premises Action flows from some combination of beliefs/values and knowledge. Informed action has a strong knowledge base. By testing and evaluating beliefs, we build knowledge, thereby making our actions more informed and empirically based.

Slide8: 

In short, ACTIONS flow from some combination of… BELIEFS and KNOWLEDGE .

Learning Organization: 

Learning Organization Moving from beliefs to knowledge as the foundation for and basis of action.

Slide10: 

What constitutes high quality, useful knowledge?

Aspirations: 

Aspirations Scientifically-based knowledge Research-validated findings Evidence-based practice Best practices/Promising practices Generalizable Lessons Learned

Slide12: 

The Debate About Randomized Controls in Evaluation: The Gold Standard Question

GOLD STANDARD:METHODOLOGICALAPPROPRIATENESS notMethodologicalorthodoxy or rigidity: 

GOLD STANDARD: METHODOLOGICAL APPROPRIATENESS not Methodological orthodoxy or rigidity

Slide14: 

Observational Data Suffices A study in the British Medical Journal by Gordon Smith and Jill Pell (2003) found that…. http://bmj.bmjjournals.com/cgi/content/full/327/7429/1459?ck=nck

Slide15: 

There have been NO randomized control trials of parachute use.

Slide17: 

Smith and Pell concluded: “Only two options exist. The first is that we accept that, under exceptional circumstances, common sense might be applied when considering the potential risks and benefits of an intervention.” OR

Slide18: 

those who criticize interventions that lack a RCT evidence base will not hesitate to demonstrate their commitment by volunteering for a double blind, randomized, placebo controlled, crossover trial.”

What is possible and appropriate:: 

What is possible and appropriate: Multiple sources of data about each proposition Triangulation of sources High Quality Lessons

Slide20: 

The Metaphor of Triangulation

Multiple Data Sources: 

Multiple Data Sources to build Knowledge

Highly Rated Knowledge: 

Highly Rated Knowledge

“Star” Sources: 

“Star” Sources basic and applied research

Limitations to published researchas the sole or primary source of knowledge: 

Limitations to published research as the sole or primary source of knowledge Errors Inadequate peer review processes Corrections over time Time for replication Generalizability issues Corruption: publishing pressure, funding pressures

“Star” Sources: 

“Star” Sources basic and applied research evaluation findings --patterns across programs

“Star” Sources: 

“Star” Sources basic and applied research evaluation findings--patterns across programs practice wisdom and experience of practitioners: individuals & teams

“Star” Sources: 

“Star” Sources basic and applied research evaluation findings--patterns across programs practice wisdom and experience of practitioners experiences and knowledge of consumers, program participants, clients, intended beneficiaries…

More “Star” Sources: 

More “Star” Sources expert opinion

“Star” Sources: 

“Star” Sources expert opinion cross-disciplinary connections and patterns

“Star” Sources: 

“Star” Sources expert opinion cross-disciplinary connections and patterns THEORY – capturing and organizing knowledge

The Theory-Practice Connection: 

The Theory-Practice Connection There’s nothing so practical as a good theory. Lazarsfeld

Slide32: 

Theory of Change Program Theory Logic Models Theory of Action Intervention Strategy Theory-driven evaluation

Focus of High QualityLessons Learned & Knowledge:: 

Focus of High Quality Lessons Learned & Knowledge: Enhancing Our Theory of Change

High Quality Lessons : 

High Quality Lessons Those lessons that are supported by and triangulated from multiple sources that reinforce and affirm the knowledge encompassed in the lesson, yielding:

Slide35: 

High Quality, Triangulated Knowledge to inform a Theory of Change

Assessing Lessons: 

Assessing Lessons Assess the importance of the lesson learned for a particular action Examine and evaluate the strength of the connection to outcomes attainment. Compare case examples to extract principles. Reflective Practice

Example: 

Example Extension’s Mission and Theory of Change

Slide38: 

LEARNING FRAMEWORK Action Beliefs Knowledge

Slide39: 

Extension’s Mission BELIEF DEBATE: Knowledge Transfer (Education) or

Slide40: 

Knowledge Transfer (Education) or Behavior & Practice Change

Bennett’s Hierarchy: 

Bennett’s Hierarchy 7. End results --impact on the overall problem   6. Practice and behavior change   5. Knowledge, attitude, and skill changes 4. Reactions & satisfaction  3. Participation  2. Activities 1. Inputs & resources

Basic Education Model Belief: 

Basic Education Model Belief Changed Behavior Knowledge change

Basic Assumptions: 

Basic Assumptions Rational, information-based decision-making and Linear, direct cause-effect outcomes model

Reality-testing: 

Reality-testing Poor information processors, beliefs trump evidence Selective perception Humans as emotional animals Neuroeconomics

Evidence: 

Evidence Evaluation findings – knowledge alone generally doesn’t work to produce lasting behavior and practice changes for difficult and major issues

Evidence: 

Evidence Research findings – lasting behavior change is multi-faceted, requiring significant incentives, on-going reinforcement of new behaviors, and integration with other beliefs, values, and knowledge

Evidence: 

Evidence Theory – Systems theory emphasizes understanding behavior change in a systems context, that is, in relation to other factors and dynamics, not as an isolated phenomenon: Farming systems Family systems work Complex adaptive systems approaches

Other Evidence: 

Other Evidence Feedback from clients and program participants Expert opinion Cross-disciplinary findings from -- behavioral economics -- systems psychology -- organizational sociology -- adult learning

Slide49: 

The nature of EXPERTISE

The Knowledge Challenge:: 

The Knowledge Challenge: Matching the knowledge transfer and program intervention strategy to the nature of the situation, the desired changes, system understandings, resources, and timeline to optimize impact within a local knowledge framework.

Some premises:: 

Some premises: Evaluation is part of initial program design, including conceptualizing the theory of change Evaluator’s role is to help users clarify their purpose, hoped-for results, and change model. Evaluators can/should offer conceptual and methodological options. Evaluators can help by questioning assumptions.

Conceptual Options: 

Conceptual Options Simple Complicated Complex

Types of Community IssuesThe Stacey Matrix: 

Types of Community Issues The Stacey Matrix Certainty Agreement Close to Far from Far from Close to

Simple: 

Simple Certainty Agreement Close to Far from Far from Close to Simple Plan, control

Slide55: 

Following a Recipe A Rocket to the Moon Raising a Child Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Simple

Technically Complicated: 

Technically Complicated Certainty Agreement Close to Far from Far from Close to Simple Plan, control Technically Complicated Experiment, coordinate expertise

Slide57: 

Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Simple

Socially Complicated: 

Socially Complicated Certainty Agreement Close to Far from Far from Close to Simple Plan, control Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground

Socially complicated: 

Socially complicated Implementing human rights agreements, like gender equity or outlawing child labor Environmental Initiatives Many different and competing stakeholders Diverse vested interests High stakes

Socially complicated: 

Socially complicated situations pose the challenge of coordinating and integrating many parts

Know When Your Challenges Are In the Zone of Complexity: 

Know When Your Challenges Are In the Zone of Complexity Certainty Agreement Close to Far from Far from Close to Simple Plan, control Zone of Complexity Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground Systems Thinking Relationship Building Collaboration Good Enough Vision Chunking Around Drivers Minimum Specifications Multiple Actions Adaptability & Organic

Slide62: 

Following a Recipe A Rocket to the Moon Raising a Child Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Every child is unique Uncertainty of outcome remains Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Simple

Know When Your Challenges Are In the Zone of Complexity: 

Know When Your Challenges Are In the Zone of Complexity Certainty Agreement Close to Far from Far from Close to Simple Plan, control Zone of Complexity Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground Chaos Massive Avoidance

Slide64: 

Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipe notes the quantity and nature of “parts” needed Recipes produce standard products Certainty of same results every time Simple

Slide65: 

Challenges: Situation Recognition and Knowledge Appropriate to the Situation

Slide66: 

Systems Parts are interdependent such a change in one part changes all parts The whole is greater than the sum of the parts Focus on interconnected relationships Systems are made up of sub-systems and function within larger systems

Slide68: 

Logic Model for Pregnant Teens Program 1. Program reaches out to pregnant teens 2. Pregnant teens enter and attend the program (participation) 3. Teens learn prenatal nutrition and self-care (increased knowledge) 4. Teens develop commitment to take care of themselves and their babies (attitude change) 5. Teens adopt healthy behaviors: no smoking, no drinking, attend prenatal clinic, eat properly (behavior change) 6. Teens have healthy babies (desired outcome)

Slide69: 

Systems web showing possible influence linkages to a pregnant teenager

Slide70: 

Program systems web showing possible institutional influences affecting pregnant teenagers' attitudes and behavior

Slide71: 

Sustainable change: Systems dynamic reinforcing feedback loops

Slide72: 

Using Different System Lenses to understand a “particular” System Biologic System Energy Well-being Cycle status Vitality Sociologic System Relationships Conversations Interdependence Status issues Meaning/sense Environment Volatility Predictability Spatial Proximities Diversity Economic System Resources Fears & rewards Aspiration Political System Power Connections Possibility Locus of control Anthropologic System Values Culture/Milieu Information System Education Speed & access Fidelity/utility Knowledge Storage Psychological System Incentives Heuristics Reinforcements Behavior Change

Slide73: 

Extension as a knowledge-based organization

Sharing Knowledge: 

Sharing Knowledge to change behavior and practice, and solve problems, based on a Theory of Change

Slide75: 

HYMN to KNOWLEDGE USE Sung to the tune of Auld Lang Syne

Slide76: 

May all Ex-ten-sion programs use Knowledge that consumers trust, That tell them how to separate What works from what’s a bust. We gather data near and far To see what we can learn The findings help us to decide What to share and what to burn.

Slide77: 

There comes a time for each of us When doubts may give us pause We wonder what results will last Will our knowledge turn out flawed? But be assured there's naught to fear Shared learning is what we seek Tri-angulate your sources now With authority you can speak.

Slide78: 

Our knowledge base we can affirm, Extension shares the truth. Our lessons learned we will apply On farms, in homes, with youth. Extension staff go forth as one, Researchers here stand tall, The world will be a better place Sharing knowledge is our call.

References: 

References Utilization-Focused Evaluation, 3rd. Edition, by Michael Q. Patton, Sage Publications, 1997. (New 4th edition forthcoming at the end of 2007.) Getting to Maybe: How the World Is Changed by Frances Westley, Brenda Zimmerman, and Michael Q. Patton, Random House Canada, 2006. http://www.randomhouse.ca/catalog/display.pperl?isbn=9780679314431

Slide80: 

Systems Concepts in Evaluation – An Expert Anthology. 2006. Bob Williams and Iraj Imam AEA Monograph, EdgePress/AEA Point Reyes CA.