Presentation Transcript
Randomized Experimentation for the Program Manager: A Quick How-To Guide : Randomized Experimentation for the Program Manager: A Quick How-To Guide Jonathan Zinman
Assistant Professor of Economics, Dartmouth College
Research Associate, Innovations for Poverty Action
May 1, 2007
Presentation for IFC M&E Conference
What’s Your Objective?: What’s Your Objective? If it’s “beat or meet the market”….
Can’t afford to focus on evaluation per se
What’s Your Strategy?: What’s Your Strategy? Focus must be….
Using evaluation to feed innovation
(Abhijit’s “dynamic learning”)
Experimentation & the Learning Organization: A Virtuous Cycle: Experimentation & the Learning Organization: A Virtuous Cycle
Randomized Evaluation: A Quick How-to Guide: Randomized Evaluation: A Quick How-to Guide What should I evaluate?
How do I design and implement the evaluation?
How do I interpret and apply the results?
At each step will highlight how randomized-control trials (RCTs) can be part of an innovation strategy producing comparative advantage for:
Client financial institutions (FIs)
Wholesale investors like IFC
What Should I Evaluate? : What Should I Evaluate? Which interventions (“treatments”)?
Which outcomes?
Which Interventions?: Which Interventions? Use RCTs to address critical questions:
Business/programmatic/policy strategy
Stakes should be high, given fixed costs of experimentation and related data collection
Existing programs you’re funding (of course!)
Also….
Which Interventions?: Which Interventions? But also: “Microfinancial Engineering”
Analogy to “high” finance
“Design-build” partnerships between academics & financial institutions
Input from funders, program managers, policymakers
Don’t just “take opportunities” to do systematic experimentation
Create opportunities. What’s a “program”?
Innovation challenges
Brokering
Funding
Which Interventions?: Which Interventions? RCT interventions can be built into any critical business function
Product development
Pricing
Marketing and targeting
Monitoring and enforcement
Risk assessment
Which Interventions?: Which Interventions? Product Development examples
Savings products with new features:
Commitment
Goals
Method and nature of communication with client
Smoking cessation performance bond
Smokers “bet themselves” they can quit (as measured by urine test in 6 months)
Financial institution fills missing market: enforcement
Micro-insurance
Evaluate by randomizing
Whether product offered
Product features
Which Interventions?: Which Interventions? Pricing examples
Loans: commercial, consumer
Savings: various products
Micro-insurance
Evaluate by randomizing
Initial offer
Dynamic pricing (future offers)
Which Interventions?: Which Interventions? Marketing and targeting examples
Marketing: what speaks to the client and why?
Direct mail: randomize content
Text messaging: randomize content
Targeting: finding your market or intended beneficiaries
Health insurance for the poor in the Philippines
Working with government, its contractors to experiment with:
Measurement techniques
Incentives for using them
Compensation
Punishment (monitoring, auditing)
Which Interventions?: Which Interventions? Loan monitoring/enforcement examples:
Require peer monitoring or liability?
Incentivize peer monitoring (referrals)?
How monitor and communicating with (delinquent) borrowers?
What are the most (cost-) effective threats and penalties?
Evaluate by randomizing:
Mechanisms, incentives, protocols
Which Interventions?: Which Interventions? Risk assessment
Possible to lend profitably to some rejected applicants?
Innovations to test:
Improve risk assessment model (credit scoring; other projects)
Provide incentives to overcome loan officer conservatism (details to follow)
Evaluate by randomizing:
The approve/reject decision (within reasonable bounds)
How-to Example From One Project: How-to Example From One Project Experiment with risk assessment
Objective: measure impacts of expanding access to consumer loans
Popular product in South Africa
Worked with a leading for-profit “microlender”
“Traditional” micro(enterprise)credit largely absent in South Africa
4-month installment loans; 200% APR
Example: Expanding Access: Example: Expanding Access What we did. Overview of the experiment:
Intervention: randomly approve loans for some close-to-creditworthy (“marginal”) applicants who would normally be rejected
This is the “treatment group”
Other marginal applicants remain rejected
This is the “control group”
Measure impacts
Difference between treatment and control
For the many outcomes of interest
Lender: use its data to calculate profits
Applicants: survey data on economic and well-being outcomes
See “Expanding Access: Using Randomized Supply Decisions to Estimate the Impacts” (with Dean Karlan): http://www.dartmouth.edu/~jzinman/Papers/Derationing.Karlan-Zinman.pdf
How Did This Evaluation Come About?: How Did This Evaluation Come About? Remember Step 1: What do we evaluate?
Which intervention(s)?
Which outcomes?
This project very much design-build
Lender motivation
Prior experiments with Lender
Identified specific market failures (asymmetric information problems)
Identified binding liquidity constraints for borrowers
These reinforced Lender’s priors that loan officers being too conservative
Profitable deals being left on the table
Open to RCT as systematic way to control and evaluate risk of liberalizing criteria
Motivation for this Evaluation: What do we Evaluate?: Motivation for this Evaluation: What do we Evaluate? Researcher/policy/funder angles:
Consumer credit controversial
Policy tends to restrict rather than encourage access
But why? Economic arguments for restricting tenuous
But consumer (micro)credit markets growing
Our methodology applicable to microenterprise credit as well
Step 2. How do we do it? Design and Implementation: Step 2. How do we do it? Design and Implementation 3 key issues in this case:
Scope of study
Implementing the intervention: how to randomize loan approvals decisions
Tracking and measuring outcomes
Design & Implementation: Scope: Design & Implementation: Scope A. Scope: How big? Where?
Required deal flow for a conclusive evaluation?
How big a sample do we need to answer the questions of interest (statistical power)
Researchers identify
Best way to obtain the required deal flow? Researchers and Lender worked together to identify:
A practical definition of “marginal” applicant
Timeframe for the intervention (2 months)
Participating branches. Chose 8 branches that would
Produce required deal flow in the 2 month timeframe
Be relatively easy to monitor and train
Be representative enough to draw conclusions re: whether or not to scale up the intervention to other branches
Design & Implementation: The Intervention: Design & Implementation: The Intervention B. How actually randomize loan approvals?
Insert research protocols into loan assessment process. In this case 2 additional steps:
Loan officers rejected applicants into “marginal” and “egregious”
New software randomizes “marginal” into “keep rejected” or “approve” (second look)
New implementations streamline this with introduction of pure credit scoring model
Train branch personnel
Monitor (& incentivize) branch personnel to comply with protocols
Design & Implementation: Measurement: Design & Implementation: Measurement C. Tracking & measuring outcomes
Lender data on sales, costs, & hence profitability
Follow-up survey data on applicant outcomes:
Economic (employment status, income, consumption)
Subjective well-being (decision power, optimism, mental health)
Researchers designed household survey instrument
Contract survey administration to survey firm
Close monitoring from pilot to final survey
Results: Lender Outcomes: Results: Lender Outcomes Lender made money: marginal loans were profitable
Less profitable than loans above the bar
But profitable nonetheless
Even on initial loan
Profits from acquiring new clients even bigger
Did Lender scale up? That was the plan, but….
Then Lender was merged into a larger bank
New senior mgmt hostile to “consultants”
Old senior mgmt (our partners) banked knowledge and took to new firms
Results: Applicant Outcomes: Results: Applicant Outcomes Large, positive, statistically significant impacts on:
Economic self-sufficiency (employment, income, above poverty line)
Consumption (avoiding hunger, food quality)
Outlook and control (decision power, optimism)
No significant impacts on:
Investment (education, housing, self-employment)
Physical health
Negative impact (90% significant) on mental health (depression, stress)
Overall impact significant and positive
If weight all outcomes equally
Step 3. How Apply The Results?: Step 3. How Apply The Results? The intervention itself
Do (social) costs exceed benefits?
In this case interpreting results simple: win-win
Often there are tradeoffs: weighing costs and benefits requires some insight into the “whys” of impacts
Here evidence of market failures from earlier experiments
Prior and project evidence of binding liquidity constraints
Opportunity cost of intervention(s)?
In consumer credit key is ruling out negative effects: default policy/programmatic approach is to restrict access
Unlike microenterprise credit, where default approach is to expand/subsidize access, and hence opportunity cost of subsidy matters
How Apply The Results?: How Apply The Results? Applying the results (external validity)
Scalability
Replicability
Three complementary approaches:
Design so that get answers re: why interventions do or don’t work
Choose sites/markets/partners carefully
Do lots of RCT experimentation
Take-Aways: Take-Aways RCTs deliver:
Gold-standard measures of impacts
Insights into the “why” questions that:
Affect scalability
Feed back into innovation
RCTs are doable:
Design-build partnerships with researchers for:
Microfinancial Engineering
Innovation that is scalable and replicable
Experimentation & the Learning Organization: A Virtuous Cycle: Experimentation & the Learning Organization: A Virtuous Cycle