Impact of Agile Quantified - A De-Mystery Thriller

Views:
 
     
 

Presentation Description

This is the 2014 edition of the ongoing research we're doing at Rally Software quantifying the impact that various behaviors, attitudes and practices to performance.

Comments

Presentation Transcript

PowerPoint Presentation: 

Lean-Agile city. This place runs on folklore, intuition, and anecdotes. If you want to know the truth about this town, stick with me. I’ll give you a tour you’ll never forget. But if you don’t want your beliefs challenged with facts, you’d better beat it, kid. I don’t want to upset you.

PowerPoint Presentation: 

My sidekick down there? That’s Larry Maccherone. He’s worked in this town his entire professional life.

PowerPoint Presentation: 

I’m going to give you the tools to find the real-world numbers that can help you make the economic case to get the resources you need and get your people to commit to change. Really . See? It says so right here: -

The Seven Deadly Sins of Agile Measurement: 

The Seven Deadly Sins of Agile Measurement To avoid creating a monster, stay away from ... 1 1 Manipulating Others 2 2 Unbalanced Metrics 3 3 Quantitative Idolatry 4 Overpriced Metrics 5 Lazy Metrics 6 Bad Analysis 7 Linear Forecasting

PowerPoint Presentation: 

Manipulating Others Sin #1 Using metrics as a lever to drive someone else’s behavior

PowerPoint Presentation: 

Self Improvement Heavenly Virtue #1 Using metrics to reflect on your own performance

PowerPoint Presentation: 

Linear Forecasting Sin #7 Forecasting without discussing probability and risk

PowerPoint Presentation: 

Probability Tools Heavenly Virtue #7 Using the proper tools to predict the likelihood of results (Not likely)

Monte Carlo Simulation: 

Monte Carlo Simulation

PowerPoint Presentation: 

Correlation does not necessarily mean causation CAUTION:

PowerPoint Presentation: 

CAUTION: Only good practices in context There are no best practices

Iteration length: 

Iteration length

PowerPoint Presentation: 

Crowd wisdom or shared delusion? Iteration length Teams using 1 week 6.2% 2 weeks 59.1% 3 weeks 23.4% 4 weeks 9.8% 5+ weeks 1.5%

SDPI current dimensions: 

SDPI current dimensions Productivity (Throughput) Predictability (Stability of Throughput) Responsiveness (Time in Process) Quality (Defect Density)

Future SDPI dimensions: 

Future SDPI dimensions Customer/ Stakeholder Satisfaction (Late 2014) Build-the- Right-Thing metric (2015) Employee Engagement/ Satisfaction (Late 2014) Code Quality from Static Analysis (2015)

Raw metrics → Percentiles = Index: 

Raw metrics → Percentiles = Index

Iteration length: 

Iteration length

Overall Performance: 

Overall Performance Relationship of Iteration Length to Overall Performance index score

Productivity: 

Productivity Relationship of Iteration Length to Productivity index score

Quality: 

Quality Relationship of Iteration Length to Quality index score

Responsiveness: 

Responsiveness Relationship of Iteration Length to Responsiveness index score

Responsiveness: 

Responsiveness Relationship of Iteration Length Transition to Responsiveness index score

Overall Performance: 

Overall Performance Relationship of Iteration Length Transition to SDPI Overall Performance

PowerPoint Presentation: 

Facts Discovered: Teams using two-week iterations have the best balanced performance Longer iterations correlate with higher Quality Shorter iterations correlate with higher Productivity and Responsiveness However, some teams are acting like “tough guys” by pretending to operate at one-week iterations when they can’t back it up

Survey-based research: 

Survey-based research

Teams represented in survey: 

Teams represented in survey

Where in the world?: 

Where in the world?

Where in the U.S.?: 

Where in the U.S.?

Where in the Europe?: 

Where in the Europe?

Ratio of testers to developers: 

Ratio of testers to developers

Testers per Developer: 

Testers per Developer The numbers

Productivity: 

Productivity Relationship of Testers per Developer to Productivity index score

Quality: 

Quality Relationship of Testers per Developer to Quality index score

Responsiveness: 

Responsiveness Relationship of Testers per Developer to Responsiveness index score

Predictability: 

Predictability Relationship of Testers per Developer to Predictability index score

PowerPoint Presentation: 

Facts Discovered: More testers lead to better Quality But they also generally lead to worse Productivity and Responsiveness Interestingly, teams that self-identify as having no testers have: the best Productivity almost as good Quality but much wider variation in Quality

Motive: 

Motive

PowerPoint Presentation: 

Evidence Found: Motive has a small but statistically significant impact on performance Extrinsic motivation does not have a negative impact on performance Executive support is critical for success with Agile. Teamwork is not the dominant factor; talent, skills, and experience are Those motivated by quality perform best

Co-location: 

Co-location

PowerPoint Presentation: 

Evidence Found: Teams distributed within the same time zone have up to 25% better productivity Is distraction a problem?

Geography: 

Geography

PowerPoint Presentation: 

Israel-based teams Find more defects overall But find fewer in production Theory: May correlate with high use of static analysis tools Theory: Could be recording bias

PowerPoint Presentation: 

Facts Discovered: Differences are slight but statistically significant Australia has the best overall performance India the worst. However, there could be a reporting bias for defects Israel seems to catch the most defects before production. Heavy use of static analysis? Recording bias worries?

PowerPoint Presentation: 

One year earlier ... rallydev.com/agilemetrics

PowerPoint Presentation: 

Stable teams result in up to: 60% better Productivity 40% better Predictability 60% better Responsiveness Dedicated teams: Teams made up of people who only work on that one team have double the Productivity Smaller teams have better Productivity Larger teams have better Quality Teams with low WiP have up to: 4x better Quality 2x faster time-to-market But 34% worse productivity

What’s next?: 

Demo of Rally Insights Implements SDPI Roadmap Self-assessment and tracking (surveys) Two more dimensions Customer/Stakeholder Satisfaction Employee Engagement/Satisfaction Probe your environment with customized surveys (maturity, practices compliance, etc.) Recommendation Engine What are the top five things we should improve next? What’s next?

More research: 

More research Over 55 variables under study

Seven Deadly Sins of Agile Measurement: 

Seven Deadly Sins of Agile Measurement You give us: a great review of this talk We give you: a poster of the Seven Deadly Sins of Agile Measurement

Start right now!: 

Start right now! Questions: Bring them to the Open Space at 1:45pm Bring them to the demo at 3:40pm (rm 215) Ask a Rally Staff member (bright red T-Shirts) Visit the Rally Booth Bring your completed card to the Rally booth for a free copy of the best-selling book, “Agile Business”

PowerPoint Presentation: 

A fact without a theory is like a ship without a sail, is like a boat without a rudder, is like a kite without a tail. A fact without a figure is a tragic final act. But one thing worse in this universe is a theory without a fact. ~ George Schultz

PowerPoint Presentation: 

Replace Folklore with Facts Swap Anecdotes with Evidence Upgrade Intuition to Insights

PowerPoint Presentation: 

Hidden slides

Team context: 

Team context

Team time together - Productivity: 

Team time together - Productivity

US time zones: 

US time zones

Europe time zones: 

Europe time zones

The Story Continues...: 

Want a demo? 30+ other variables being investigated Survey-based assessment and tracking: Customer/Stakeholder Satisfaction Employee Satisfaction/Engagement We now have a predictive Bayesian regression model for performance Recommendations The Story Continues...

Productivity = Throughput: 

Productivity = Throughput Throughput is simply the count of User Stories completed in a given time period. Productivity (by default) is the percentile scoring of the raw Throughput metric for User Stories normalized by team size.

Predictability = Stability of Throughput: 

Predictability = Stability of Throughput Predictability measures how consistent you are at producing the same amount of work each month as measured by the Coefficient of Variation (CoV) of Throughput. Predictability (by default) is the percentile scoring of the raw CoV of Throughput.

Responsiveness = Time in Process: 

Responsiveness = Time in Process TiP shows how long it takes to get one work item through your system. It's the work days that a User Story spends in development and testing. Similar to lead time or cycle time. Responsiveness (by default) is the percentile scoring of the raw Time In Process (TiP) metric for User Stories.

Quality = Defect Density: 

Quality = Defect Density Defect Density is a representation of the number of defects found in your code. It's the count of defects found in a given time period, normalized by team size. Quality (by default) is the percentile scoring of the raw defect density metrics for both defects found in test as well as those found in production.

Controlling WiP: 

Controlling WiP

Responsiveness: 

Responsiveness Relationship of WiP per Team Member to Time in Process (TiP) LOWER IS BETTER Most obvious finding: Little’s Law

Quality: 

Quality Relationship of WiP per Team Member to Defect Density LOWER IS BETTER Most dramatic finding

Predictability: 

Predictability Relationship of WiP per Team Member to Coefficient of Variation LOWER IS BETTER

Productivity: 

Productivity Relationship of WiP per Team Member to Throughput HIGHER IS BETTER

PowerPoint Presentation: 

Facts Discovered: Teams that most aggressively control WiP: Have ½ the Time in Process (TiP) Have ¼ as many defects But have 34% lower productivity

PowerPoint Presentation: 

Recommendations: If your WiP is high, reduce it If your WiP is already low, consider your economic drivers If Productivity drives your bottom line, don’t push WiP too low If time to market or quality drives your bottom line, push WiP as low as it will go

Iteration length: 

Iteration length Coming back to ...

Overall Performance: 

Overall Performance Relationship of Iteration Length to Overall Performance index score

Productivity: 

Productivity Relationship of Iteration Length to Productivity index score

Quality: 

Quality Relationship of Iteration Length to Quality index score

Predictability: 

Predictability Relationship of Iteration Length to Predictability index score

Responsiveness: 

Responsiveness Relationship of Iteration Length to Responsiveness index score

Responsiveness: 

Responsiveness Relationship of Iteration Length Transition to Responsiveness index score

Overall Performance: 

Overall Performance Relationship of Iteration Length Transition to SDPI Overall Performance

PowerPoint Presentation: 

Facts Discovered: Teams using two-week iterations have the best balanced performance Longer iterations correlate with higher Quality Shorter iterations correlate with higher Productivity and Responsiveness However, some teams are acting like “tough guys” by pretending to operate at one-week iterations when they can’t back it up

Estimating process: 

Estimating process

PowerPoint Presentation: 

Process Type Teams Using No Estimates 3% Full Scrum 79% Lightweight Scrum 10% Hourly-Oriented 8%

Scrum Types vs. Performance: 

Scrum Types vs. Performance

Quality: 

Quality Defects vs. Process Choice

Improve Quality by 250%: 

Teams that follow the Full Scrum process perform better than most alternatives, but Lightweight Scrum is actually better overall. Improve Quality by 250%

PowerPoint Presentation: 

Facts Discovered: Teams doing Full Scrum have 250% better Quality than teams doing no estimating Lightweight Scrum performs better overall, with better Productivity, Predictability, and Responsiveness

PowerPoint Presentation: 

Recommendations: Experienced teams may get best results from Lightweight Scrum If new to Agile or focused strongly on Quality, choose Full Scrum

Team stability & Dedication to one team: 

Team stability & Dedication to one team

Productivity: 

Productivity Relationship of Team Dedication to Throughput

Percent Dedicated Work: 

Percent Dedicated Work

Team Stability: 

Team Stability

PowerPoint Presentation: 

Another Fact Discovered: One out of four team members changes every three months!

PowerPoint Presentation: 

Facts Discovered: Stable teams result in up to: 60% better Productivity 40% better Predictability 60% better Responsiveness

PowerPoint Presentation: 

Recommendations: Dedicate people to a single team Keep teams intact and stable

Team size: 

Team size

Balance your team’s Performance: 

Agile recommends that the ideal team size is 7± 2. How ideal is that when we actually look at the data? Balance your team’s Performance

PowerPoint Presentation: 

Facts Discovered: Small teams (of 1-3) people have: 17% lower Quality But 17% more Productivity than teams of the recommended size.

PowerPoint Presentation: 

Recommendations: Set up team size of 7±2 people for the most balanced performance If you are doing well with larger teams, there’s no evidence that you need to change