The Seven Deadly Sins of Agile Measurement


Presentation Description

Using Measurement as levers rather than for feedback, is sin #1. What's the difference? Levers are employed to change someone else's behavior. Feedback is employed to improve your own performance. The distinction is subtle but critical. The above is just one example. This talk will walk you through the seven deadly "sins" to look out for when implementing an agile measurement regime, but it's not all fire and brimstone. We will leave you with a list of "heavenly virtues" or good practices to follow when implementing your measurement regime and we will present examples of companies that we have worked with and whose metrics regimes exhibit these virtues. This information should give you the means to bend your own execs towards risk evaluation rather than absolutes; toward measurement as an insight amplification and feedback mechanism rather than a club to beat people up; as something that your teams will seek out rather than something that they will dread.


Presentation Transcript

The Seven Deadly Sins of Agile Measurement:

The Seven Deadly Sins of Agile Measurement Larry Maccherone Director of Analytics – Rally Software Ph.D. Candidate – Carnegie Mellon Twitter: @ LMaccherone


THERE ARE NO BEST PRACTICES… ONLY GOOD PRACTICES… …IN CONTEXT Twitter: @ LMaccherone “Would you like to start with the Bayesian soup or maybe the Monte Carlo oeuvres.” Imagine your order being taken by a snooty waiter at a five star restaurant and then he turns around and yells the order back to the kitchen like he was working at a diner. Both are good practices in their context but wrong outside of it. All too often we misapply. CMMi is not list of practices, but is associated with a culture that developed certain practices. SOME OF THOSE PRACTICES SHOULD NOT BE APPLIED IN AN AGILE ENVIRONMENT. THIS TALK WILL HELP YOU FIGURE OUT HOW TO AVOID MISAPPLICATION OF PRACTICES IN AN AGILE ENVIRONMENT.

PowerPoint Presentation:

LARRY MACCHERONE Twitter: @ LMaccherone

What is Agile?:

What is Agile? VALUES + Controller System Sensor Reference Measured error Measured output System input System output + - Twitter: @ LMaccherone In it’s simplest form agile is: 1) Do something; 2) Inspect; and 3) Adapt. It’s essentially about feedback loops. But isn’t all process about feedback loops? Continuous improvement (ISO-9000, TQM), Spiral/RUP, DMAIC/6Sigma, etc. AGILE IS A CHANGE IN EMPHASIS IN THOSE FEEDBACK LOOPS. Generally away from the process and toward the product. Mark Kennaley does a great job of explaining this in his book, “SDLC 3.0: Beyond a Tacit Understanding of Agile ”

PowerPoint Presentation:

TRADITIONAL AGILE PROCESS FEEDBACK FORMAL and HEAVY LIGHT and OFTEN PRODUCT AND DESIGN FEEDBACK Varies, sometimes silent EARLY, OFTEN, and CLOSE to the CUSTOMER PLAN FEEDBACK HEAVY, UP-FRONT and in LONGER ITERATIONS LIGHT, THROUGHOUT and in SHORT ITERATTIONS DOMINANT VALUE CONTROL for the ORGANIZATION COLLABORATION by the TEAM Twitter: @ LMaccherone If you think of ALM work as having 3 key feedback loops: Process, Product, and Plan, traditional methods emphasize the process and some emphasize the plan. Agile clearly emphasizes feedback on the design of the product. The cultural difference are also key. Measurement for the folks seeking to improve is key. That's usually the team but management seeking measurement to improve their ability to fill there "servant leadership" role is also great. So management dashboards are still important. You just want them for a different reason in this culture .

What about?:

What about? Twitter: @LMaccherone Scrum?, XP?, Crystal?, Kanban ? TDD?, Continuous integration?, Pair programming? Daily standups? Burndown charts? Sprints? The agile manifesto’s 4 values and 12 principles? Agile Manifesto: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan DoD Context: You'll need tools; you'll have contract negotiation; and you'll be required to provide documentation. Focus on product feedback. Seek out ways to reduce coordination costs (XP practices). Emphasize “responding to change” as well as team collaboration in DoD contexts.

The seven deadly sins of agile measurement:

The seven deadly sins of agile measurement Twitter: @LMaccherone

Why measure?:

Why measure? Tweet alert: Feedback measures are used to improve y our own performance . Lever measures are employed to change someone else’s behavior . Twitter: @LMaccherone

Sin #1 Using metrics as levers:

Sin #1 Using metrics as levers Twitter: @LMaccherone

When to NOT take a shot:

When to NOT take a shot Good players? Monta Ellis 9 th highest scorer (8 th last season) Carmelo Anthony ( Melo ) 8 th highest scorer (3 rd last season) Twitter: @LMaccherone Driving behavior directly with a metric like high score can actually be counter to your own best interests. When they are in the game, their teams actually have a lower chance of winning. Wining games is the outcome you really want but Monta and Melo are motivated to take shots that they shouldn't... stealing shots from their own teammates who have a better chance of scoring. If they had measures that helped them make better decisions about when to take the shot AND WHEN NOT TO TAKE THE SHOT, they might help their team .

The ODIM Framework:

The ODIM Framework Twitter: @LMaccherone Start with the outcomes. You can think of this as the agile version Vic Basili’s Goal-Question-Metric. The difference between ODIM and GQM is related to sin #1. This is for self improving feedback. The emphasis is insight for the folks doing the work .

Sin #2 Using a convenient metric rather than one that provides critical insight:

Sin #2 Using a convenient metric rather than one that provides critical insight Twitter: @LMaccherone

What is normal?:

What is normal? Cumulative -> 0.1% 2.3% 15.9% 50.0% 84.1% 97.7% 99.9% Roughly -> 85% 98% Twitter: @LMaccherone

Are you normal?:

Are you normal? Twitter: @LMaccherone

The A-Team Data:

The A-Team Data 3X to 10X the risk Twitter: @LMaccherone

Risk calculation off:

Risk calculation off 3x-10x when assuming Normal distribution 2.5x-5x when assuming Poisson distribution 7x-20x if you use Shewhart’s method Heavy tail phenomena are not incomprehensible… but they cannot be understood with traditional statistical tools. Using the wrong tools is incomprehensible. ~ Roger Cooke and Daan Nieboer Twitter: @LMaccherone

Sin #3 Bad analysis:

Sin #3 Bad analysis Twitter: @LMaccherone

Bad application of control chart:

Bad application of control chart Control is an illusion, you infantile egomaniac. Nobody knows what's gonna happen next: not on a freeway, not in an airplane, not inside our own bodies and certainly not on a racetrack with 40 other infantile egomaniacs. ~Days of Thunder Twitter: @LMaccherone When you try to "control" cycle time. .. When you try to reduce variability, you (ironically) invariably end up hiding information from yourself because nobody wants to be responsible for those red colored dots. They will artificially split work so nothing ever goes outside of the line.

Time in Process (TIP) Chart A good alternative to control chart:

Time in Process (TIP) Chart A good alternative to control chart Twitter: @LMaccherone Rather, show the scatter and distribution and enable discussion. Maybe there is information in the outliers that the team can discuss to remove obstacles and speed things up in the future. Maybe that's where the research work is being done and it's actually desirable to have a few Black Swans .

Sin #4 Motivating folks to hide information:

Sin #4 Motivating folks to hide information Twitter: @LMaccherone


Collection Perceived cost is higher than actual cost but developer happiness is key to success. Plenty to learn in passive data from ALM and other tools Use a 1-question Net Promoter Score (NPS) surveys for turning qualitative insight into quantitative insight How you use the tools will drive your use of metrics from them Twitter: @LMaccherone

Sin #5 Too costly measures:

Sin #5 Too costly measures Twitter: @LMaccherone

PowerPoint Presentation:

Twitter: @LMaccherone

Sin #6 Too many measures (information overload):

Sin #6 Too many measures (information overload) Twitter: @LMaccherone

PowerPoint Presentation:

Twitter: @LMaccherone You need a minimum of 4 measures (one from each quadrant) at the very start of your metrics regime

Sin #7 Too few measures (unbalanced):

Sin #7 Too few measures (unbalanced) Twitter: @LMaccherone

Heavenly virtues:

Heavenly virtues Start with outcomes and use ODIM to make metrics choices. Make sure your metrics are balanced so you don’t over-emphasize one at the cost of others. Be careful in your analysis. The TIP chart is a good alternative to control chart. Consider collection costs and perceived burden. Get maximal value out of passively gathered data. Data visualization is like photography. Impact is a function of perspective, illumination, and focus. ~Larry Maccherone Twitter: @LMaccherone

PowerPoint Presentation:

LARRY MACCHERONE Twitter: @ LMaccherone