logging in or signing up The 7 deadly sins of agile measurement LMaccherone Download Post to : URL : Related Presentations : Share Add to Flag Embed Email Send to Blogs and Networks Add to Channel Uploaded from authorPOINT lite Insert YouTube videos in PowerPont slides with aS Desktop Copy embed code: Embed: Flash iPad Copy Does not support media & animations WordPress Embed Customize Embed URL: Copy Thumbnail: Copy The presentation is successfully added In Your Favorites. Views: 722 Category: Science & Tech.. License: All Rights Reserved Like it (0) Dislike it (0) Added: September 17, 2012 This Presentation is Public Favorites: 0 Presentation Description This is a short talk I gave at the Software Engineering Institute. Some of it is targeted at contrasting their context to the typical agile context, but most of it is generally applicable. Comments Posting comment... Premium member Presentation Transcript The Seven Deadly Sins of Agile Measurement: The Seven Deadly Sins of Agile Measurement Larry Maccherone Director of Analytics – Rally Software Ph.D. Candidate – Carnegie Mellon Twitter: @ LMaccheroneTHERE ARE NO BEST PRACTICES… ONLY GOOD PRACTICES… …IN CONTEXT: THERE ARE NO BEST PRACTICES… ONLY GOOD PRACTICES… …IN CONTEXT Twitter: @ LMaccherone “Would you like to start with the Bayesian soup or maybe the Monte Carlo oeuvres.” Imagine your order being taken by a snooty waiter at a five star restaurant and then he turns around and yells the order back to the kitchen like he was working at a diner. Both are good practices in their context but wrong outside of it. All too often we misapply. CMMi is not list of practices, but is associated with a culture that developed certain practices. SOME OF THOSE PRACTICES SHOULD NOT BE APPLIED IN AN AGILE ENVIRONMENT. THIS TALK WILL HELP YOU FIGURE OUT HOW TO AVOID MISAPPLICATION OF PRACTICES IN AN AGILE ENVIRONMENT.PowerPoint Presentation: LARRY MACCHERONE Twitter: @ LMaccheroneWhat is Agile?: What is Agile? VALUES + Controller System Sensor Reference Measured error Measured output System input System output + - Twitter: @ LMaccherone In it’s simplest form agile is: 1) Do something; 2) Inspect; and 3) Adapt. It’s essentially about feedback loops. But isn’t all process about feedback loops? Continuous improvement (ISO-9000, TQM), Spiral/RUP, DMAIC/6Sigma, etc. AGILE IS A CHANGE IN EMPHASIS IN THOSE FEEDBACK LOOPS. Generally away from the process and toward the product. Mark Kennaley does a great job of explaining this in his book, “SDLC 3.0: Beyond a Tacit Understanding of Agile ”PowerPoint Presentation: TRADITIONAL AGILE PROCESS FEEDBACK FORMAL and HEAVY LIGHT and OFTEN PRODUCT AND DESIGN FEEDBACK Varies, sometimes silent EARLY, OFTEN, and CLOSE to the CUSTOMER PLAN FEEDBACK HEAVY, UP-FRONT and in LONGER ITERATIONS LIGHT, THROUGHOUT and in SHORT ITERATTIONS DOMINANT VALUE CONTROL for the ORGANIZATION COLLABORATION by the TEAM Twitter: @ LMaccherone If you think of ALM work as having 3 key feedback loops: Process, Product, and Plan, traditional methods emphasize the process and some emphasize the plan. Agile clearly emphasizes feedback on the design of the product. The cultural difference are also key. Measurement for the folks seeking to improve is key. That's usually the team but management seeking measurement to improve their ability to fill there "servant leadership" role is also great. So management dashboards are still important. You just want them for a different reason in this culture .What about?: What about? Twitter: @LMaccherone Scrum?, XP?, Crystal?, Kanban ? TDD?, Continuous integration?, Pair programming? Daily standups? Burndown charts? Sprints? The agile manifesto’s 4 values and 12 principles? Agile Manifesto: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan DoD Context: You'll need tools; you'll have contract negotiation; and you'll be required to provide documentation. Focus on product feedback. Seek out ways to reduce coordination costs (XP practices). Emphasize “responding to change” as well as team collaboration in DoD contexts.The seven deadly sins of agile measurement: The seven deadly sins of agile measurement Twitter: @LMaccheroneWhy measure?: Why measure? Tweet alert: Feedback measures are used to improve y our own performance . Lever measures are employed to change someone else’s behavior . Twitter: @LMaccheroneSin #1 Using metrics as levers: Sin #1 Using metrics as levers Twitter: @LMaccheroneWhen to NOT take a shot: When to NOT take a shot Good players? Monta Ellis 9 th highest scorer (8 th last season) Carmelo Anthony ( Melo ) 8 th highest scorer (3 rd last season) Twitter: @LMaccherone Driving behavior directly with a metric like high score can actually be counter to your own best interests. When they are in the game, their teams actually have a lower chance of winning. Wining games is the outcome you really want but Monta and Melo are motivated to take shots that they shouldn't... stealing shots from their own teammates who have a better chance of scoring. If they had measures that helped them make better decisions about when to take the shot AND WHEN NOT TO TAKE THE SHOT, they might help their team .The ODIM Framework: The ODIM Framework Twitter: @LMaccherone Start with the outcomes. You can think of this as the agile version Vic Basili’s Goal-Question-Metric. The difference between ODIM and GQM is related to sin #1. This is for self improving feedback. The emphasis is insight for the folks doing the work .Sin #2 Using a convenient metric rather than one that provides critical insight: Sin #2 Using a convenient metric rather than one that provides critical insight Twitter: @LMaccheroneWhat is normal?: What is normal? Cumulative -> 0.1% 2.3% 15.9% 50.0% 84.1% 97.7% 99.9% Roughly -> 85% 98% Twitter: @LMaccheroneAre you normal?: Are you normal? Twitter: @LMaccheroneThe A-Team Data: The A-Team Data 3X to 10X the risk Twitter: @LMaccheroneRisk calculation off: Risk calculation off 3x-10x when assuming Normal distribution 2.5x-5x when assuming Poisson distribution 7x-20x if you use Shewhart’s method Heavy tail phenomena are not incomprehensible… but they cannot be understood with traditional statistical tools. Using the wrong tools is incomprehensible. ~ Roger Cooke and Daan Nieboer Twitter: @LMaccheroneSin #3 Bad analysis: Sin #3 Bad analysis Twitter: @LMaccheroneBad application of control chart: Bad application of control chart Control is an illusion, you infantile egomaniac. Nobody knows what's gonna happen next: not on a freeway, not in an airplane, not inside our own bodies and certainly not on a racetrack with 40 other infantile egomaniacs. ~Days of Thunder Twitter: @LMaccherone When you try to "control" cycle time. .. When you try to reduce variability, you (ironically) invariably end up hiding information from yourself because nobody wants to be responsible for those red colored dots. They will artificially split work so nothing ever goes outside of the line.Time in Process (TIP) Chart A good alternative to control chart: Time in Process (TIP) Chart A good alternative to control chart Twitter: @LMaccherone Rather, show the scatter and distribution and enable discussion. Maybe there is information in the outliers that the team can discuss to remove obstacles and speed things up in the future. Maybe that's where the research work is being done and it's actually desirable to have a few Black Swans .Sin #4 Motivating folks to hide information: Sin #4 Motivating folks to hide information Twitter: @LMaccheroneCollection: Collection Perceived cost is higher than actual cost but developer happiness is key to success. Plenty to learn in passive data from ALM and other tools Use a 1-question Net Promoter Score (NPS) surveys for turning qualitative insight into quantitative insight How you use the tools will drive your use of metrics from them Twitter: @LMaccheroneSin #5 Too costly measures: Sin #5 Too costly measures Twitter: @LMaccheronePowerPoint Presentation: Twitter: @LMaccheroneSin #6 Too many measures (information overload): Sin #6 Too many measures (information overload) Twitter: @LMaccheronePowerPoint Presentation: Twitter: @LMaccherone You need a minimum of 4 measures (one from each quadrant) at the very start of your metrics regimeSin #7 Too few measures (unbalanced): Sin #7 Too few measures (unbalanced) Twitter: @LMaccheroneHeavenly virtues: Heavenly virtues Start with outcomes and use ODIM to make metrics choices. Make sure your metrics are balanced so you don’t over-emphasize one at the cost of others. Be careful in your analysis. The TIP chart is a good alternative to control chart. Consider collection costs and perceived burden. Get maximal value out of passively gathered data. Data visualization is like photography. Impact is a function of perspective, illumination, and focus. ~Larry Maccherone Twitter: @LMaccheronePowerPoint Presentation: LARRY MACCHERONE Twitter: @ LMaccherone You do not have the permission to view this presentation. In order to view it, please contact the author of the presentation.