Cost/Benefit-Aspects of Software Quality Assurance Master-Seminar Software-Quality :Cost/Benefit-Aspects of Software Quality Assurance Master-Seminar Software-Quality Wednesday, 4th June 2008 Marc Giombetti


Overview :Overview Software Quality Measurement & Estimations Software Quality Costs Costs & Schedule: COCOMO & COCOMO II Quality, Costs & Schedule: COQUALMO Return on Investments of Quality Assurance – Case Study Estimations Reflections on Cost Effectiveness & Quality Future development & outlook


Software Quality Measurement & Estimations :Software Quality Measurement & Estimations Software (product) quality attributes (ISO 1926) Functionality Reliability Usability Efficiency Maintainability Portability Measurability affects estimations: Availability vs. Usability Defining the right quality for a certain purpose. Availability – 99,9%  8,76 hours/year Word processor vs. Nuclear power plant control software Requirements to these characteristics affect the quality & price of the software,but: difficult to measure


Software Quality Costs :Software Quality Costs There is a price to pay for quality. (Quality is not Free) Types of software quality costs – Cost of Quality Approach: Prevention: Avoiding defects, planning, preparation, training, reviews, tool costs. Quality Assurance prevents the injection of various types of defects. Appraisal: Finding defects by inspection, audit, developing test cases, test and measurement. Internal Failure: Costs emerging during development. Rework, redesign, modifications, corrective action, down time, concessions and overtime. External Failure: Costs emerging from a failure at the customer: Equipment failure, loss of sales, downtime, administrative cost in dealing with failure, law costs, loss of reputation.


Constructive Cost MOdel: COCOMO :Constructive Cost MOdel: COCOMO First version developed by Barry Boehm in the 1970s. Estimates the development effort based on: Project Characteristics / Type of project Project size in KDSI. Effort adjustment factors EAF (intermediate/detailed model) Calibration based on static analysis of 63 projects EFFORT = a ∙ EAF ∙ (KDSI)b


Constructive Cost MOdel: COCOMO :COCOMO Cost Drivers:EAF = Effort Adjustment Factors Product of the cost drivers: Constructive Cost MOdel: COCOMO EFFORT = a ∙ EAF ∙ (KDSI)b organic: EFFORT = 2,4 ∙ EAF ∙ (KDSI)1,05 semi-detached: EFFORT = 3,0 ∙ EAF ∙ (KDSI)1,12 embedded: EFFORT = 3,6 ∙ EAF ∙ (KDSI)1,20


COCOMO II :COCOMO II COCOMO II was developed during the 1990s Adjustment to modern development lifecycles Calibration on a broader set of data Cost estimation models are necessary to Gain certainty in respect to the price/effort. Market competitiveness  Avoid loss for fix price projects Drawbacks of COCOMO and COCOMO II Hard to accurately estimate the KDSI of the project. Extremely vulnerable to miscalibration. Good performance requires historical data, which might not exist ( Experience Factory)


COnstructive QUality MOdel – COQUALMO (1/4) :COnstructive QUality MOdel – COQUALMO (1/4) Extension to COCOMO II Impact of various defect removal techniques Interaction among strategies Relates Cost, Schedule and Quality. Defect Introduction Model Defects based on their origin (requirements, design, code) Severity of defects (medium, high, critical) Defects conceptually flow into a „holding tank“


COnstructive QUality MOdel – COQUALMO (2/4) :COnstructive QUality MOdel – COQUALMO (2/4) Defect Removal Model Post processor of the Introduction Model Predicts the residual defects per KDSI Estimated number of residual defects in artifact j:


COnstructive QUality MOdel – COQUALMO (3/4) :COnstructive QUality MOdel – COQUALMO (3/4) Defect Removal Profiles: Defect Removal Fractions are associated to every element of the matrix:


COnstructive QUality MOdel – COQUALMO (4/4) :COnstructive QUality MOdel – COQUALMO (4/4) Example: Estimated delivered defect densities: Removal Rating: Very Low: 10 Requirements defects, 20 Design defects, 30 Code defects


Return on Investments in Software Quality Assurance – Case Study (1 / 3) :Return on Investments in Software Quality Assurance – Case Study (1 / 3) Analyzed projects : Mountain bike order-processing system & NASA Planetary Rover Different project characteristics, project value Different requirements in respect to availability Different costs in case of a failure / down time. Information Dependability Attribute Value Estimation (iDAVE-Approach by Barry Boehm et al.) Return on investment in respect to quality (RELiabilitY) Quality assurance is seen as an investment (cost equations in business)


Return on Investments in Software Quality Assurance – Case Study (2 / 3) :Return on Investments in Software Quality Assurance – Case Study (2 / 3) Availability improvement from Nominal  High Mountain bike order-processing system (Total cost 531M$) – Increased value (-1% availability ~ 1% system value loss): Return on investment of the quality improvement:


Return on Investments in Software Quality Assurance – Case Study (3 / 3) :Return on Investments in Software Quality Assurance – Case Study (3 / 3) ROI comparison of theorder processing systemand the NASA PlanetaryRover :Improvements for changing to the nexthigher level of availability. Negative ROIs are not necessarily a reason to stop QA. Further improvements reduces the risk of mission failure Important customer relationship, loss of reputation Other motivators emerge (e.g. reduce security risks)


Estimations :Estimations „To estimate the time, cost and quality of next time, you must know and be able to repeat what you did last time“ Advantages Removes uncertainty Are necessary to predict budget, devlopment time  competitiveness Comparability of estimates and actual performance Disadvantages Estimates are always made a priori There is no general technique which leads to the best estimate. Requires a lot of experience Techniques: Delphi method, Wideband delphi Keep track of estimates (also wrong ones  Experience Base)


Reflections on Cost Effectiveness & Quality :Reflections on Cost Effectiveness & Quality Model Calibration Empirical data; Expert opinion Empirical software engineering: repeatability of experiments Considerations in respect to costs: Assure: Changeability & Maintainability (  otherwise: loose business advantages, lawsuits, …) Total Cost of Ownership, consider: Hardware / Software / Maintenance Power consumption in datacenters / Virtualization  Green IT Usability User frustration, ineffective work, loss of time Expensive training for new employees Tool support is a necessity, but quality estimation tools are rare: Construx, Circa –Tassc, …


Future development & outlook :Future development & outlook Presented techniques are a good starting point. Cost/quality awareness  Top-Management There exist no push-button techniques. Future development in companies: Collection of project data to improve estimations  Experience Factory Future development among companies / research: Improve existing prediction methods & empirical work Find new approaches to measure quality attributes Improve estimation tools to handle complexity (data mining to capture patterns in project execution data)


Summary :Summary Software Quality Measurement & Estimations „Cost and quality is all about estimations and measurability“ Software Quality Cost Competitiveness in the market, Software quality plays a major role, but not at any price. COCOMO & COCOMO II Cost estimation COQUALMO Quality estimation ROI of Software Quality Assurance Cost effectiveness Reflections on Cost Effectiveness & Quality Future development & outlook


Discussion :Discussion


Slide 20:BACKUP-Slides


COCOMO Example :COCOMO Example Project Characteristics Mode: organic Size = 200 KDSI Cost drivers Low reliability => 0,88 High product complexity => 1,15 Low application experience => 1,13 High programming language experience => 0,95 Other cost drivers assumed to be nominal => 1,00 EAF = 0,88 ∙ 1,15 ∙ 1,13 ∙ 0,95 = 1,086 Effort = 2,4 ∙ 1,086 ∙ 2001,05 = 679 MM Development time = 2,5 ∙ 6790.38 (TDEV = c * MMd) BACKUP-Slide