The influence of software quality requirements on the suitability of software cost estimation methods 24th International Forum on COCOMO and Systems/Software Cost Modeling :The influence of software quality requirements on the suitability of software cost estimation methods 24th International Forum on COCOMO and Systems/Software Cost Modeling Marc Giombetti (TUM & MIT)Dr. Ricardo Valerdi (MIT)Dr. Stefan Wagner (TUM)


Outline :Outline Motivation Advantages and limitations of different software cost estimation methods Understanding & estimating software quality A controlled experiment – COCOMO II vs. Expert judgment Conclusions & Future work


Formal estimation models or expert judgment ? (1/2) :Formal estimation models or expert judgment ? (1/2) M. Jørgensen and B. Boehm, “Software Development Effort Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19. Formal estimation models or expert judgment […] figure out when, and under what conditions, each method would be best.  This research adjusts this question and asks: When are formal models more useful than expert judgment; given a set of specific quality requirements.


Formal estimation models or expert judgment ? (2/2) :Formal estimation models or expert judgment ? (2/2) Formal Estimation Models COCOMO, FP based models Mechanical quantification step Dependent on calibration data Objective & repeatable ? Industry usage? – Trust in a method which is not fully understood by PM? Expert judgment Checklist-based estimation, WBS, “gut feelings”, Judgment-based quantification step Dependent on individual estimator Biased towards optimism Only 15% of the journal papers analyzed judgment-based effort estimation* The difference between formal estimation models and expert judgment is not always clear  Combination of methods * M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53.


Software Quality :Software Quality Software quality is an important factor for the success of software of any kind. How much software quality assurance is enough?* For several industries quality is not negociatable! One of the challenges of Software Quality is that "everyone feels they understand it“** Quality requirements need to be taken into consideration while estimateing the cost of future projects. * L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95.   ** P.B. Crosby, Quality Is Free, Signet, 1980.


ISO/IEC 9126 - Software Quality Model :ISO/IEC 9126 - Software Quality Model International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001.


The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (1/4) :The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (1/4) Estimation for development time (no schedule compression) Expert judgment : Top-down or bottom-up approach H1 H2


The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (2/4) :The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (2/4) System requirements and project settings Domain: Financial application Type: Web application Development team size: 1 Project Manager1 Senior developer,1 QA, 3 Junior developers Expert estimation: Graduate students Security: 5 students Usability: 6 students


The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (3/4) :The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (3/4) Research hypotheses H10: In the case of additional security requirements There is no significant difference between the average development time estimated using expert estimation and the intermediate COCOMO II model estimate. H20: In the case of additional usability requirements: There is no significant difference between the average development time estimated using expert estimation and the intermediate COCOMO II model estimate.


The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (4/4) :The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (4/4) Experiment Goals G.1. Analyze the relative impact of security and usability requirements on the development time of a web-application using intermediate COCOMO II. G.2. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by security requirements made using expert estimation compared to intermediate COCOMO II. G.3. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by usability requirements made using expert estimation compared to intermediate COCOMO II.


Experiment: COCOMO II Estimate :Experiment: COCOMO II Estimate


Experiment: Results (1/2) :Experiment: Results (1/2) Different mental models for the concepts security and usability???


Experiment Results (2/2) :Experiment Results (2/2) T-Test & Wilcoxon signed rank test at 5% significance level: Reject H1  There IS a significant difference between the estimates of expert estimatiors and COCOMO II for the security enhancement. Reject H2  There IS a significant difference between the estimates of expert estimatiors and COCOMO II for the usability enhancement. Threats to validity Students in the role of expert estimators* Classroom exercise: The requirements provided to the students were simplified, but chosen to be comparable to real requirements (in size & complexity). Selection of quality requirements Small dataset * P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176.


Conclusions & Future work :Conclusions & Future work Perception of quality attributes influences the estimators, but not the COCOMO model estimate. Human estimators underestimated security; overestimated usability Influence of quality for cost estimation - Improvements Conduction of experiment with more participants Planned: Graduate students at TU Munich Preferable: Software professionals with real estimation experience Focus on other quality aspects (maintainability, reliability,…) Conduction of a second experiment on the influence of usability (Focus on FP counting & COCOMO II)


Future work: COCOMO II – Estimating the influence of usability :Future work: COCOMO II – Estimating the influence of usability


Slide 17:Questions?


References :References M. Jørgensen and B. Boehm, “Software Development Effort Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19.   M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53. L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95. P.B. Crosby, Quality Is Free, Signet, 1980.     International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001. P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176.


Slide 19:Backup Slides


Expert estimation approach chosen by participants :Expert estimation approach chosen by participants