Book Risk Panel

Tenet #4: 

Tenet #4 NASA Cost-Risk Assessment is Composed of Cost-Estimating Relationship (CER) and Technical Risk Assessment plus Cost-Element Correlation Assessment

A Project’s Technical Descriptionis Not Enough: 

A Project’s Technical Description is Not Enough A Technical Description (as provided in the CARD, for example) Does not Contain All Information Needed for a Realistic Cost Estimate The Technical Description Does not Describe How Difficult It is to Build the System, vis-à-vis … … Beyond State-of-the-Art Technology … Software Development, Integration, and Test … Other Risk Issues Yet System Cost Depends Heavily on How Difficult it is to Overcome the Risk Issues Difficulty Can be Translated into Additional Money and/or Additional Time Ignoring Such Difficulty Can (and Does) Lead to Cost Overruns and Schedule Slips

The Risk-Management Plan: 

The Risk-Management Plan A Project’s Risk-Management Plan Supplements its Technical Description by Providing Project Managers with Additional Information A “Watch List” of Risk Issues that May Cause Problems in Bringing the Project to a Successful Conclusion on Budget and on Time An Assessment of How Each Listed Risk Issue Can be Circumvented or Satisfactorily Resolved An Estimate of Additional Time and Resources, Including Personnel, that May Have to be Applied to Each Risk Issue Information from the Risk-Management Plan Can Support the Cost-Estimating Process (Additional Time)x(Additional Personnel) = Additional Cost But Not All Risks Will Come to Pass – That’s Why They are Discussed in the Risk-Management Plan, Rather than the Technical Description

Error Sources in Estimating Costs: 

Error Sources in Estimating Costs Basic Estimating Methodology Statistical Error Inherent in CER Not Quite Perfect Analogy Variability of Bottom-up Assessment Unreliability of Vendor Quote Characteristics of Specific Program Technical Risk Programmatic Risk, Including Schedule Risk Risk Associated with GFE and COTS

Error Sources of CER-BasedCost Estimates: 

Error Sources of CER-Based Cost Estimates Inability of Any CER to Account for All Influences on Cost, No Matter How Many Inputs it Allows Incorrectness of Algebraic CER Model to which Cost Numbers in Data Base are Statistically Fit Explicit CERs are Derived from Historical Cost Data by Minimizing a Quality Metric, typically the Standard Error of the Estimate (SEE), that Depends on the Algebraic Model SEE Is Calculated by Minimizing Sum of Squared Differences between CER-Based Estimates and Actuals (in Either Dollar or Percentage Terms) and Dividing by a Factor Involving Number of Data Points Contributing to Development of CER SEE is an Estimator of “True” Standard Deviation  of “Errors” in the Knowledge Base of Historical Cost Data Points, Assuming the Algebraic Model is Correct Location of Cost Driver Value x among Parameter Values Comprising Historical Cost Data Base If x is Located Near Center of Range of Parameter Values, CER will Provide Fairly Precise Estimate of the System’s Cost If x is Located Far From Center of Range, CER-based Estimate will be Considerably Less Precise

CER-Based-Estimating State of the Art: 

CER-Based-Estimating State of the Art Ordinary Least Squares (OLS) Model Cost as a Linear Function of One or More Cost Drivers Estimating Problem is Completely Solved Explicit Algebraic Formulas Exist for the Upper and Lower Bounds of Confidence and Prediction Intervals for Any Value of Cost-Driving Parameter at Any Level of Confidence Width of Interval Depends on Both the CER’s Standard Error of the Estimate and Location of the Cost-Driver Value x Special Nonlinear CER Forms Model Cost as One of a Particular Class of Nonlinear Functions Such Nonlinear Forms Can be Made OLS-Solvable by an Algebraic (usually Logarithmic) Transformation Confidence and Prediction Intervals Can be Calculated in a Roundabout Way by Applying the Inverse Transform Unfortunately, the Geometric Distortion that Results from the Inverse Algebraic Transformation Makes It … … Impossible to Establish Symmetric Intervals … Difficult to Compute the Most Efficient Intervals Based on the Data Available General Nonlinear CER Forms Model Cost Using Any Nonlinear Functional Form Standard Error of the Estimate Can be Calculated, as Well as Some Information About Variances of the Coefficients But Problem of Confidence and Prediction Intervals Appears Not to Have Been Solved

Precision of Estimate Over Entire Range of Possible Cost-Driver Values: 

Precision of Estimate Over Entire Range of Possible Cost-Driver Values Cost Driver Mean = 67.30 Standard Error of Estimate = 5.26 Cost Driver Range: 53 to 79

Bounding the Predicted Cost at Cost-Driver Value x: 

Bounding the Predicted Cost at Cost-Driver Value x Prediction Interval Based on the Variance of the Difference Between the Actual Cost Y and the Estimated Cost Ŷ is   Degree of Confidence Associated With This Interval is Again (1-)100%, which is Enforced by the Choice of the “Percentage Point of the t Distribution,” Namely t/2,n-2

Fred Timson on Value of Prediction Intervals to Cost Estimators: 

Fred Timson on Value of Prediction Intervals to Cost Estimators “The prediction intervals are so wide for even the best regression that there is little likelihood that the realized cost of a future weapon system acquisition program will be near the predicted cost.” “The predictive statistics (sampling distributions for future observations) overlap to such an extent for some regressions that the ability to discriminate between the distributions for airframes that differ in weight by a factor of two is very doubtful.”

Mathematical Issues in Prediction Using General-Error Regression CERs: 

Mathematical Issues in Prediction Using General-Error Regression CERs Deriving the General-Error CER For the Model yi = (a + bxic)*Ei, the Sum of Squared Multiplicative Errors is to be Minimized It is Generally Impossible to Obtain Explicit Formulas for A, B, and C by Calculus or any Other Mathematical Method, so Some Kind of Iterative Procedure is Needed for Convergence to Acceptable Numerical Values of the Coefficients In 1974 Wedderburn Published Approximate Expressions for the Variances and Covariances of the Coefficients A, B, and C – However, He Did not Carry Through his Derivation to the Point Needed for Prediction Intervals, Namely to the Point of Calculating the Variance of the CER-based Estimate Itself

Wedderburn’s Matrix for IRLS-Based CERs of the Form Y = a+bXc: 

Wedderburn’s Matrix for IRLS-Based CERs of the Form Y = a+bXc Wedderburn’s Variance/Covariance Matrix Has the Form In the Case of Y = a+bXc, D is the Matrix Inverse of

Specific Program-Related Risks: 

Specific Program-Related Risks Define Impact of Program-Related Risks Using WBS-Element Probability Distributions Technical Risk, e.g., Probable Additional Development Costs due to Requirements for Beyond State-of-the-Art Technology Programmatic Risk, e.g., Probable Additional Costs Associated with Schedule Slippage due to Various Causes GFE and COTS Risk, e.g., Probable Additional Funding Needed to Cover for GFE and COTS Inadequacies Program-Related Risks are Typically Correlated Ignoring or Failing to Account for Inter-Element Correlation Leads to Narrow Total-Cost Distributions Assumption that Correlations are Negligible Masks Estimating Uncertainty

Tenet #6: 

Tenet #6 NASA Cost-Risk Probability Distributions are Justifiable and Correlation Levels are Based on Actual Cost History to the Maximum Extent Possible

A Project’s Technical Descriptionis Not Enough: 

A Project’s Technical Description is Not Enough A Technical Description (as provided in the CARD, for example) Does not Contain All Information Needed for a Realistic Cost Estimate The Technical Description Does not Describe How Difficult It is to Build the System, vis-à-vis … … Beyond State-of-the-Art Technology … Software Development, Integration, and Test … Other Risk Issues Yet System Cost Depends Heavily on How Difficult it is to Overcome the Risk Issues Difficulty Can be Translated into Additional Money and/or Additional Time Ignoring Such Difficulty Can (and Does) Lead to Cost Overruns and Schedule Slips

The Risk-Management Plan: 

The Risk-Management Plan A Project’s Risk-Management Plan Supplements its Technical Description by Providing Project Managers with Additional Information A “Watch List” of Risk Issues that May Cause Problems in Bringing the Project to a Successful Conclusion on Budget and on Time An Assessment of How Each Listed Risk Issue Can be Circumvented or Satisfactorily Resolved An Estimate of Additional Time and Resources, Including Personnel, that May Have to be Applied to Each Risk Issue Information from the Risk-Management Plan Can Support the Cost-Estimating Process (Additional Time)x(Additional Personnel) = Additional Cost But Not All Risks Will Come to Pass – That’s Why They are Discussed in the Risk-Management Plan, Rather than the Technical Description

Achievable Software-DevelopmentSchedules: 

Achievable Software-Development Schedules

CER for Military SpaceGround-System Test Software: 

CER for Military Space Ground-System Test Software

Software Cost-Risk Experience: 

Software Cost-Risk Experience Cost Histories of Software-Development Projects Show a Definite Trend Toward Significant Underestimation of Number of Lines of Code and Cost Aerospace Corp. Study Found Lines-of-Code Growth of about 150% for Space-Related Ground-System Software Projects Naval Center for Cost Analysis Found Average Lines-of-Code Growth of 63% for Software Projects of Various Types (http://www.ncca.navy.mil/software/handbook/software.htm) Developer Productivity, Measured in Lines of Code per Developer-Month, is Typically Overestimated This Results in Cost Growth, Even if Lines-of-Code Estimate is Accurate Data Collected Over Time Appear to Show Some Productivity Improvement, but not Enough to Overcome Estimating Optimism

Lines-of-Code Estimating Risk: 

Lines-of-Code Estimating Risk

Slide20: 

Historical Software Coding Rates

What is the Risk Multiple for Software?: 

What is the Risk Multiple for Software? It’s 8 (times the contactor estimate) Why? Number of Lines of Code Grows by a Factor of About 2.5 Programmer Productivity, Almost Always Initially Estimated at 300 Lines Per Programmer-month, Inevitably Slips to Around 85 As the Project Moves Forward: Equivalent to a Cost-growth Factor of 300/85 = 3.5 2.5 × 3.5 = 8.75 So We’re Being Nice About It This Multiple is Applied Wherever in Each WBS Element the Cost of Software is Estimated The Cost Distribution Will Be Right-Triangular L = M, but H = 8M

Maximum Possible Underestimation of Total-Cost Sigma (Theoretical): 

Maximum Possible Underestimation of Total-Cost Sigma (Theoretical) Percent Underestimated When Correlation Assumed to be 0 Instead of r

Selection of Correlation Values: 

Selection of Correlation Values “Ignoring” Correlation Issue is Equivalent to Assuming that Risks are Uncorrelated, i.e., that All Correlations are Zero Square of Correlation Represents Percentage of Variation in one WBS Element’s Cost that is Attributable to Influence of Another’s Reasonable Choice of Nonzero Values Brings You Closer to Truth Most Elements are, in Fact, Pairwise Correlated 0.2 is at “Knee” of Curve on Previous Charts, thereby Providing Most of the Benefits at Least Commitment