Bates_-_SE_&_DA_Conference

Slide 0: 

Exploring a Range of Decision Making Techniques Nov 18,2009 John Bates ( bates_john@bah.com )

Slide 1: 

The Range of Decision Making Techniques Considerations when selecting a technique… How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have? Discussion Group (BOGSAT) Poorly Done Decision Analysis Decision Analysis Incorporating Mod & Sim Decision Analysis Incorporating SMEs Coin Toss or Guess Least Desirable - Most Desirable Advantages/Disadvantages to each technique. Focus of Today’s Discussion Fielded system; record data throughout lifecycle. Operational Exercise and Testing

Slide 2: 

Symptoms of poorly done decision analysis Lack of creative and significant Alternatives Solving the Wrong Problem/Objective Not involving the real decision makers Competing Objectives Avoiding Uncertainty Success measured on weight and thickness of final brief Not sure about the definition of the attributes Provided the solution at the beginning

Slide 3: 

How important is your decision?

Slide 4: 

Recommend a structured process that includes all stakeholders and is traceable/defendable Scoring scales can be nominal, ordinal, interval, or ratios Utility curves transform scoring scales into a common measurable output (Utility) Weighting of the tree structure determines relative importance There are various methods to determine importance AHP Relative Weighting Overall Goal: Determine best solution to fill need Sub-Goals: Maximize Technical Performance, Measures of Effectiveness Maximize Operational Suitability Minimize Risks, etc. Criteria must be independent, discriminatory, meaningful, and measurable Assess each alternative relative to the criteria in the decision tree Utility curves then transform the scoring scales into measurable data ranging from 0 to 1 with 1 being the most preferred Rank ordering of alternatives can be further decomposed to determine major contributors to the overall decision Conduct estimates for each alternative Can use cost objectives and thresholds Cost estimates and be treated separate or added into the evaluation model Sensitivity analysis can be conducted to analyze the results due to: Changes in weights Changes in scores Utility scores can be can be graphed versus cost to determine the "Best Value" alternative Determine Requirements Identify/Clarify Goal Define Criteria 1 Determine Priorities Develop Utility Curves Develop Scoring Scales 2 Conduct Sensitivity Analysis Determine “Best Value” 5 Assess Alternatives Rank Order Alternatives 4 Generate Alternatives Develop Estimates Incorporate Estimates into Model 3 Understand Problem/Vision Modeling Decide and Communicate

Slide 5: 

Prioritization (Weighting) Techniques Analytic Hierarchy Process Weightings To Prioritize User Needs Calculation: 1 / 2.667 = 0.375 Sum Rows Calculate Row Average (Called the Row Average of the Normalized Columns (RANC) method) Assign Scores Using a Scale Sum Column Developed by Thomas L. Saaty in the 1970s. The AHP provides a comprehensive and rational framework for structuring a decision problem, for representing and quantifying its elements, for relating those elements to overall goals, and for evaluating alternative solutions. Determine Priorities Develop Utility Curves Develop Scoring Scales 2 Importance 1 Equal 3 Moderate 5 Strong 7 Very Strong 9 Extreme

Capturing SME input: 

Capturing SME input Participants will be asked to evaluate criteria through a series of pairwise comparisons, which calculates each factor’s relative importance. To capture these comparisons, each participant uses a remote control to assign a number (1-9) that reflects the relative weight or importance of each decision factor. Determine Priorities Develop Utility Curves Develop Scoring Scales 2 # Inputs = (N * N-1)/2

Slide 7: 

Methodology for collecting group input Pairwise Comparisons ensure all combinations are considered independently The Analytic Hierarchy Process is used to determine the criteria priority. Pairwise comparisons are made for each unique combination. During the session, the facilitator will focus on areas of disagreement by identifying divergent points of view and encouraging participants to elaborate. Participants are encouraged to state their opinions and change their minds – and votes – if they so desire. During facilitated discussion, participants may continue to disagree on the relative importance of different decision factors but they will have a better understanding of different views within the group. Determine Priorities Develop Utility Curves Develop Scoring Scales 2

Slide 8: 

Decision Analysis at multiple levels? Mission Areas UIC (ASSIGNED TO RESPORG) Capability Area 1 Capability Area 2 Capability Area 3 PESTO Resource Elements Capability 2 Capability 3 Capability 1 O T S E P NTA2 NTA1 Joint Capability Areas (Future) NUMBER OF INPUTS REQUIRED ~10 ~50 ~250 ~3000 Determine Priorities Develop Utility Curves Develop Scoring Scales 2

Relative Scoring Example: 

Relative Scoring Example Solution Mission A Formula % Importance Car 1 =1/4 25% Tank Plane 3 =3/4 75% Boat TOTAL 4 100% Criteria Scale Supports Mission Success 1 Important to Mission Success 3 Critical to Mission Success 9 Determine Priorities Develop Utility Curves Develop Scoring Scales 2

Slide 10: 

Expert Choice – Great discussion/consensus using clickers, takes a lot of time Group Systems – Great at capturing multiple user input simultaneously , can be performed remotely (surveys), requires extensive setup. Excel AHP – Used to Prioritize Resource Categories within a level - Simple to setup, easy to facilitate potential for a lot of intersections ((n * n-1)/2), lose input variation and correlation checking Excel (0,1,3,9) Scale – Simple, quick, needs to be strongly facilitated, lose input variation and consistency check The choice of techniques is primarily based on number of intersections and time needed to perform the analysis TIME TO COLLECT Summary of Different Prioritization Techniques

Slide 11: 

Scaling Examples for Alternative Scoring Utility Signature Improvement in undetectable range (meters) 1 0 0 1000 Utility Range (meters to Refuel) 1 0 2142 6000 The range for the Measure of Merit for Signature is based on the improvement in detection range the vehicle realizes through IR signature reduction. The range of the Measure of Merit for Range is based on the ORD Requirement on meters to recognition. Determine Priorities Develop Utility Curves Develop Scoring Scales 2

Slide 12: 

Alternative Comparison Technique DIV A DIV B DIV C Capability/MOE Definition : The ability to transport passengers 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 …. T O T O T O Number of Passengers Determine Priorities Develop Utility Curves Develop Scoring Scales 2

Functional Analysis: 

Functional Analysis Conduct Operations AND AND Command and Control (C2) Survive Threats and Conditions Surveillance (Find/Fix/Track/ID/Assess) Launch Recover Embark Team/Equip Transit to OpArea Transit from OpArea Interdict Escort Vessel Force Protection Logistics &Service Support Plan/Direct Organize Train/Equip Arm/ Fuel C2 Employment (Air) Employment (Sea) Employment (Fwd Basing) Maneuverability/Navigation Employment (Sea) Employment (Fwd Basing) Speed Seakeeping OR Abort Mission Patrol Speed Deploy Waterborne Checkpoint, PSHD, HAD, Vessel Escort Enabling Tasks (Off-board) Enabling Functions (Off-board) Key Capabilities/MOPs LEGEND Capacity 1 3 4 5 9 11 8 10 12 13 Detectability OR Speed Lethality Sustain Force 15 14 Endurance Loiter 7 16 Staging Survivability 6 Transportability 2 Systems Engineering-based Functional Decomposition provides structured process to define Required Mission Capabilities to assess Alternatives

Slide 14: 

Modeling and Simulation Techniques Discrete Event Simulation Behavior/Functional Modeling Physical or Prototype Modeling Wargaming/Simulations Sensitivity Analysis Linear Programming Reliability ,Failure Modes and Effects Analysis Mathematical/Monte Carlo or Probability Modeling

Slide 15: 

Method to Communicate Analysis Results of Analysis 1. System B 2. System A 3. System C

Slide 16: 

Method to Communicate Analysis TRL LEVELS 9-10 7-8 <6

Slide 17: 

Another Method of Communicating Decisions This quad chart represents the capability areas that should be pursued first. Quad 1 High Priority Poor Capabilities Quad 3 Low Priority Poor Capabilities Quad 4 Low Priority Good Capabilities Quad 2 High Priority Good Capabilities

Slide 18: 

The Range of Decision Making Techniques So what are the key considerations when selecting a technique… How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have?

Questions: 

Questions ?