Quantitative Risk Management: Quantitative Risk Management Tom Tuduc


Introduction : Introduction Risk is ubiquitous - We are all familiar with risks: RISK board games and video games. Download risk-free product trials Buy products that reduce risks of illness Terrorist threats Take calculated risks (video clip ReturnOfTheKing) Avoid running the risk of ... Risk attitude Eliminating risk by getting more information and/or controlling outcome (video clip ValueInfoControl) Who is at risk, high risk groups (video clip highrisk)


Summary: Summary Homeland Security is complex and include uncommon and/or hypothetical uncertainties. It takes both qualitatively and quantitatively models to consider hundreds of intelligences with different credibility and accuracy. Topics: 1. How Influence Diagrams/Decision Analysis help experts communicate and model Homeland Security decisions, threats, and countermeasures visually, qualitatively, and quantitatively. 2. How Decision Analysis enable calculations that maximize security, make decision policies, quantify insights of each threat factor, and the worth of additional information and control on each factor. 3. A review of several examples in the literature: influence diagrams in terrorist threat countermeasures, early warming systems, toxin containment policies, and intrusion-aware information systems. 4. Security categories, Application trends, technology integration possibilities, and online resources


Table of Content: Table of Content Introduction Summary Table of Content What is Security Analytics? Security Risk Methodology - the Four Steps Risk Management: Dealing with Uncertainty Example of Security Application Areas Characteristics of complex risk problem, Decision Analysis & Influence Diagrams Tradeoffs & Risk Preference Differences between Trees and Diagrams Certainty Equivalence, Utility & Risk Premium Risk taking Risk averse Tutorial Example Best Policy and value of Control Risk Profiles Gaining Insights Sensitivity Analysis Similar security ROI starting point Similar Intrusion Detection problem A more complex party problem A more complex security ROI problem Complex Intrusion-Aware Model Homeland Security Infrastructures & Assets Homeland Security - System View Homeland Security - Decision View Infrastructure Elements Homeland Security Decision Analysis & Influence Diagrams Examples Example 1 Overarching Influence Diagram Example 2: Site Profiler Architecture & Influence Network Example 3 – Using Analytica       Security Categories Where are the numbers Tools & Resources Conclusion


Overview : Overview DEFINITION: Security Analytics (Table1) are the use of analytics  to optimize security and security ROI. Applications: Model Processes Policies Systems Game theory Utility theory Negotiations Markov Graph theory Information theory Stochastic Dynamic programming Probability Statistics


Security Methodology - the Four Steps: Security Methodology - the Four Steps 1. Determine risk: Assets and risk to assets. Making security ROI known. 2. Analyze risk: * Qualitative Quantitative: Analytics 3. Design and Implement: policies, architectures, technologies, trainings, and countermeasures 4. Management: Monitoring, audits, and evaluate  * "One of the major problems is that security risk assessment and the benefits of using the results of risk assessment cannot be measured in any sufficiently accurate to provable way... Positive benefit is absence of unknown possible loss" Tom Peltier, "Risk Analysis Vs. Security Controls." NetSec 2002


Risk Management: Dealing with Uncertainty: Risk Management: Dealing with Uncertainty  Fundamental Approaches Frequentist Based on hundreds or thousands of events. Probability lies objectively in the world, not in the observer.   Bayesian Based on personal experience. Probability is different for people having different past experiences.


 Example of Security Applications:  Example of Security Applications Security ROI   Risk assessment and management  Knowledge management and Information retrieval  (1) SPAM filtering (2) Intrusion Detection Systems Other examples: Search engines, portfolio management, polling, etc. (1) 21 US agencies with 200,000 employees has deployed Autonomy, a knowledge management tool based on Bayes and Shannon theorems, for homeland security functions (Business Weekly, 31 October 2002.) (2) Baysian-based SPAM filters:  http://www.webarches.com/filters.html


Characteristics of complex problems: Characteristics of complex problems Many uncertainty/probabilities cannot be obtained from empirical frequency distributions because the events are uncommon and/or hypothetical. Probabilities come from expert opinions with different experience of the same problem In a closed-loop system, the probabilities improves over time with repeated cycles. Time is a luxury not always available.


Decision Analysis/ Influence Diagram (DAID) : Decision Analysis/ Influence Diagram (DAID)  Advantages


 Modeling:  Modeling Decisions: made by the decision maker Uncertain events: events with discrete outcomes or probability functions Consequences: values resulting from the decisions and uncertain events outcomes Risk Preferences: how the decision maker feels about the consequences (1) Objectives: direction and value, i.e. eliminate risk areas, maximizing ROI, minimizing loss of data and/or resources. (1) Will the real risk-preference stand up: A popular misconception is that security managers in private sectors are risk-averse and overspend on security. IDC research data shows otherwise. A typical organization of 5,000 employees, on average spends $1 million on security products ($200/person, or $500 for each $1 million in revenues).   


Tradeoffs & Risk Preference: Tradeoffs & Risk Preference Conflicting objectives: A policy may be optimal for one objective, but not for all objectives, i.e. how much expected loss of data availability is an agency willing to accept to increase data integrity to 100 percent. Tradeoffs (conflicting objectives): 10 percent loss in data integrity is equivalent to 50 percent loss in data availability Risk Preference: which Risk Profile is your organization's (video clip riskProfiles)


Differences between Decision Trees and Influence Diagrams: Differences between Decision Trees and Influence Diagrams Influence diagrams show dependencies among variables clearly: good visuals for communication and qualitative relationship. Influence diagrams are compact - one or two order of reduction in node representation in typical problems. Decision trees show details of possible paths/scenarios: relatively good visual for small problems. Best for quantitative calculations. Decision trees show asymmetric outcome trees.


Certainty Equivalence, Utility & Risk Premium: Certainty Equivalence, Utility & Risk Premium Common decision rule: maximize expected value, often expected monetary value. However, this is not realistic for the risk-averse. Better decision rule: Expected Value with minimum risk variance (portfolio investment) Best decision rule: maximize expected utility. Utility is found by presenting simple lotteries to decision makers. Certainty Equivalence: taking monetary equivalence instead of playing the lottery. Risk premium: EV of lottery - CE of lottery


RISK TAKING: RISK TAKING Risk premium = EV  - CE or -$2. CE is larger than EV Buying a lot of superlotto tickets is risk taking


RISK AVERSE: RISK AVERSE Even though EV is higher now (50 versus 23), Certainty equivalence is lower (25 versus -5)  Risk premium = 50 - 5 = $55. CE is less than EV This is analogous to hiring a consultant, or outsourcing instead of performing a function internally.


Tutorial Example: Tutorial Example


Best Policy and Value of Control: Best Policy and Value of Control The Influence Diagram and Decision Tree show the Location Decision is made independent of knowing the weather Conclusion: If we don't know what the weather will be, we should locate it on the porch because that has the highest payoff of $38 million. Best case saving: (60-38) or $22 million. This is Value of Control * * The Department of Energy benefits by eliminating security-update risks (Value of Control) when it required Oracle to deliver its 9i database with all security features and to take responsibility of maintaining security updates. This is an unusual but excellent example of cyber-security practice.


Risk Profiles: Risk Profiles Locating the party by the pool can give negative utility if it rains. But if it’s sunny it’s the best decision. If it’s cloudy, it might rain.


Gaining Insights: Gaining Insights 1. If we know what the weather will be, we can make a better decision. Thus the new expected payoff is now $47 million, instead of $38 million. 2. If we want to ask a security expert (clairvoyant) about what the weather will be, we should only pay a maximum of $9 million (new expected payoff - old expected payoff) New expected payoff: (.2*45) + (.5*40) + (.3*60)= 47 Old expected payoff: $38 3. New Value of Control: new best case saving is (60- Expected  Value) = 60-47 = 13


Sensitivity Analysis: Sensitivity Analysis


Similar Security ROI starting point: Similar Security ROI starting point Budgets: basic security (firewall, VPN, antiVirus), audits, realtime intrusion detection, advanced access control, encryption, etc.


Similar Intrusion Detection problem: Similar Intrusion Detection problem


Complex Intrusion-Aware Model: Complex Intrusion-Aware Model TRIAD (Trustworthy Refinement through Intrusion-Aware Design): an intrusion-aware model developed at CMU/SEI (TECHNICAL REPORT CMU/SEI-2003-TR-002) SUMMARY PROBLEM: Military and business systems face increasingly sophisticated and coordinated computer network attacks. Existing security system development are typically isolated solutions resulting in patchwork designs that are not robust under attack. TRIAD, a model, helps IT decision makers to formulate and maintain a coherent and justifiable survivability strategy that addresses mission-compromising threats. TRIAD uses DAID to model the dynamics of fraud and authentication.  TRIAD's goals are: Develop a development methodology for security systems to resist, recognize, recover from, and adapt to mission-compromising attacks. to provide a documented response to the primary threats to the mission; to provide a justification for and the limitations of the system design; to support the design and implementation of the desired system behavior across multiple systems and multiple development teams; and to support maintenance and evolution as the system operations and threat environment evolve over time.


Homeland Security Infrastructures & Assets: Homeland Security Infrastructures & Assets Critical Infrastructures Agriculture Food Water Public Health Emergency Services Government Defense Industrial Base Information and Telecommunications Energy Transportation Banking and Finance Chemical Industry and Hazardous Materials Postal and Shipping Key Assets National Monuments Dams Nuclear Power Plants Government Facilities Commercial Key Assets


Homeland Security- System View: Homeland Security- System View


Homeland Security- Decision View: Homeland Security- Decision View


Infrastructure elements: Infrastructure elements


Homeland Security Decision Analysis & Influence Diagrams Examples: Homeland Security Decision Analysis & Influence Diagrams Examples Probabilistic Modeling of Terrorist Threats: A Systems Analysis Approach to Setting Priorities Among Countermeasures Site Profiler, a system being used in bio-terrorism early warning systems, passenger and cargo profiling,  vulnerability assessments, threat warnings and dissemination. Using Analytica: Toxin Containment Model and Analysis TRIAD (Trustworthy Refinement through Intrusion-Aware Design): an intrusion-aware model Others: GIS and Decision Analysis Journal, COPLINKS (Search and match given incomplete information), Paper "Warning and Response in Homeland Security“, and Sandia/CA’s Weapons of Mass Destruction Decision Analysis Center


Example 1 & Influence Diagram: Example 1 & Influence Diagram Probabilistic Modeling of Terrorist Threats: A Systems Analysis Approach to Setting Priorities Among Countermeasures. Elisabeth Paté-Cornell and Seth Guikema. Department of Management Science and Engineering. Stanford University. Military Operations Research, Vol. 7, No 4, pp. 5-20 December 2002.  SUMMARY PROBLEM:   assess the benefits of risk reduction by different countermeasures and their costs OBJECTIVE: - Prioritize the protection of US infrastructures, networks and socio-economic components - Discover most effective means of reducing the overall threat, i.e. the disruption of the terrorists’ supply chain - Prioritize intelligence information that needs to be gathered given accuracy, time, and constraints.


Example 2 – Architecture and Diagram: Example 2 – Architecture and Diagram Site Profiler, a system developed by Bryan Ware, Anthony Beverina, Lester Gong, and Brian Colder at Booz Allen Hamilton and Digital Sandbox. Site Profiler is used in bio-terrorism early warning systems, passenger and cargo profiling,  vulnerability assessments, threat warnings and dissemination. Site Profiler applies DAID to combined data from various sources. SUMMARY PROBLEM: Build a system to sift through massive amount of information to determine terrorist risk OBJECTIVE: Determine the following: how likely a terrorist will attempt attacks including tactic, weapon, delivery system against an asset how likely the terrorist will succeed consequences of successful attacks CHALLENGES: High volumes of data. Disparate sources of data and information Diverse forms of information Significant organizational friction among producers, owners, and consumers of information


Example 3 – Using Analytica: Example 3 – Using Analytica Using Analytica to model and analyze the cost and benefit of Toxin Containment (Adopted from Analytica’s Toxic Emission Control) SUMMARY PROBLEM: Determine costs and benefits of containing an airborn toxin that is potentially fatal. Objective: Maximize the expected benefit, defined as benefits(1) less the cost(2) to contain toxin. (1) Cost to contain toxins depends on the containment level (logarithmic) (2) Benefits as the reduced mortality multiplied by the value of a life


Slide41: Problem: how much to contain and eliminate certain toxins including the option of reducing them by zero.


Security Categories: Security Categories Access Controls, Authentication Anti-eavesdropping Anti Virus Virus protection/detection Automated Patch Management Biometrics Authentication of users/terminals Business Continuity & Disaster Recovery Content Delivery Network Security  Email spam filters  Encryption Extranet Security Integration   Firewalls and Internet Security Intrusion Detection & Network Monitoring Media Security Destruction Devices Media Protection Safes Media Security Physical/Facility Security - Anti-Theft Devices Physical/Facility Security -Entrance Control Systems Physical/Facility Security - Environmental Controls Physical/Facility Security - Power Management Risk Management Risk Analysis Security Incident Management  Single Sign On Software Controls Telecom & Remote Access Security Wireless Security 


Where to find statistics: Where to find statistics 1. Symantec Internet Security Threat Report Volume IV - Every six months. During the first half of 2003, Symantec saw a 50% increase in confidential data attacks using backdoors. In the past six months, Web application vulnerabilities increased 12 percent, malicious codes were up 20 percent, and worms and viruses increased 19 percent 2. Computer Security Institute/ FBI Computer Crime and Security Survey - Yearly 3. @Stake Advisories and Research Labs (see Table below)


Tools: Tools Traditional Decision Analysis and/or Influence Diagrams: Analytica, DATA, Decide, DecisionPro, DPL, Expression Tree, Precision Tree, Risk Detective, Supertree/Sensitivity, TreePlan. Risk management tools: Analytica, DLP,LHS, Fuldek,SAPHIRE, SETS, SANET, SABLE, FTAP,  SEATree, Stepwise,


The End: The End "[T]he U.S. Air Force …is faced with a multitude of decisions- programmatic, technical, personnel, strategic, and yes, cultural - that we must make based on knowledge of, and respect for, the relevant underlying data. In that spirit …operations research and decision analysis are and will continue to contribute to national security decision-making."- Secretary of the Air Force James G. Roche, OR/MS December 2002