Slide1: Information Extraction and Transport, Inc. under the direction of Murray Burke, DARPA RKF CY 2002 BS CP Planning 27 February, 2002


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans IPB Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization CP Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


Domain Overview: Domain Overview Intelligence Preparation of the Battlefield (IPB) Purpose: Provide actionable information for battlefield commanders. Method: Define the battlefield environment. Describe the battlefield’s effects. Evaluate the threat. Determine threat courses of action.


IPB Method—Step One: IPB Method—Step One Define the battlefield environment. Done in terms of Area of Operations (AO), Area of Interest (AI) and Battle Space (BS). Identifies the significant characteristics of the environment; pulls in relevant data from available databases. Uses METT-T principles: Mission (the objective) Enemy (the threat—Red Force) Terrain (land classes, attributes, elevation/slope) Troops (units and equipment available to Blue Force commander) Time Available (any time constraint imposed) IET will provide the above information as relevant to the CP.


IPB Method—Step Two: IPB Method—Step Two Describe the battlefield’s effects. Terrain analysis using OCOKA principles: Observation and Fields of Fire Cover and Concealment Obstacles Key Terrain Avenues of Approach Weather analysis We will focus on terrain in CY 2002, providing data for elevation, vegetation, slope, soil, etc. Weather analysis is a future goal.


IPB Method—Step Three: IPB Method—Step Three Evaluate the threat. Utilizes doctrinal templates depicting how the enemy would like to fight if not restricted by terrain. Identifies the High Value Targets (HVTs) in the eyes of the enemy. May consider unconventional templates such as religious overlays, ethnic overlays, and personality diagrams. Identifies threat capabilities—what can the enemy do to us? IET will provide the above information as relevant to the CP.


IPB Method—Step Four: IPB Method—Step Four Determine Enemy COAs (ECOAs). IDs threat objectives and desired end state. Produces situation templates, taking into consideration environmental factors. These depict how the enemy might deploy and operate within environmental constraints. SitTemps provide a snapshot at one moment in time of where the enemy might be—one SitTemp for each major action. Produces event templates depicting locations where critical events and activities are expected to occur. Event templates align several situation templates to show progression in time. Based on templates, commander prepares a COA in response to the ECOA. We want to capture IPB analyst SME knowledge needed to construct situation and event templates.


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


IPB Needs: IPB Needs Most AI-based IPB tool development to date has focused overly much on the knowledge contained in field manuals. Field manuals present general battlefield doctrine but leave aside expert knowledge and situational information. KR has traditionally been done by KEs. Creates a KR bottleneck Misses SME knowledge/reasoning RKF tools can break the bottleneck. Knowledge bases can capture SME knowledge/reasoning.


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


IPB Situational Reasoning: IPB Situational Reasoning In moving away from doctrine, we begin to focus on actual mission situations. OCOKA and METT-T principles help guide analysis but are no longer abstract considerations. Situational reasoning begins to place constraints on actions and locations (e.g., perhaps not all of the terrain will be traversable by Red’s equipment) It also moves beyond doctrinal guidelines and abstract data-based threat information.


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


IPB Doctrinal vs. Functional Knowledge: IPB Doctrinal vs. Functional Knowledge Doctrinal knowledge is captured in field manuals, general battle field principles, generic guidance. It supports situational reasoning but is not itself situation specific. IPB analyst SME knowledge used in situational specialization of doctrine can be characterized as “functional knowledge”.


IPB Doctrinal Knowledge: IPB Doctrinal Knowledge Captures theories of military actions in general terms—without reference to explicit situations except in idealized terms. Emphasizes general rules without dealing heavily with exceptions. Terrain, weather, troop fatigue, equipment loss, and similar realities of battle space and engagement are treated conceptually. Places a huge burden of situational analysis on the shoulders of soldiers in the field.


IPB Doctrinal Knowledge Examples: IPB Doctrinal Knowledge Examples Characteristics of operational activities (e.g., average speeds for marches, typical formations) Conventional battle space wisdom (e.g., “Base of fire moves forward as troops advance.”) Templates indicating typical unit compositions Standard equipment specs (e.g., factory-specified performance for weapons, vehicles) General principles/desirable invariant properties (e.g., own-force security, concealment; OPFOR degradation, mission defeat) OCOKA principles of terrain suitability analysis METT-T principles of tactical factor consideration


IPB Functional Knowledge: IPB Functional Knowledge Provides the practical “how” to doctrinal knowledge’s theoretical “what.” Supports the application of doctrinal knowledge in a given battle space. Including specific mission, terrain, weather, and own/opposition-force deployment. Draws on doctrinal knowledge as the theoretical underpinning from which to proceed. Allows analysts to make sense of exceptions to doctrinal guidance. Encoding the knowledge that enables human analysts to adapt doctrinal knowledge to a battlefield situation is the main target of the BS CP.


IPB Functional Knowledge Examples (1): IPB Functional Knowledge Examples (1) Examples of exceptions requiring functional knowledge: Situation- (e.g., terrain-) influenced atypical formations, activities, tactics Non-standard unit compositions Degraded unit conditions (e.g., fatigue, demoralization, undersupply) Non-standard/non-working equipment Non-standard/novel applications of units or equipment. Situation-specific exceptions to conventional wisdom (e.g., a commander may choose not to move the base of fire forward with troops when terrain precludes this) Situation-specific variations on (priorities among) general principles/desirable invariant properties, including mission factors


IPB Functional Knowledge Example (2): IPB Functional Knowledge Example (2) Consider moving a tank unit across some terrain. Doctrinal knowledge will specify a formation and movement pattern for minimizing overall/aggregate target profile for the unit if moving across flat, trafficable terrain. Functional knowledge will enumerate and assess different candidate formations and movement patterns given the actual terrain/environment in which the unit is operating.


IPB Functional Knowledge Example (3): IPB Functional Knowledge Example (3) Functional knowledge considerations will include: Terrain Unit characteristics Time constraints Terrain features vs. equipment specs (e.g., tank width may block access to forested areas) Common-sense knowledge (e.g., a deciduous forest will not provide canopy cover in winter) Assessment knowledge to enable ranking of candidates And more…


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


CP Overview: CP Overview Initial spec (Version 0.1) released Feb 25. Authoring tasks described in phases. CP focuses on IPB-related problem solving tasks notionally culminating in situation differencing. IET will be preparing several products in support of the CP. A Source Packet (SP) with pointers to doctrinal materials Situation Descriptions (SDs) with instances of battlefield situations (units, missions, terrain, etc.)


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


Authoring Phases: Authoring Phases Phase 0: pump-priming and interface development Phase 1: SME authoring of generic knowledge Regarding terrain, doctrinal templates, units, equipment, capabilities, etc. Phase 2: SME authoring of instance-level knowledge Guided by provided SDs Phase 3: SME authoring of functional knowledge rules and constraints Related to situation differencing and IPB problem solving tasks SMEs and Integrators can interact during all phases.


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


Source Packet: Source Packet IET will develop a source packet (SP) containing materials (with pointers to sections of interest) covering the following topics: Unit specifications personnel, vehicles, equipment—from table of organization and equipment—TOE Equipment specifications E.g., firing range, maximum speed Doctrinal information, such as: Templates pertaining to stereotypical tactics, techniques, and procedures (TTP) Templates pertaining to terrain/unit/equipment interactions E.g., equipment placement guidance OCOKA principles Terrain descriptions/schemas E.g., land classes, vegetation, attributes Initial release: April 1, 2002


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


Situation Descriptions (1): Situation Descriptions (1) SDs will include: Terrain maps with overlays: Elevation, slope, combined obstacle overlay (COO), complex factor overlay (CFO), vegetation, water features, unit placement, etc. Unit descriptions Type of unit, number of men/vehicles Mission descriptions E.g., “The SCUD unit’s mission is to shoot, hide, reload, and redeploy under the following constraints…” Past observed battlefield behavior / preferences (if known) Initial release: April 1, 2002


Situation Descriptions (2): Situation Descriptions (2) Equipment descriptions OPFOR-type data and suspected capabilities not in doctrinal descriptions Identified entities of interest E.g., targets Document-based functional knowledge materials developed by evaluator-collaborating SMEs. Ideally teams would themselves have direct access to terrain-savvy IPB analyst SMEs during Phase 0 efforts. Initial release: April 1, 2002


Sample Situation Description: Sample Situation Description Task: Red Unit destroys Blue target Method: Red Unit deploys MAZ-543 SCUD missile launcher, fires on Blue target, hides, reloads. Terrain: Shown on map. Additional info: Overlays depicting slope, soil, vegetation, etc. Location of reload site. Timeline as needed. Relevant information in Source Packet: Generic SCUD unit mission profile (Fire, Hide, Reload, Re-deploy) Equipment specs for MAZ-543


Situation Descriptions (4): Situation Descriptions (4) Equipment spec for MAZ-543


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


Problem-solving Task Characterization: Problem-solving Task Characterization Situation differencing Notional / ideal CY 2002 goal Question- / PQ-framed tasks Progression towards above goal Range of difficulty


Situation Differencing: Situation Differencing Present SMEs with pairs of similar battlefield situations (with one plausibly temporally downstream of the other). Ask them to create / augment the procedural and / or behavioral knowledge necessary to understand the transformation from initial to final situation. The product of this task will be an action sequence to effect the transformation. We take this to be a reasonable step on the path to future COA generation.


Question-framedProblem-solving Tasks: Question-framed Problem-solving Tasks Given an entity and its mission, rate a given location with respect to its suitability for performing an identified mission task with respect to a stated tactical consideration such as concealment, speed, visibility, etc. Compare two possible routes through specified terrain for a given unit with a specified mission and rate them according to various tactical considerations or combinations of such considerations. Given the terrain, unit, mission, origin and desired destination, identify the best routes according to various tactical considerations.


Question-framedProblem-solving Tasks (2): Question-framed Problem-solving Tasks (2) Given the terrain, identify likely locations for various specified activities/entities (such as missile launches, ambushes, location of base of fire, etc.). Given a high-level mission description (such as a SCUD launcher unit’s mission to shoot, hide, reload, and begin again) and terrain, identify the best locations for achieving the mission in its entirety. Tactical considerations could be specified here as well.


PQ-framedProblem-solving Tasks: PQ-framed Problem-solving Tasks PQ versions of foregoing questions: Rate w.r.t. . Compare , w.r.t. . What are good w.r.t. ? What is the best w.r.t. ?


Supporting PQ Pseudo-classes: Supporting PQ Pseudo-classes = {, ; Concealment, security, etc. ; OCOKACriterion, ; METT-TCriterion , ; A combination of criteria, logical, or with arithmetic weighting } ; Supporting situation differencing = {, ; Qualitative sequence/graph of static SDs , ; Action/task, action sequence/COA , } ; (Qualitative) , , ;


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


Pump Priming Issues (1): Pump Priming Issues (1) What are we going to pump prime? Terrain data Choose a region and stick with it? Which? Equipment types and specs Choose a few subtypes? Which? Missions How many mission types? Which? At what level do we want to describe missions? What sorts of constraints do we want to consider? How many? Doctrinal knowledge for SD representation Pump primed by teams? With IET’s assistance?


Pump Priming Issues (2): Pump Priming Issues (2) Considerations for SD-supporting KR and pump priming in general: We can construct SDs by varying some subset of {terrain, unit/equipment, mission} and holding other elements constant. The varied element will require more pump priming.


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


Problem-solving Task Issues (1): Problem-solving Task Issues (1) Static-only vs. static-and-dynamic SDs To support situation differencing, we need both types Dynamic SDs up the complexity of the problem. Can teams support dynamic SDs? Rating-only vs. enumeration-only vs. rating-and-enumeration tasks Situation differencing requires enumeration. Constraints could be formulated in a “hard”/satisfying (vice “soft”/satisficing) way. Quantitative reasoning would not be required. Ratings could be binary ratings (“go/no-go”).


Problem-solving Task Issues (2): Problem-solving Task Issues (2) Invariant principles range How many tactical considerations do we want to consider? E.g., Concealment, security, speed, morale, supply, etc. Address all of OCOKA and/or METT-T principles? We can use our choices regarding SD generation to restrict focus here.


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


SME Access: SME Access CY 2002 SMEs are a scarce resource. Propose to hold at least two dry runs (of one or two weeks’ duration each). Propose a two to four week E2E evaluation. Plan on at least one SME per team during dry runs; at least two SMEs per team during E2E evaluation. Teams are encouraged to employ their own SMEs to have continuous, ready feedback.


Mixed-skills authoring setting: Mixed-skills authoring setting How should we monitor SMEs and IKEs interactions? Track GKE- or IKE-initiated additions and deletions? IET-monitored email channels? Scheduled SME-IKE interactions? Annotate, in the KB, the axioms/constants authored w.r.t. nature or degree of collaboration? What should be the SME/IKE make-up per team? More than one mixed-skills team/integration team? IKEs participating in more than one mixed-skills team (with more than one SME)? SMEs collaborating? (Probably not, for the same reasons as concluded in CY 2001.)


Evaluation Model: Evaluation Model Quantitative evaluation Diagnostic questions Covering Phase 1 and 2 material Problem solving questions Covering Phase 3 material Metrics Qualitative evaluation KB review by KR experts and SMEs


CY 2002 CP Agenda: CY 2002 CP Agenda CY 2002 CP plans Domain Overview Needs Situational reasoning Doctrinal vs. functional knowledge CP Overview Phases Source packet Situation descriptions Problem-solving task characterization Issues Pump-priming Problem-solving tasks Evaluation procedures Schedule


CY 2002 BS CP Schedule: CY 2002 BS CP Schedule


Extras: Extras


Changes for CY 2002 Evaluation: Changes for CY 2002 Evaluation Mixed-skill / role KB authoring teams Natural SME / OE ( / tool builder) interaction No gatekeeper (more suitable, alternative interaction tracking) Move from TK emphasis to EK emphasis. More evaluation points: Dry-runs all summer (June through August) E2E evaluation late August or early September Focus effort: Fewer SMEs, fewer TQs, shorter evaluations (but more often)


Major Conclusions about Y1 Evaluation: Major Conclusions about Y1 Evaluation “Lone SME” KB authoring ineffective Predicate, rule authoring too difficult for SMEs / tool support Pump priming focus off-target SMEs burned out Tool ennui ECB textbook ennui Overall length


Tiered KB Authoring Society: Requirements Tiered KB Authoring Society Subject Matter Expert (SME) Knowledge Engineer (KE) Curator—IET, Veridian Advice Inference Desiger (ID) Cycorp, SRI Content Methods


Mixed-skills Authoring Team: Mixed-skills Authoring Team Continue to exploit RKF’s Y1 SME-appropriate KB authoring tools. And develop more / better tools Maximally exploit differing strengths. SMEs characterize rough predicate needs better / faster than OEs could. OEs draft predicates better / faster than SMEs could.