Ray Flores Roadmap

USAF T&E DAYSTRANSFORMING THE T&E ENTERPRISEVISION FOR T&E PANEL7 December 2005: 

Air Force Materiel Command I n t e g r i t y - S e r v i c e - E x c e l l e n c e Developing, Fielding, and Sustaining America’s Aerospace Force USAF T&E DAYS TRANSFORMING THE T&E ENTERPRISE VISION FOR T&E PANEL 7 December 2005

Slide2: 

T&E ENTERPRISE VISION Goals - Develop and Enable our People Operate Quality Installations Align the T&E Enterprise Identify, Develop and Transition New Test Capabilities Support Development, Fielding & Sustainment Change Strategy Document, streamline, and manage T&E test and business processes Institute T&E expeditionary mindset over weapon system life cycle Enable Integrated Test Teams with independent verification capability Develop Enterprise prioritization traceable to requirements Institutionalize feedback on T&E processes Vision Credible T&E Enterprise delivering quality information for risk management to our customers and stakeholders through clear lines of responsibility and accountability End State Effective T&E capabilities in place to enable delivery of decisive, integrated combat systems Integrated AF T&E Enterprise “To-Be” Independent AFMC Stovepipes Collaborative AF Centers/Wings Case for Action FY03 NDAA, DTRMC Budget Constraints Lack of credibility among all stakeholders Complexity and diversity of emerging weapon systems Need for system of systems testing and seamless verification Change Plan First Align the Enterprise Develop an integrated roadmap Define meaningful Balanced Scorecard measures Second Enable test & training in a joint environment Develop and resource an effective enterprise Build integrated DT/OT processes “As-Is” “Was” Objectives Develop and provide expertise to satisfy our customers Deliver properly sized and well maintained infrastructure Use modern business practices to integrate the Enterprise Implement capabilities to support new technologies and system-of-systems testing Increase T&E effectiveness to reduce acquisition program cycle-time, risk and life-cycle cost Many Minds, Many Capabilities . . . Single AF T&E Enterprise Mission The Air Force T&E Enterprise Delivers Effective Test and Evaluation Capabilities (People, Facilities, Processes & Equipment) to Support the Research, Development, Test, and Evaluation of Decisive Warfighting Capability

Roadmap Development Teams: 

Roadmap Development Teams Exec Leader Team

Roadmap Development Teams: 

Roadmap Development Teams Planning Team Exec Leader Team

Roadmap Development Teams: 

Roadmap Development Teams Planning Team Exec Leader Team Execution Team

Challenge Summary: 

Challenge Summary Funding is Insufficient to Meet Current Requirements Funding Does Not Allow for Preparing for Future Requirements Need to Work With AF to Either Grow the Funding “pie” or Reduce the Requirements “pie” AF T&E Roadmap provides a process for communicating impacts of decisions

Roadmap Objectives: 

Roadmap Objectives Define AF T&E Requirements and Capabilities to Effectively and Efficiently Meet Current and Future Warfighter Requirements Articulate an Envisioned End State and Goals and Objectives to Meet that Defined End State Establish a Deliberate Process for Prioritizing Resources – Across the AF T&E Enterprise Define a Strategy for Balancing Workload With Resources in The POM to Enable Execution of Roadmap

The Roadmap Process: 

The Roadmap Process Common Range Inst Air Combat Sea Combat Land Combat C4ISR Armaments/Munitions Test Environments Electronic Combat Space Combat Targets Derive T&E Needs Evaluate T&E Capabilities Capability Exists and Is Needed Capability Exists and not Needed Capability Does Not Exist and is Needed T&E Reliance Areas & Plans Provide & Sustain Capabilities Develop/Restore/ Improve/Modernize Capabilities & Techniques Excess Dispose Divest Requirements Sources CRRA CONOPS Mothball

Roadmap Description: 

Roadmap Description Living Document Top-level document framing big picture Fully integrated (USAF infrastructure and process requirements) Updated annually (must use a repeatable process) Used to manage funds (POM build, advocate funding, execution planning) Identifies Future Challenges and Critical Enabling Areas Rolls Up to Provide Inputs to DTRMC Strategic Planning WG Rolls Up to Provide Inputs to AF Roadmaps

Roadmap Spiral Development: 

Roadmap Spiral Development Spiral 1 Influence FY07 APOM Assess Sustainment of Current Capability Focused on Infrastructure Spiral 2 Impact FY08 POM Air Force T&E Roadmap Document Includes AFMC and AFOTEC Emphasis on Capabilities vice Infrastructure Spiral 3 Supports FY09 APOM Continue Maturity; Includes CAF & MAF Spiral 4 Supports FY10 POM; Incorporates All AF T&E Requirements Spiral 5 Institutionalize Process; Annual Updates

AF T&E Roadmap Schedule: 

AF T&E Roadmap Schedule As Of: 7 Dec 05 Spiral 3 Working Group Kick-off Draft I of Spiral I Release I of Spiral I Spiral 1 Final Release of Spiral I Working Group Kick-off Draft I of Spiral II Release I of Spiral II Final Release of Spiral II Program Concept Dev WG Body/Append Build Spiral 2 ELT On-Sites GO Briefs Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Feb Mar Apr May Jun Jul Aug Sep Jan 2005 2006 T&E Days 2007 2008 2009 2010 Spiral 4 Spiral 5 FY09 APOM FY10 POM & SPACE Institutional Process

Spiral 2 DirectionRoadmap Initiatives: 

Spiral 2 Direction Roadmap Initiatives Maintain and Improve Required Capability 1.1 Integrate Systems of Systems Testing 1.2 New System Technologies 1.3 New Test Methodologies 1.4 Modern Business Practices 1.5 Test Support Transformational Support Warfighter Support Weapons Sys Perf & Survivability Weapons Effects Directed Energy High Speed & Hypersonics Network Centric Warfare Test Fleet Support Test Processes Business Processes POM 08 Capability Gaps ADVOCACY INV INITIATIVES WARFIGHTER FOCUS GAP ANALYSIS REQUIREMENT ANALYSIS TRANSFORMATIONAL ENABLERS

Roadmap Development Teams: 

Roadmap Development Teams Execution Team

1.1 Maintaining and Improving Current Capability: 

1.1 Maintaining and Improving Current Capability REQUIREMENT Spiral 1 Develop a Quantitative & Qualitative method to manage potential FY07 APOM budget cuts Create Data Tools for the Enterprise to understand - facility utilizations customer requirements DBA $ required to support each test capability capacity (pay, non-pay) Spiral 2 Develop a method that ties the T&E enterprise capabilities (REUs) to national program priorities (CRRA) Provide a diagrammed method to maintain, improve, or divest of all the various test capabilities across the enterprise

1.1 Maintaining and Improving Current Capability: 

EXISTING CAPABILITY Ground Test Wind Tunnels RF/EO Chambers Space Environment Chambers Environmental Facilities Etc. Flight Test F-16, F-15, etc. AAMRAM, Small Diameter Bomb Weapon to Platform Integration/Separation Testing Etc. Modeling and Simulation Threat Simulation Aerodynamic Models Etc. 1.1 Maintaining and Improving Current Capability

1.1 Maintaining and Improving Current Capability: 

CAPABILITY GAP Weapons Systems Performance and Survivability (WSPS) Weapons Effects (WE) Directed Energy (DE) High Speed & Hypersonics (HS&H) Network Centric Warfare (NCW) Test Fleet Support (TFS) GAP SOLUTION WSPS & WE have test capability, but new technologies needed DE a relatively new area needing significant T&E support HS&H basis for many future weapon systems NCW – taking advantage of the latest computer & information technology Test Fleet Support – Aging Aircraft 1.1 Maintaining and Improving Current Capability

Roadmap Development Teams: 

Roadmap Development Teams Execution Team

1.2 Systems of Systems: 

1.2 Systems of Systems REQUIREMENT Test weapons systems in realistic environments Increasingly, this means with joint and coalition forces Address how well SUT performs as it interoperates with weapons systems and operators Weapon system integration, weapon system to weapon system, and integration into the C2 system Systems of systems testing will require live, virtual, constructive (LVC) entities operating at multiple, geographically separate locations: a distributed environment

1.2 Systems of Systems: 

1.2 Systems of Systems EXISTING CAPABILITY Numerous distributed training capabilities and some test sites DMOC (Kirtland) Virtual Warfare Center (St. Louis) CEIF (Hanscom) TDR (Gunter) CAOC-X (Langley) CAOC-N (Nellis) JRAAC (Huntsville) Datalink Test Facility (Eglin) Multi-Service Distributed Event OSD is championing effort to build distributed environment Testing in a Joint Force Environment Roadmap – Nov 2004 Testing in a Joint Force Environment Implementation Plan – Fall 2005 Joint Test & Evaluation Methodology (JTEM) Joint Test will define best practices

1.2 Systems of Systems: 

1.2 Systems of Systems CAPABILITY GAP A persistent robust networking infrastructure to connect distributed LVC resources Systems engineering, DT&E, OT&E Strategic partnerships between DOT&E, AT&L and Air Force T&E communities with OUSD(P&R), JFCOM, Air Force MAJCOMs, Instrumentation Lack of AF OPR for distributed test GAP SOLUTION Conduct a comprehensive study of systems of systems test infrastructure requirements Follow approach employed by the DETEC Tri-Service Study

Roadmap Development Teams: 

Roadmap Development Teams Execution Team

1.3 New Systems Technology: 

1.3 New Systems Technology REQUIREMENT: T&E needs for Air Force CONOPS Capabilities beyond current ground and flight test capabilities for New System Technology Directed Energy Weapons High Speed/Hypersonic Systems Large Foot Print Weapons Early and persistent involvement of AF T&E Provide right test capabilities on time and cost Robust system level testing through System Design and Development Phase Effective Early Operational Testing through understanding capabilities and effects

1.3 New Systems Technology: 

1.3 New Systems Technology EXISTING CAPABILITIES: Directed Energy Weapons capabilities are limited with regard to power levels affecting all aspects of T&E for new systems, e.g. Survivable instrumentation Threat emulators and simulators Modeling and simulation validation for T&E High Speed/Hypersonics test capabilities are limited and do not represent true physics, e.g. Limited propulsion test capability run time at higher Mach numbers Mach Number range inadequate for propulsion Limited run-time for aerothermodynamic material assessment Modeling and simulation validation for T&E Large Foot Print Weapons require larger range

1.3 New Systems Technology: 

1.3 New Systems Technology CAPABILITY GAPS: Existing test capabilities cannot meet the needs for new system technology without significant investment. A few examples are: Ground Test Cannot adequately simulate operating environment for high speed/hypersonic systems Instrumentation systems for Directed Energy Weapons Integration of Modeling and Simulation Flight Test Large footprint weapons range requirements Directed Energy weapons range requirements need to be addressed Communication, telemetry and data acquisition systems band width and range GAP SOLUTIONS: Complete an analysis of alternatives including risk management strategies for test capabilities (people, processes, and resources) for new systems

Roadmap Development Teams: 

Roadmap Development Teams Execution Team

1.4 New Test Methodologies: 

1.4 New Test Methodologies REQUIREMENT Test Environment must be relevant to Warfighter Operations Weapons Systems are increasingly complex Weapons Systems are being designed, developed and deployed in an integrated operational environment Mobility Integrated Live-Virtual-Constructive test methods Seamless Information Transfer and Access Knowledge Management Integrated Test Teams Methods for determination of what to keep and what to divest

1.4 New Test Methodologies: 

EXISTING CAPABILITIES & GAPS Several Fixed ‘distributed’ Test Sites AEDC AFFTC AAC (46TW) Limited L-V-C Integration Limited Information Portability Limited Test Team Integration Limited Mobility Limited Investment/Retention/Divestiture Capabilities 1.4 New Test Methodologies

1.4 New Test Methodologies: 

GAP SOLUTIONS (Initiatives) Integration of Test Capabilities, where practical L-V-C Integration Integrated DT/OT/OPS Knowledge/Information Sharing Integrated Test Teams (ITT) (ACQ) Expeditionary Test Teams (OPS) Validated/Aligned/Managed Test Tools Design of Experiments Data Management – Knowledge Retention & … Vision Driven Capabilities Planning (Investment, Retention & Divestiture) 1.4 New Test Methodologies

Roadmap Development Teams: 

Roadmap Development Teams Execution Team

Modern Business Practices“As-Is”: 

Modern Business Practices “As-Is” “Enterprise” Consists of 3 Test Organizations Plus HQ/AFMC A3 Each Test Organization Has Separate: Financial/Management Processes Level of Detail/Type of Information varies Financial/Management Reporting Systems Capability to Respond to Ad Hoc Questions/Timeliness of Response Varies Definitions for the Same Thing Difficulties in Communication HQ AFMC/A3 has Limited Insight and Knowledge Into Requirements and Actual Utilization

Modern Business Practices“To-Be” – By 2011: 

Modern Business Practices “To-Be” – By 2011 Common Processes Document Work Account for Requirements and Resources Track Utilization Report Performance Common Software Tools Common Definitions Everyone Speaks the Same Language Enable Activity Based Costing/Management and Capacity Management National Defense Authorization Act of FY03 Drives the Requirement

Modern Business PracticesNear-Term Activities: 

Modern Business Practices Near-Term Activities Feasibility Study 4 Month Study by Outside Contractor Ability for T&E to Reach 2011 Goal ID Risks/Costs/Schedules Recommend a Resource Mix and Management Structure for Successful Completion Mature the Resource Earning Unit (REU) Concept Complete the Designed Operational Capability (DOC) like Statements for T&E Define the Status of Resources and Training Systems (SORTS) like Reporting Process for T&E Conduct Independent Manpower Studies to Document the Resource Requirements

Modern Business PracticesLonger-Term Activities: 

Modern Business Practices Longer-Term Activities Establish a Dedicated Enterprise Team Fully Vet the Requirements Review the Study Plan the Action Manage the Implementation Responsible to the Entire Enterprise Senior Leadership for Implementation

Spiral 2 DirectionRoadmap Initiatives: 

Spiral 2 Direction Roadmap Initiatives Maintain and Improve Required Capability 1.1 Integrate Systems of Systems Testing 1.2 New System Technologies 1.3 New Test Methodologies 1.4 Modern Business Practices 1.5 Test Support Transformational Support Warfighter Support Weapons Sys Perf & Survivability Weapons Effects Directed Energy High Speed & Hypersonics Network Centric Warfare Test Fleet Support Test Processes Business Processes POM 08 Capability Gaps ADVOCACY INV INITIATIVES WARFIGHTER FOCUS GAP ANALYSIS REQUIREMENT ANALYSIS TRANSFORMATIONAL ENABLERS

AF T&E Roadmap Document: 

AF T&E Roadmap Document Executive Summary Introduction Vision, Mission, Goals, and Objectives T&E Capability Drivers T&E Capability Gaps Solution Strategy Future Strategic Planning Activities Summary/Recommendations

Spiral 2 DirectionTransformational Initiatives: 

Spiral 2 Direction Transformational Initiatives Test Processes Knowledge Management (DT- OT Data Collection) Expeditionary Test Teams T&E CONOPS for M&S Design of Experiments Vision Driven Planning Business Processes

Slide37: 

Test Processes & Knowledge Management Capability Objective Vision Driven Planning Expeditionary Test teams Integrated DT/OT Knowledge/ Information Sharing Validated/Aligned/Managed M&S Tools Knowledge Management/ Communities of Interest Design of Experiments Benefit Decreased Cost & Cycle Time Decreased Rework Increased Confidence in T&E Enterprise by Customers Independent Government Evaluation

Spiral 2 DirectionEnabling Initiatives: 

Spiral 2 Direction Enabling Initiatives Weapons Systems Performance & Survivability Weapons Effects Directed Energy Network Centric Warfare High Speed/ Hypersonic Test Support Fleet

Slide39: 

Capability Objective Operationally Representative Environment to Conduct NCW T&E Adequate Integration, Instrumentation & Data Acquisition Adequate Technical Expertise Benefit Sufficient Support of NCW Operational Development Reduced Risk (equip, cost, program, …) Training Capabilities for NCW Tactics Development & Operational Capabilities Network Centric Warfare Validation Study Requirements Seed $$ FY06-07

T&E Unconstrained Investment Support Requirement: 

T&E Unconstrained Investment Support Requirement

Initiative Funding Summary: 

Initiative Funding Summary

08 POM Initiative Funding Summary: 

08 POM Initiative Funding Summary High Speed Hypersonics Requirements Removed

Q&AMorning BREAKPanel Continues…: 

Q&A Morning BREAK Panel Continues…

RDT&EEffectiveness: 

Air Force Materiel Command I n t e g r i t y - S e r v i c e - E x c e l l e n c e Developing, Fielding, and Sustaining America’s Aerospace Force Dr. Ed Kraft AEDC/CA December 7, 2005 RDT&E Effectiveness

Efficiency vs Effectiveness: 

Efficiency vs Effectiveness Contractor System T&E Cost (Average $795.6M in FY 2001 $)* *Fox et al “Test and Evaluation Costs for Aircraft and Guided Weapons” Rand Project Air Force Report., 2004 + GAO-05-301, Assessment of Selected Major Weapon Programs, March 2005 GAO Program Reviews+

RDT&E Value HierarchyWhat you see depends on where you sit : 

RDT&E Value Hierarchy What you see depends on where you sit I II III IV V PLAN INSTALL TEST REMOVE SHIP Wind Tunnel Facility Test Program DESIGN BUILD TEST ANALYZE REPORT Wind Tunnel Test Program Air Vehicle Development Program Loads Prop Int Stores Perf S&C Wind Tunnel Flight CR TD SD&D P&D O&S A B C IOC FOC Acquisition Program C4ISR Avionics Stores Propulsion Airframe System Integration Focus Acquisition Cycle Platform Integration Subsystem Development Functional Discipline Test Facility Process

Aeronautical RDT&E MOPs, MOEs: 

Aeronautical RDT&E MOPs, MOEs Level Measures of Performance Measures of Effectiveness V IV III II I Required capability delivered within cost and schedule Weapon systemMOPs/ MOEs met or exceeded Subsystem integration issues eliminated before 1st flight Pass IOT&E on first attempt Interoperability issues resolved before IOT&E Elimination of subsystem defects before 1st flight Elimination of subsystem integration issues before 1st flight Vehicle MOPs met Accurate flight predictions Discovery of design flaws in first pass Accurate simulation of key phenomena – no surprises in flight Discovery of design flaws in first pass Insight into design refinements required to meet vehicle MOPs SDD$/Total $ T&E $/SDD $ # and severity of quality escapes CIP $/SDD $ SDD Time/Acquisition Time Total Wind Tunnel hours Total Flight Test hours # design defects discovered Time and cost to resolve defects Relative impact on LCC of defects discovered # design defects discovered Time and cost to resolve defects Relative impact on LCC of defects discovered Total Wind Tunnel hours Total Flight Test hours # design defects discovered Time and cost to resolve defects Relative impact on LCC of defects discovered Data accuracy (bias and precision) $/data point $/UOH Polars/hour Vehicle mission coverage

Summary: 

Summary Major Gains to be Made by Emphasizing Effectiveness, Not Just Efficiency Increasing Effectiveness Drives Several Key Aspects of the AF T&E Roadmap Vision Institute T&E expeditionary mindset over weapon system life cycle Enable Integrated Test Teams with independent verification capability Institutionalize feedback on T&E processes Build Integrated DT/OT processes …And We Haven’t Even Talked about System of Systems and Interoperability Testing Trained People and Improved Processes Are More Important to Effectiveness than Facilities

Vision for Air ForceTest & Evaluation: 

Vision for Air Force Test & Evaluation Or how we all survived without killing each other… George Kailiwai III Technical Advisor, Air Force Flight Test Center

Vision for Air Force Test & EvaluationAir Force Flight Test Center : 

Vision for Air Force Test & Evaluation Air Force Flight Test Center We’re in this all together. A “family” can overcome any adversity. Family members may not always agree, but in the end, family members generally make decisions that are best for the family. Collaboration & teaming to date has been “UNPRECEDENTED” Collaboration & teaming will be crucial to future success Thank AFMC/A3 for leading the effort and AEDC, 46th TW, AFOTEC and AF/TE for having the courage to take the first steps….

46TW PerspectiveMr. Bob Arnold: 

46TW Perspective Mr. Bob Arnold

AFOTEC PerspectiveCol. Joe Arvai for Mr. Doug Marlowe: 

AFOTEC Perspective Col. Joe Arvai for Mr. Doug Marlowe

AF/TE Perspective Mr. Wally Beard for Mr. Dave Hamilton: 

AF/TE Perspective Mr. Wally Beard for Mr. Dave Hamilton

Panel DiscussionModerator: Mr. Dave Bond: 

Panel Discussion Moderator: Mr. Dave Bond