USAF T&E DAYS TRANSFORMING THE T&E ENTERPRISE VISION FOR T&E PANEL 7 December 2005 : Air Force Materiel Command I n t e g r i t y - S e r v i c e - E x c e l l e n c e Developing, Fielding, and Sustaining America’s Aerospace Force USAF T&E DAYS TRANSFORMING THE T&E ENTERPRISE VISION FOR T&E PANEL 7 December 2005
Slide2: T&E ENTERPRISE VISION Goals
- Develop and Enable our People
Operate Quality Installations
Align the T&E Enterprise
Identify, Develop and Transition New Test Capabilities
Support Development, Fielding & Sustainment
Change Strategy
Document, streamline, and manage T&E test and business processes
Institute T&E expeditionary mindset over weapon system life cycle
Enable Integrated Test Teams with independent verification capability
Develop Enterprise prioritization traceable to requirements
Institutionalize feedback on T&E processes
Vision
Credible T&E Enterprise delivering quality information for risk management to our customers and stakeholders through clear lines of responsibility and accountability End State
Effective T&E capabilities in place to enable delivery of decisive, integrated combat systems Integrated AF
T&E Enterprise “To-Be” Independent
AFMC
Stovepipes Collaborative
AF Centers/Wings Case for Action
FY03 NDAA, DTRMC
Budget Constraints
Lack of credibility among all stakeholders
Complexity and diversity of emerging weapon systems
Need for system of systems testing and seamless verification Change Plan
First
Align the Enterprise
Develop an integrated roadmap
Define meaningful Balanced Scorecard measures
Second
Enable test & training in a joint environment
Develop and resource an effective enterprise
Build integrated DT/OT processes
“As-Is” “Was” Objectives
Develop and provide expertise to satisfy our customers
Deliver properly sized and well maintained infrastructure
Use modern business practices to integrate the Enterprise
Implement capabilities to support new technologies and system-of-systems testing
Increase T&E effectiveness to reduce acquisition program cycle-time, risk and life-cycle cost Many Minds, Many Capabilities . . . Single AF T&E Enterprise Mission
The Air Force T&E Enterprise Delivers Effective Test and Evaluation Capabilities (People, Facilities, Processes
& Equipment) to Support the Research, Development, Test, and Evaluation of Decisive Warfighting Capability
Roadmap Development Teams: Roadmap Development Teams Exec Leader Team
Roadmap Development Teams: Roadmap Development Teams Planning Team Exec Leader Team
Roadmap Development Teams: Roadmap Development Teams Planning Team Exec Leader Team Execution Team
Challenge Summary: Challenge Summary Funding is Insufficient to Meet Current Requirements
Funding Does Not Allow for Preparing for Future Requirements
Need to Work With AF to Either Grow the Funding “pie” or Reduce the Requirements “pie” AF T&E Roadmap provides a process for communicating impacts of decisions
Roadmap Objectives: Roadmap Objectives Define AF T&E Requirements and Capabilities to Effectively and Efficiently Meet Current and Future Warfighter Requirements
Articulate an Envisioned End State and Goals and Objectives to Meet that Defined End State
Establish a Deliberate Process for Prioritizing Resources – Across the AF T&E Enterprise
Define a Strategy for Balancing Workload With Resources in The POM to Enable Execution of Roadmap
The Roadmap Process: The Roadmap Process
Common Range Inst
Air Combat
Sea Combat
Land Combat
C4ISR
Armaments/Munitions
Test Environments
Electronic Combat
Space Combat
Targets
Derive
T&E
Needs Evaluate
T&E
Capabilities Capability Exists
and Is Needed Capability Exists
and not Needed Capability Does
Not Exist
and is Needed T&E Reliance
Areas & Plans Provide & Sustain
Capabilities Develop/Restore/
Improve/Modernize
Capabilities & Techniques Excess
Dispose Divest Requirements
Sources
CRRA
CONOPS Mothball
Roadmap Description: Roadmap Description Living Document
Top-level document framing big picture
Fully integrated (USAF infrastructure and process requirements)
Updated annually (must use a repeatable process)
Used to manage funds (POM build, advocate funding, execution planning)
Identifies Future Challenges and Critical Enabling Areas
Rolls Up to Provide Inputs to DTRMC Strategic Planning WG
Rolls Up to Provide Inputs to AF Roadmaps
Roadmap Spiral Development: Roadmap Spiral Development Spiral 1
Influence FY07 APOM
Assess Sustainment of Current Capability
Focused on Infrastructure
Spiral 2
Impact FY08 POM
Air Force T&E Roadmap Document
Includes AFMC and AFOTEC
Emphasis on Capabilities vice Infrastructure
Spiral 3
Supports FY09 APOM
Continue Maturity; Includes CAF & MAF
Spiral 4
Supports FY10 POM; Incorporates All AF T&E Requirements
Spiral 5
Institutionalize Process; Annual Updates
AF T&E Roadmap Schedule: AF T&E Roadmap Schedule As Of: 7 Dec 05 Spiral 3 Working Group Kick-off Draft I of Spiral I Release I of Spiral I Spiral 1 Final Release of Spiral I Working Group Kick-off Draft I of Spiral II Release I of Spiral II Final Release of Spiral II Program Concept Dev WG Body/Append Build Spiral 2 ELT On-Sites GO Briefs Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Feb Mar Apr May Jun Jul Aug Sep Jan 2005 2006 T&E Days 2007 2008 2009 2010 Spiral 4 Spiral 5 FY09 APOM FY10 POM
& SPACE Institutional Process
Spiral 2 Direction Roadmap Initiatives: Spiral 2 Direction Roadmap Initiatives Maintain and Improve Required Capability
1.1 Integrate Systems of Systems Testing
1.2 New System Technologies
1.3 New Test Methodologies
1.4 Modern Business Practices
1.5 Test Support Transformational Support Warfighter Support Weapons Sys Perf & Survivability Weapons Effects Directed Energy High Speed & Hypersonics Network Centric
Warfare Test Fleet Support Test
Processes Business Processes POM 08 Capability Gaps ADVOCACY INV INITIATIVES WARFIGHTER
FOCUS GAP ANALYSIS REQUIREMENT
ANALYSIS TRANSFORMATIONAL ENABLERS
Roadmap Development Teams: Roadmap Development Teams Execution Team
1.1 Maintaining and Improving Current Capability: 1.1 Maintaining and Improving Current Capability REQUIREMENT
Spiral 1
Develop a Quantitative & Qualitative method to manage potential FY07 APOM budget cuts
Create Data Tools for the Enterprise to understand -
facility utilizations
customer requirements
DBA $ required to support each test capability capacity (pay, non-pay)
Spiral 2
Develop a method that ties the T&E enterprise capabilities (REUs) to national program priorities (CRRA)
Provide a diagrammed method to maintain, improve, or divest of all the various test capabilities across the enterprise
1.1 Maintaining and Improving Current Capability: EXISTING CAPABILITY
Ground Test
Wind Tunnels
RF/EO Chambers
Space Environment Chambers
Environmental Facilities
Etc.
Flight Test
F-16, F-15, etc.
AAMRAM, Small Diameter Bomb
Weapon to Platform Integration/Separation Testing
Etc.
Modeling and Simulation
Threat Simulation
Aerodynamic Models
Etc.
1.1 Maintaining and Improving Current Capability
1.1 Maintaining and Improving Current Capability: CAPABILITY GAP
Weapons Systems Performance and Survivability (WSPS)
Weapons Effects (WE)
Directed Energy (DE)
High Speed & Hypersonics (HS&H)
Network Centric Warfare (NCW)
Test Fleet Support (TFS)
GAP SOLUTION
WSPS & WE have test capability, but new technologies needed
DE a relatively new area needing significant T&E support
HS&H basis for many future weapon systems
NCW – taking advantage of the latest computer & information technology
Test Fleet Support – Aging Aircraft 1.1 Maintaining and Improving Current Capability
Roadmap Development Teams: Roadmap Development Teams Execution Team
1.2 Systems of Systems: 1.2 Systems of Systems REQUIREMENT
Test weapons systems in realistic environments
Increasingly, this means with joint and coalition forces
Address how well SUT performs as it interoperates with weapons systems and operators
Weapon system integration, weapon system to weapon system, and integration into the C2 system
Systems of systems testing will require live, virtual, constructive (LVC) entities operating at multiple, geographically separate locations: a distributed environment
1.2 Systems of Systems: 1.2 Systems of Systems EXISTING CAPABILITY
Numerous distributed training capabilities and some test sites
DMOC (Kirtland)
Virtual Warfare Center (St. Louis)
CEIF (Hanscom)
TDR (Gunter)
CAOC-X (Langley)
CAOC-N (Nellis)
JRAAC (Huntsville)
Datalink Test Facility (Eglin)
Multi-Service Distributed Event
OSD is championing effort to build distributed environment
Testing in a Joint Force Environment Roadmap – Nov 2004
Testing in a Joint Force Environment Implementation Plan – Fall 2005
Joint Test & Evaluation Methodology (JTEM) Joint Test will define best practices
1.2 Systems of Systems: 1.2 Systems of Systems CAPABILITY GAP
A persistent robust networking infrastructure to connect distributed LVC resources
Systems engineering, DT&E, OT&E
Strategic partnerships between DOT&E, AT&L and Air Force T&E communities with OUSD(P&R), JFCOM, Air Force MAJCOMs, Instrumentation
Lack of AF OPR for distributed test
GAP SOLUTION
Conduct a comprehensive study of systems of systems test infrastructure requirements
Follow approach employed by the DETEC Tri-Service Study
Roadmap Development Teams: Roadmap Development Teams Execution Team
1.3 New Systems Technology: 1.3 New Systems Technology REQUIREMENT:
T&E needs for Air Force CONOPS Capabilities beyond current ground and flight test capabilities for New System Technology
Directed Energy Weapons
High Speed/Hypersonic Systems
Large Foot Print Weapons
Early and persistent involvement of AF T&E
Provide right test capabilities on time and cost
Robust system level testing through System Design and Development Phase
Effective Early Operational Testing through understanding capabilities and effects
1.3 New Systems Technology: 1.3 New Systems Technology EXISTING CAPABILITIES:
Directed Energy Weapons capabilities are limited with regard to power levels affecting all aspects of T&E for new systems, e.g.
Survivable instrumentation
Threat emulators and simulators
Modeling and simulation validation for T&E
High Speed/Hypersonics test capabilities are limited and do not represent true physics, e.g.
Limited propulsion test capability run time at higher Mach numbers
Mach Number range inadequate for propulsion
Limited run-time for aerothermodynamic material assessment
Modeling and simulation validation for T&E
Large Foot Print Weapons require larger range
1.3 New Systems Technology: 1.3 New Systems Technology CAPABILITY GAPS:
Existing test capabilities cannot meet the needs for new system technology without significant investment. A few examples are:
Ground Test
Cannot adequately simulate operating environment for high speed/hypersonic systems
Instrumentation systems for Directed Energy Weapons
Integration of Modeling and Simulation
Flight Test
Large footprint weapons range requirements
Directed Energy weapons range requirements need to be addressed
Communication, telemetry and data acquisition systems band width and range
GAP SOLUTIONS:
Complete an analysis of alternatives including risk management strategies for test capabilities (people, processes, and resources) for new systems
Roadmap Development Teams: Roadmap Development Teams Execution Team
1.4 New Test Methodologies: 1.4 New Test Methodologies REQUIREMENT
Test Environment must be relevant to Warfighter Operations
Weapons Systems are increasingly complex
Weapons Systems are being designed, developed and deployed in an integrated operational environment
Mobility
Integrated Live-Virtual-Constructive test methods
Seamless Information Transfer and Access
Knowledge Management
Integrated Test Teams
Methods for determination of what to keep and what to divest
1.4 New Test Methodologies: EXISTING CAPABILITIES & GAPS
Several Fixed ‘distributed’ Test Sites
AEDC
AFFTC
AAC (46TW)
Limited L-V-C Integration
Limited Information Portability
Limited Test Team Integration
Limited Mobility
Limited Investment/Retention/Divestiture Capabilities 1.4 New Test Methodologies
1.4 New Test Methodologies: GAP SOLUTIONS (Initiatives)
Integration of Test Capabilities, where practical
L-V-C Integration
Integrated DT/OT/OPS Knowledge/Information Sharing
Integrated Test Teams (ITT) (ACQ)
Expeditionary Test Teams (OPS)
Validated/Aligned/Managed Test Tools
Design of Experiments
Data Management – Knowledge Retention & …
Vision Driven Capabilities Planning (Investment, Retention & Divestiture) 1.4 New Test Methodologies
Roadmap Development Teams: Roadmap Development Teams Execution Team
Modern Business Practices “As-Is”: Modern Business Practices “As-Is” “Enterprise” Consists of 3 Test Organizations Plus HQ/AFMC A3
Each Test Organization Has Separate:
Financial/Management Processes
Level of Detail/Type of Information varies
Financial/Management Reporting Systems
Capability to Respond to Ad Hoc Questions/Timeliness of Response Varies
Definitions for the Same Thing
Difficulties in Communication
HQ AFMC/A3 has Limited Insight and Knowledge Into Requirements and Actual Utilization
Modern Business Practices “To-Be” – By 2011: Modern Business Practices “To-Be” – By 2011 Common Processes
Document Work
Account for Requirements and Resources
Track Utilization
Report Performance
Common Software Tools
Common Definitions
Everyone Speaks the Same Language
Enable Activity Based Costing/Management and Capacity Management National Defense Authorization Act of FY03 Drives the Requirement
Modern Business Practices Near-Term Activities: Modern Business Practices Near-Term Activities Feasibility Study
4 Month Study by Outside Contractor
Ability for T&E to Reach 2011 Goal
ID Risks/Costs/Schedules
Recommend a Resource Mix and Management Structure for Successful Completion
Mature the Resource Earning Unit (REU) Concept
Complete the Designed Operational Capability (DOC) like Statements for T&E
Define the Status of Resources and Training Systems (SORTS) like Reporting Process for T&E
Conduct Independent Manpower Studies to Document the Resource Requirements
Modern Business Practices Longer-Term Activities: Modern Business Practices Longer-Term Activities Establish a Dedicated Enterprise Team
Fully Vet the Requirements
Review the Study
Plan the Action
Manage the Implementation
Responsible to the Entire Enterprise Senior Leadership for Implementation
Spiral 2 Direction Roadmap Initiatives: Spiral 2 Direction Roadmap Initiatives Maintain and Improve Required Capability
1.1 Integrate Systems of Systems Testing
1.2 New System Technologies
1.3 New Test Methodologies
1.4 Modern Business Practices
1.5 Test Support Transformational Support Warfighter Support Weapons Sys Perf & Survivability Weapons Effects Directed Energy High Speed & Hypersonics Network Centric
Warfare Test Fleet Support Test
Processes Business Processes POM 08 Capability Gaps ADVOCACY INV INITIATIVES WARFIGHTER
FOCUS GAP ANALYSIS REQUIREMENT
ANALYSIS TRANSFORMATIONAL ENABLERS
AF T&E Roadmap Document: AF T&E Roadmap Document Executive Summary
Introduction
Vision, Mission, Goals, and Objectives
T&E Capability Drivers
T&E Capability Gaps
Solution Strategy
Future Strategic Planning Activities
Summary/Recommendations
Spiral 2 Direction Transformational Initiatives: Spiral 2 Direction Transformational Initiatives
Test Processes
Knowledge Management (DT- OT Data Collection)
Expeditionary Test Teams
T&E CONOPS for M&S
Design of Experiments
Vision Driven Planning
Business Processes
Slide37: Test Processes & Knowledge Management Capability Objective
Vision Driven Planning
Expeditionary Test teams
Integrated DT/OT Knowledge/ Information Sharing
Validated/Aligned/Managed M&S Tools
Knowledge Management/ Communities of Interest
Design of Experiments Benefit
Decreased Cost & Cycle Time
Decreased Rework
Increased Confidence in T&E Enterprise by Customers
Independent Government Evaluation
Spiral 2 Direction Enabling Initiatives: Spiral 2 Direction Enabling Initiatives Weapons Systems Performance & Survivability
Weapons Effects
Directed Energy
Network Centric Warfare
High Speed/ Hypersonic
Test Support Fleet
Slide39: Capability Objective
Operationally Representative Environment to Conduct NCW T&E
Adequate Integration, Instrumentation & Data Acquisition
Adequate Technical Expertise
Benefit
Sufficient Support of NCW Operational Development
Reduced Risk (equip, cost, program, …)
Training Capabilities for NCW Tactics Development & Operational Capabilities
Network Centric Warfare Validation Study Requirements Seed $$ FY06-07
T&E Unconstrained Investment Support Requirement: T&E Unconstrained Investment Support Requirement
Initiative Funding Summary: Initiative Funding Summary
08 POM Initiative Funding Summary: 08 POM Initiative Funding Summary High Speed Hypersonics Requirements Removed
Q&A Morning BREAK Panel Continues…: Q&A Morning BREAK Panel Continues…
RDT&E Effectiveness: Air Force Materiel Command I n t e g r i t y - S e r v i c e - E x c e l l e n c e Developing, Fielding, and Sustaining America’s Aerospace Force Dr. Ed Kraft
AEDC/CA
December 7, 2005 RDT&E Effectiveness
Efficiency vs Effectiveness: Efficiency vs Effectiveness Contractor System T&E Cost
(Average $795.6M in FY 2001 $)* *Fox et al “Test and Evaluation Costs for Aircraft and Guided Weapons” Rand Project Air Force Report., 2004 + GAO-05-301, Assessment of Selected Major Weapon Programs, March 2005 GAO Program Reviews+
RDT&E Value Hierarchy What you see depends on where you sit : RDT&E Value Hierarchy What you see depends on where you sit I II III IV V PLAN INSTALL TEST REMOVE SHIP Wind Tunnel Facility Test Program DESIGN BUILD TEST ANALYZE REPORT Wind Tunnel Test Program Air Vehicle Development Program Loads
Prop Int
Stores
Perf
S&C Wind Tunnel Flight CR TD SD&D P&D O&S A B C IOC FOC Acquisition Program C4ISR
Avionics
Stores
Propulsion
Airframe System Integration Focus Acquisition
Cycle Platform
Integration Subsystem
Development Functional
Discipline Test Facility
Process
Aeronautical RDT&E MOPs, MOEs: Aeronautical RDT&E MOPs, MOEs Level Measures of Performance Measures of Effectiveness V
IV
III
II
I Required capability delivered within cost and schedule
Weapon systemMOPs/ MOEs met or exceeded Subsystem integration issues eliminated before 1st flight
Pass IOT&E on first attempt
Interoperability issues resolved before IOT&E Elimination of subsystem defects before 1st flight
Elimination of subsystem integration issues before 1st flight Vehicle MOPs met
Accurate flight predictions
Discovery of design flaws in first pass Accurate simulation of key phenomena – no surprises in flight
Discovery of design flaws in first pass
Insight into design refinements required to meet vehicle MOPs SDD$/Total $
T&E $/SDD $
# and severity of quality escapes
CIP $/SDD $
SDD Time/Acquisition Time Total Wind Tunnel hours
Total Flight Test hours
# design defects discovered
Time and cost to resolve defects
Relative impact on LCC of defects discovered # design defects discovered
Time and cost to resolve defects
Relative impact on LCC of defects discovered
Total Wind Tunnel hours
Total Flight Test hours
# design defects discovered
Time and cost to resolve defects
Relative impact on LCC of defects discovered Data accuracy (bias and precision)
$/data point
$/UOH
Polars/hour
Vehicle mission coverage
Summary: Summary Major Gains to be Made by Emphasizing Effectiveness, Not Just Efficiency
Increasing Effectiveness Drives Several Key Aspects of the AF T&E Roadmap Vision
Institute T&E expeditionary mindset over weapon system life cycle
Enable Integrated Test Teams with independent verification capability
Institutionalize feedback on T&E processes
Build Integrated DT/OT processes
…And We Haven’t Even Talked about System of Systems and Interoperability Testing Trained People and Improved Processes Are More Important to Effectiveness than Facilities
Vision for Air Force Test & Evaluation: Vision for Air Force Test & Evaluation Or how we all survived without killing each other… George Kailiwai III
Technical Advisor, Air Force Flight Test Center
Vision for Air Force Test & Evaluation Air Force Flight Test Center : Vision for Air Force Test & Evaluation Air Force Flight Test Center We’re in this all together.
A “family” can overcome any adversity.
Family members may not always agree, but in the end, family members generally make decisions that are best for the family.
Collaboration & teaming to date has been “UNPRECEDENTED”
Collaboration & teaming will be crucial to future success Thank AFMC/A3 for leading the effort and AEDC,
46th TW, AFOTEC and AF/TE for having the courage
to take the first steps….
46TW Perspective Mr. Bob Arnold: 46TW Perspective Mr. Bob Arnold
AFOTEC Perspective Col. Joe Arvai for Mr. Doug Marlowe: AFOTEC Perspective Col. Joe Arvai for Mr. Doug Marlowe
AF/TE Perspective Mr. Wally Beard for Mr. Dave Hamilton : AF/TE Perspective Mr. Wally Beard for Mr. Dave Hamilton
Panel Discussion Moderator: Mr. Dave Bond: Panel Discussion Moderator: Mr. Dave Bond