BreakoutC BobVallario

Slide1: 

Planning and Prospective Evaluations: Office of Science Highlights Bob Vallario Office of Science U.S. Department of Energy

The Office of Science Multi-Dimensional Mission: 

The Office of Science Multi-Dimensional Mission Environmental Quality Energy Science Nuclear Security Critical U.S. Infrastructure and Core Capabilities Advanced Computation Nanoscience Properties of Matter & Energy Genomics Climate Change Fusion Power Powerful Tools Human Capital Knowledge Foundations for DOE’s Applied Missions Interdisciplinary Challenges in Scientific Complexity 3 2 1 Critical Disciplines

Evaluation and Planning:The Challenge: 

How do you characterize and attribute value to the creation of new knowledge? How do you trace the broad diffusion of scientific knowledge? How do you gauge the impacts of diffused knowledge on downstream outcomes (usually built upon many complex, contributing factors)? How do you use the results to bring value and focus to your programs in a context of: mission emphases and core capabilities evolving national priorities significant opportunities a dynamic science landscape Evaluation and Planning:The Challenge

Some Commonly Held Views: 

Some Commonly Held Views Evaluation for basic research is most successful around the process and outputs of science (sometimes referred to as process-outcomes) Quality of Science Relevance of Science Leadership and/or Performance of Science Expert review is the cornerstone for evaluating the process and outputs of science programs. Evaluating the outcomes (and estimating the benefits) of science, while a desirable goal, cannot effectively account for the complexities of knowledge diffusion, and the broad range of interactions and factors that contribute to innovation, and the full range of benefits. Instead we select parts of the story to examine in detail, and rely on indicators. Despite the difficulties, we can all be doing better!

Established Methods: 

Established Methods Regular peer review for the university grant programs, laboratory research programs, and facilities. Cost, schedule, technical scope, and management reviews (“Lehman Reviews”) of construction projects, i.e., major scientific facilities Regular Advisory Committee reviews of facilities and programs Regular external assessments of programs, e.g., by JASONS or NRC Metrics and customer survey for facilities Historical retrospectives and annual highlights Tracking of a few selected metrics such as prizes/publications Discipline/sub discipline benchmarking Committee of Visitors (review of science management) Tracking of PART indicators Case studies

New/Evolving Methods: 

New/Evolving Methods Patent & Bibliometric Analyses, Data Mining, & Visualization Tools Foresighting Options Theory Network Analysis Benchmarking Innovative S&T Metrics Mission Mapping Outcome Mapping

In Particular, Prospective Methods: 

In Particular, Prospective Methods Advisory Committee Long-Range Plans Science/Technology Roadmaps Foresight Studies Patent & Bibliometric Analysis, Data Mining, & Visualization Tools Portfolio Analyses Prospective Case Studies Options Theory Network Analysis Expanded International Benchmarking Innovative S&T Metrics

A Sample of Prospective Evaluations: 

A Sample of Prospective Evaluations Advisory Committee Evaluations Joint Science-Technology Roadmaps

A Sample of Prospective Evaluations – Contd.: 

A Sample of Prospective Evaluations – Contd. Foresight Studies Data Mining (Patent Analysis Shown Below) The 2002 edition of the State of the Future addresses in a coherent and forward-looking way many of the issues that impact on the evolution of our planet and the people living on it. At a time of accelerating globalization and an increasing need to organize our societies accordingly, it is essential for policy-makers at all levels to be able to rely on such solid analysis of global trends. The study is a good example of building collective intelligence based on a large network across the world. Philippe Busquin, Commissionner for Science & Technology The European Commission

A Sample of Prospective Evaluations – Contd.: 

A Sample of Prospective Evaluations – Contd. Portfolio Analysis: Research Mission Mapping (BES Program)

A Sample of Prospective Evaluations – Contd.: 

A Sample of Prospective Evaluations – Contd. Portfolio Analysis: Research Outcome Mapping Producing Clean Fuels Catalysis for Efficient Pathways Molecular understanding of catalysts for petroleum refining and chemical manufacturing…. catalysis of carbon-hydrogen bonds….catalysis for fuel cells…biocatalysis Improved Fuel Feedstocks Through Plant Biology Genetic, metabolic, and enzymatic properties of plants supporting biomass fuels….. regulation of photosynthetic processes…biochemistry… bioenergetics… New Approaches for Capturing Solar Energy Photochemical processes for solar energy conversion….. ….photocatalysis …. foundations for future efficient light energy to chemical energy conversions… Hydrogen/Clean Fuel Systems through Exotic Microbial Processes Genetic components that regulate methane and hydrogen producing organisms….enzymatic processes for possible fuel conversion….possible exploitation of extremophiles to ingest feedstocks, sequester carbon, and give off hydrogen… Bio-fuels Through Microbial Conversion Organisms and processes for conversion of lignocellulosics into fuels…biochemistry… bioenergetics…. microbial fermentation…..

Culmination: Integrated SC Long-Term Plans: 

Culmination: Integrated SC Long-Term Plans Crosscutting perspective Transparent decision criteria and matrix Structured evaluations Integrated timelines Companion document to Strategic Plan DRAFT Strategic Plan Nearing completion, still in draft Strategic framework, 20 year timeframe Three key features, influenced by PART and Administration focus: Budget, management, performance alignment Strategic (Performance) Timelines for each program Major priorities identified and articulated up front Other: incorporates PART long-term measures, maps with DOE Strategic Plan, DOE plan intermediate objectives built into timelines

DRAFT SC Strategic Plan: 

DRAFT SC Strategic Plan DRAFT

Take-Away Messages: 

Take-Away Messages SC has a long-tradition and successful record in conducting technical evaluations, and our advisory committee process is well regarded within and outside the Department Results from evaluations are used to inform key programmatic decisions, including planning, budgeting, management and more, but the linkages are not “formula driven”. Over the last few years, we have worked to create more seamless connections between planning, budget, and evaluative processes and our PART scores reflect these improvements (Christine Chalk’s presentation). Perhaps these connections are best illustrated by our draft Strategic Plan, and our goal is to communicate the importance of these connections to our science community through this and other top-tier venues.