ExtOps12 05 00

Independent Oversight Office Evaluation ofBNL Self-Assessment (SA) Programs : Independent Oversight Office Evaluation of BNL Self-Assessment (SA) Programs Presentation to Extended Operations Council December 5, 2000 John Usher, Self-Assessment Oversight


IO Evaluation of BNL SA ProgramsOverview : IO Evaluation of BNL SA Programs Overview Background What is IO’s Role? FY99 and FY00 Evaluation Results Strengths Areas for Improvement FY01 Evaluations


IO Evaluation of BNL SA ProgramsBackground : IO Evaluation of BNL SA Programs Background What drives SA Programs? ISM: Feedback and Improvement QA Program BNL Critical Outcomes Regulatory Requirements


IO Evaluation of BNL SA ProgramsBackground : IO Evaluation of BNL SA Programs Background How are requirements met? BNL’s Integrated Assessment Program (IAP) provides the means to measure performance against established objectives, to analyze performance, and to identify opportunities for improvement.


IO Evaluation of BNL SA ProgramsBackground : IO Evaluation of BNL SA Programs Background How do BNL organizations implement IAP? BNL organizations develop Self-Assessment Plans, conduct assessment activities, and analyze assessment results in conformance with the SBMS Subject Area: Integrated Assessment.


IO Evaluation of BNL SA ProgramsBackground : IO Evaluation of BNL SA Programs Background How do BNL organizations improve performance? BNL organizations develop (corrective) action plans based on identified improvement opportunities and then carry out these action plans to effect improvements in performance.


IO Evaluation of BNL SA ProgramsWhat is IO’s Role? : IO Evaluation of BNL SA Programs What is IO’s Role? The IO Office evaluates the effectiveness, efficiency and adequacy of organizational self-assessment programs: Verifies that the organization’s SA Plan conforms to IAP expectations (approach) Verifies that the organization’s SA process is implemented as planned (deployment) Verifies that the organization makes effective use of the results of self-assessment


IO Evaluation of BNL SA ProgramsWhat is IO’s Role? : IO Evaluation of BNL SA Programs What is IO’s Role? How does IO evaluate SA programs? Evaluators measure the organizational SA process against established criteria based on expectations derived from IAP, etc. Criteria are listed at http://www.io.bnl.gov/safy01.htm


IO Evaluation of BNL SA ProgramsFY99 and FY00 Evaluation Results : IO Evaluation of BNL SA Programs FY99 and FY00 Evaluation Results FY99 FY00 Outstanding Excellent Good Program Ratings


IO Evaluation of BNL SA ProgramsFY99 and FY00 Evaluation Results : IO Evaluation of BNL SA Programs FY99 and FY00 Evaluation Results Organizational programs improved dramatically from FY99 to FY00. One Directorate (EM) improved from good to outstanding by turning 'opportunities for improvement' into 'strengths'.


IO Evaluation of BNL SA ProgramsFY99 and FY00 Evaluation Results : IO Evaluation of BNL SA Programs FY99 and FY00 Evaluation Results What enabled these improvements? Improved direction and guidance Technical assistance Organizational experience (learning)


IO Evaluation of BNL SA ProgramsStrengths : IO Evaluation of BNL SA Programs Strengths Management involvement in, and commitment to, self-assessment and continuous improvement was uniformly observed. Managers expressed 'pride of ownership', stressed the importance of serving as positive examples and even as 'salespersons'.


IO Evaluation of BNL SA ProgramsStrengths : IO Evaluation of BNL SA Programs Strengths Almost all BNL organizations have developed comprehensive self-assessment programs covering research/operations, business functions, ESandamp;H, and human resources. Organizations largely built their SA programs on a broad spectrum of existing assessment activities.


IO Evaluation of BNL SA ProgramsStrengths : IO Evaluation of BNL SA Programs Strengths Organizational performance objectives are clearly linked to Critical Outcomes, IAP framework, other higher level goals/objectives, and regulations. Assessment activities are linked to organizational objectives and include assessments required by BNL.


IO Evaluation of BNL SA ProgramsStrengths : IO Evaluation of BNL SA Programs Strengths Organizations exhibit a strong customer focus in their SA programs and objectives by appropriately considering customer expectations. Organizations actively solicit customer feedback regarding the organization’s performance, services, or products.


IO Evaluation of BNL SA ProgramsAreas for Improvement : IO Evaluation of BNL SA Programs Areas for Improvement Some SA Plans do not contain a description of the entire SA process (cycle) or each process element. Responsibilities for SA elements (except assessments) are not clearly described including those for stewardship of the SA process.


IO Evaluation of BNL SA ProgramsAreas for Improvement : IO Evaluation of BNL SA Programs Areas for Improvement Processes for analyzing and evaluating the results of assessment activities are not clearly documented. Except for quantitative measures, bases for 'rating' of performance are not documented.


IO Evaluation of BNL SA ProgramsAreas for Improvement : IO Evaluation of BNL SA Programs Areas for Improvement In some organizations, there is a need to improve: Tracking of assessment activities Tracking of corrective actions Records management


IO Evaluation of BNL SA ProgramsAreas for Improvement : IO Evaluation of BNL SA Programs Areas for Improvement What about improvements in IAP? Recommendations for both specific and programmatic improvements have been communicated and included in: Integrated Assessment subject area IO Evaluation Criteria


IO Evaluation of BNL SA ProgramsFY01 Evaluations : IO Evaluation of BNL SA Programs FY01 Evaluations