Pharmacists’ Performance in the PEBC Qualifying Examination – Part II (OSCE): Pharmacists’ Performance in the PEBC Qualifying Examination – Part II (OSCE) Primary Author: Carol O’Byrne, B.S.P.- PEBC Co-Authors: John Pugsley, Pharm.D.- PEBC Lila Quero Munoz, Ph.D. – PEBC consultant Ashifa Keshavji, B.Sc.(Pharm.) - CPBC Doreen Leong, B.Sc.Pharm.- CPBC


Session Aims: Session Aims Identify issues in remediating competence Examine and compare CP (continuing practice) and RTP (return to practice) pharmacists’ OSCE results similarities and differences competency subscores and scale scores pass-fail frequencies Explore impact of differences in training and experience Explore implications and resource needs for remediation and rehabilitation Consider utility of an OSCE in pharmacists’ learning needs assessment


What are the Issues?: What are the Issues? Global demographics and demands Mobility, transitions and return to practice (RTP) Accountability and evidence-based practice Public access to information Increasing public scrutiny Learning needs / competency assessment Continuing professional development support systems Accessible, motivational resources and programs Mentors and other supports Incentives Partnerships


What is the Potential?: What is the Potential? Ontario ~10000 registered Part A pharmacists (2001) 14% of sample did not meet initial quality assurance standards 69% of these (10% of total sample) met standards on reassessment after remediation 31% (4 % of total sample) did not meet standards after remediation 1100+ may not meet initial standards; 300+ may need intensive remediation British Columbia (BC) 4066 registered pharmacists including 300 new (135 UBC, 165 external) 42 ‘return to practice’ 33% selected for QA 8 to 10% did not meet initial quality assurance standards Follow-up in progress Up to 400 pharmacists may not meet initial standards


Some Questions: Some Questions Who is responsible? Are all lapsed pharmacists capable of meeting standards? Is it cost-effective to develop and administer remedial programs? If we build them will they come? What do we need to build?


The Foundation: The Foundation Practice standards Desire to change/enhance practice Assessment tools Self-assessment plus peer review Performance or practice-based assessment Cognitive assessment Needs-focused solutions and supports Identified learning needs Targeted, engaging, supportive remediation Support systems Partnerships Regulation


Pharmacist Quality Assurance Programs – Canadian example #1: Pharmacist Quality Assurance Programs – Canadian example #1 COLLEGE OF PHARMACISTS OF BRITISH COLUMBIA All registered pharmacists assessed every 6 years Benchmark - Framework of Professional Practice Three phases Screening (KA or LPP) Confirmatory (any) Diagnostic (several) Multiple assessment tools, choices Knowledge assessment (KA) Learning and practice portfolio (LPP - assessed) OSCE (in partnership with PEBC) Practice audit QA and CPD resources Mentorlink program Canadian Pharmacy Practice Program (UBC) CPD programs (various) Regulations


Pharmacist Quality Assurance Programs – Canadian example #2: Pharmacist Quality Assurance Programs – Canadian example #2 ONTARIO COLLEGE OF PHARMACISTS A sample of Part A pharmacists must complete Benchmark – national competencies and standards of practice Assessment centre – includes Knowledge assessment OSCE stations Learning portfolio (review, consultation and learning plan) Diagnostic QA and CPD resources CPD programs (various) Mentors International Pharmacy Graduates program (UofT) Regulations


PEBC Certification Exam: PEBC Certification Exam 2-part exam: MCQ and 15-station OSCE Based on national competencies Summative assessment national certification at entry-to-practice Run concurrently twice yearly In multiple sites across Canada Bilingual (English and French) Annually, about 700 Canadian graduates and 1000 international graduates


The PEBC OSCE: The PEBC OSCE Practice-based assessment Scored using rating scales and checklists Scale scores for Communications, Problem-Solving and Overall Performance (incl. misinformation, risk, thoroughness, etc.) Checklist subscores for each competency assessed Performance reports for failing candidates Individual mean scores compared to Reference Group mean See Sample Station: Travelers Diarrhea


Sample OSCE Station: Sample OSCE Station Objectives Candidate instructions SP script Trained pharmacist assessors Checklist Critical and non-critical items Rating guidelines


Performance Reports: Performance Reports Exam not intended to be diagnostic Limited sample Stability of ratings Stability of subscores Sample report


PEBC Scoring Research: PEBC Scoring Research 2 pilot studies – applying generalizability theory  The PEBC OSCE is a reliable and defensible exam G coefficients 0.72 to 0.81 D coefficients 0.79 to 0.93  Assessors, SPs and site staff are responsible for a small proportion of candidates’ score variance (error) 81 to 91% of total variance was due to candidates and their differential performance in stations of varying contexts & difficulty 0 to 2% of total variance due to assessors 12 to 15% unexplained variance (SPs, materials, site staff other) Internal consistency (coefficient alpha) high overall (>0.9); scale and competency subscores vary


PEBC OSCE Findings: PEBC OSCE Findings By candidate group Reference group candidates (Canadian grads – 1st attempt) Non-reference group candidates (Canadian repeaters, all others) Non-certification candidates (includes CP, RTP, delayed licensure) Data analyses and comparisons Candidate comments Preparatory strategies Recommendations Assessor comments regarding failing non-certification candidates


Comments from & about candidates: Comments from & about candidates Noncert candidate comments Preparatory strategies and resources Include more prescriptions for evaluation Provide electronic references (vs print) Does not reflect hospital practice Assessor comments about failing noncert candidates Asked many questions but ignored patient’s answers Took a legal vs ethical approach Incomplete counselling – missed ‘shake well’; mentioned ‘local irritation’ only (no other side effects); provided no general/nondrug advice (e.g. diabetic care) Did not mention monitoring or possible need to refer


What about that ~10%?: What about that ~10%? CPBC Prior research indicated significant differences between gender and age Those working in ‘isolation’ or do not have a ‘community of practice’ are more vulnerable; often they don’t know what they don’t know Mentorship and self-directed learning works for some CPP recommended for some Inquiry Committee revoew OCP 70% pass on second attempt after self-guided learning A few have failed repeatedly (up to the 4th attempt) Language/communications skills a common factor Mentorship program developed for individual support, along with targeted workshops; several have completed the IPG program RPSGB Action plan to support pharmacist improvement: buddy/mentor, training/workshops, CPD, supervised practice, RPSGB inspector support Assessments Review/change context, system Fitness to practice review


Where do we go from here?: Where do we go from here? What did we learn from the PEBC OSCE? What can we learn from other professions? What does our profession need to do or change? Recommendations? Responsibility? Collaboration?


For more information…: For more information… Carol O’Byrne obyrnec@pebc.ca John Pugsley jpugsley@pebc.ca PEBC Web site: www.pebc.ca Ashifa Keshavji ashiva.keshavji@bcpharmacists.org Doreen Leong doreen.leong@bcpharmacists.org CPBC Web site: www.bcpharmacists.org