new format presentation ipe ltsn leeds 24 11 04

Interprofessional assessment: challenges and opportunities: 

Interprofessional assessment: challenges and opportunities Workshop Best practice in constructing MCQs and EMQs Principles of item writing developed by UMAP: how interprofessional can it be?

Slide2: 

About UMAP Challenges of exam construction Examples Question writing for IPE What should be assessed Principles of question construction Question writing Question reviewing Session plan

Slide3: 

Leeds, Liverpool, Manchester, Newcastle, Sheffield New partners 2005 Quality process Learning blueprint About UMAP

Slide5: 

Purpose What is being tested What is the most appropriate test format Consider reliability and validity “Basic Principles of Assessment” www.ltsn-01.ac.uk/docs/assessment_priciples_umap.pdf Challenges of exam construction

Slide6: 

Theme - Pharmacology Focus - Mechanism of action Stem A 54 year old man underwent a laparotomy 10 days ago. Three days following surgery he developed a purulent discharge from the wound for which he was given antibiotics. A swab of the wound now shows Staphylococcus Aureus. Options A Increase in phosphodiastrase B Inhibition of bacterial cell wall synthesis C Inhibition of DNA transcription D Inhibition of mitochondrial function E Inhibition of RNA translation MCQ example

Slide7: 

Theme - Abdominal pain Focus - Diagnosis Options A Abdominal aneurysm K Kidney stone B Appendicitis L Mesenteric adenitis C Bowel obstruction M Mesenteric artery thrombosis D Cholecystitis N Ovarian cyst – ruptured E Colon cancer O Pancreatitis F Constipation P Pelvic inflammatory disease G Diverticulitis Q Peptic ulcer disease H Ectopic pregnancy R Perforated peptic ulcer I Endometriosis S Pyelonephritis J Hernia T Torsion Lead in For each of the following patients with abdominal pain, select the most likely diagnosis. Each option may be used once, more than once or not at all. Stem 1 of 5 An 84 year old man in a nursing home has increasingly poorly localised lower abdominal pain recurring every 3-4 hours over the past three days. He has no nausea or vomiting; the last bowel movement was not recorded. Examination shows a soft abdomen with a palpable, slightly tender, lower left abdominal mass. Haematocrit is 28%. Leukocyte count is 10,000/mm3. Serum amylase activity is within normal limits. Test of the stool for occult blood is positive. EMQ example

Slide8: 

Fit for a number of purposes Reduce burden of exam preparation Suitable for large number of topic areas Evidence of reliability Validity good – particularly content validity Why bank MCQs and EMQs

Slide9: 

How can we construct IPE relevant questions? - Design question stems to reinforce key interprofessional learning scenarios? - Set questions which directly test knowledge relevant to interprofessional contexts? Topic areas - Common or critical topics - “Barn door” rather than “red herring” Question writing for IPE

Slide10: 

What can be assessed?

Slide11: 

Let’s compare notes

Slide12: 

Stem Which muscle is responsible for lateral rotation and extension of the hip joint and whose contraction supports the extended knee? Options A Gluteus maximus B Gluteus minimus C Hamstrings D Iliopsoas E Obturator internus Principles of MCQ Writing – Example 1a

Slide13: 

Stem A 65 year old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty in straightening his leg. Lead in Which muscle is most likely to have been injured? Options A Gluteus maximus B Gluteus minimus C Hamstrings D Iliopsoas E Obturator internus Principles of MCQ Writing – Example 1b

Slide14: 

Knowledge Comprehension Application Analysis Synthesis Evaluation Principles of MCQ Writing – Bloom’s Taxonomy

Slide15: 

Stem The way to a man’s heart is… Options A Through his aorta B Through his stomach C Through his chest D Via cardiac surgery E The same as the way to a woman’s From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp. PROBLEM? SUBJECTIVE! Principles of MCQ Writing – Example 2

Slide16: 

Stem Adults should drink…. Options A 8 glasses of water per day B Be given too much salty food C Take folic acid in early stages of pregnancy D Heart disease when they are over 40yrs old and should take aspirin daily E 1 in 10 babies PROBLEM? GRAMMATICAL CUES! Principles of MCQ Writing – Example 3

Slide17: 

Stem A patient presents with a fractured rib and complains of frequent vomiting. Lead in If the patient is lucid, which of the following would you NOT do in the management of this patient? Options A Discharge to GP B Keep in for observation C Admit to hospital D Section under Mental Health Act E Prepare for surgery PROBLEM? AND, OR, IF ETC. SHOULD BE AVOIDED! Principles of MCQ Writing – Example 4

Slide18: 

What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? A B *C* D E top 20% 1 0 99 0 0 bottom 20% 8 1 90 1 0 A 2 year old boy has a 1 week history of oedema. Blood pressure is 100/60 mmHg, and there is generalized oedema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. top 20% 0 0 98 2 0 bottom 20% 5 2 82 8 1 A 2 year old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mmHg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. top 20% 0 1 98 1 0 bottom 20% 10 9 66 10 5 From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp. Principles of MCQ Writing – Stem Length

Slide19: 

D C A E B Least Most Correct Correct Answer Answer SINGLE CONTINUUM OF CHOICES Principles of MCQ Writing – Option lists

Slide20: 

Question writing

Slide21: 

Question reviewing

Slide22: 

Comments

Slide23: 

Thank you