Using Written Evaluation Tools_Ch20

Using Written Evaluation Tools Chapter 20:

Using Written Evaluation Tools Chapter 20 Galveston College EMS EMS Instructor Development 2009

Slide 2:

2 / 32 “To those of you who received honors, awards, and distinctions, I say well done. And to the “C” students, I say you too may one day become President of the United States.” –George W. Bush

Slide 3:

3 / 32 “C’s Get Degrees” – Unknown “A great teacher never strives to explain his vision—he simply invites you to stand beside him and see for yourself.” – Rev. R. Inman

Topics:

4 / 32 Topics The Written Evaluation Constructing Written Exams Using Limited Response Items Using Open Response Items Homework & Research Projects Administering Exams Analysis of Exams Grading Strategies

The Written Evaluation:

5 / 32 The Written Evaluation Formal evaluation Provides insight on student knowledge Evaluates cognitive domain May evaluate lower levels of affective domain Easier for statistics collection

Purpose of Written Exam:

6 / 32 Purpose of Written Exam Provides information about: Student knowledge of subject Mastery of cognitive objectives Allows progress to advanced objectives Measures presentation of presented material

Construction of Written Exams:

7 / 32 Construction of Written Exams Carefully consider purpose Use a Blueprint Develop draft items Constructing the Exam

Slide 8:

8 / 32 Figure 20-1

Sample Blueprint:

9 / 32 Sample Blueprint Table A

Sample Blueprint:

10 / 32 Sample Blueprint The instructor next considers the level of thinking required for the objectives. He assesses objectives written for the module and determines the percentage of objectives for each content area that are provided for each cognitive level. Table B shows the results. Table B

Sample Blueprint:

11 / 32 Sample Blueprint The instructor next combines Tables A and B to determine the percentage of items that will be needed for each area and each level. This is calculated by multiplying the percentages assigned to each content area (Adjusted Average column from Table A) and the percentage of each level shown on Table B. The results are Shown in Table C.

Sample Blueprint:

12 / 32 Sample Blueprint The instructor previously decided that the examination would consist of 100 items of equal weight (each item worth one point). The number of question required is determined by multiplying the percentage in each column by 100 (the total number of items on the examination). Table D shows the number of items needed for each level with each area.

Develop Draft Exam Items:

13 / 32 Develop Draft Exam Items Pick twice the number determined by blueprint Review & edit stage Editor’s considerations: Grammatical or spelling problems Items clearly stated/meet course objectives Material presented to class Item constructed properly Correct answers

Exam Construction Process:

14 / 32 Exam Construction Process Be consistent Use consistent strategy Use capital/lower case formatting consistently Answer sheets consistent with exam Provide clear/complete instructions

Exam Construction Process (Cont):

15 / 32 Exam Construction Process (Cont) Short answer spaces Organize exam in logical manner Begin with easiest to hardest questions (ideally) With Scenarios have logical sequence

Using Limited Response Items:

16 / 32 Using Limited Response Items True/False Items Matching Items Multiple-Choice Items

Using Open Response Items:

17 / 32 Using Open Response Items Completion Items Essay Items Short Answer Items

Homework & Research Projects:

18 / 32 Homework & Research Projects Homework Research Projects

Homework:

19 / 32 Homework Used as formative evaluation Spread out evenly across course Should have grade to show importance Include mix of easy & difficult Encourage collaboration

Homework Examples:

20 / 32 Homework Examples Workbook assignments Definition work sheets Short research projects Writing summaries of objectives or key points from lectures Should provide examples of what to expect for summative evaluations

Research:

21 / 32 Research To give students ability to synthesize information on their own Assess higher level cognitive learning Promotes student autonomy Allow students to choose their topic But prepare a list of topics Provide guidelines Process more important than product

Administering Exams:

22 / 32 Administering Exams Environmental Considerations Proctoring Supervise Be available to answer questions Time Limits ½ minute per T/F items 1 minute per multiple-choice items 2 minutes per short answer items 10-15 minutes per limited essay items 30 minutes for broader essay items

Analysis of Written Exams:

23 / 32 Analysis of Written Exams Posttest Review Good teaching tool Allows students to see why Difficulty Level Percentage of students who answer each item correctly Discrimination Index Comparison of those who scored well against those who did not

Difficulty Index:

24 / 32 Difficulty Index ID = (C/T)*100 ID = Item difficulty C = number of correct responses T = total number of students Example: (8/10)*100 = 80% Goal is to use items above 90% or below 30%

Calculating Item Discrimination:

25 / 32 Calculating Item Discrimination Determine 10 highest & 10 lowest scores For each question record number of students in both groups that answered correctly Compute the discrimination ratio by subtracting number of students in bottom group answered correctly from number of students in top group then divide by 10

Calculating Item Discrimination:

26 / 32 Calculating Item Discrimination The discrimination ratio should fall between -1.0 and +1.0 The closer to +1.0 the more effective the item distinguishes students who know the material Example: If 9 from the top group and only 2 from bottom group then 9-2=7 7/10 = +0.7 and thus the item is effective

Grading Strategies:

27 / 32 Grading Strategies Grading Essays Norm-Referenced Grading Criterion-Referenced Grading Setting a Cut Score

Grading Essays:

28 / 32 Grading Essays Grading essays are inherently subjective Skim assignment quickly before grading Have predetermined guidelines Have previous examples for students Grade essays question by question rather than student by student Avoid judging based on extraneous factors Repeat process days later or use multiple graders

Norm-Referenced Grading:

29 / 32 Norm-Referenced Grading Normative grading is comparing performance with the performance of other students Usually referred as grading on a curve This method compensates for poorly constructed exams

Criterion-Referenced Grading:

30 / 32 Criterion-Referenced Grading Base grades on mastery of course objectives Requires specific course objectives Based on blueprint Example: 90% mastery = A 80% mastery = B Etc., etc.

Setting Cut Scores:

31 / 32 Setting Cut Scores Two Primary Strategies Based on difficulty of exam Set preset passing score Sometimes an institutional policy EMS usually higher than academics Basics 70 or 75% Advanced 80% Different assignments may have different cut scores

Conclusion:

32 / 32 Conclusion Well constructed written exams most effective means of assessing cognitive objectives Instructors must consider purpose for exams Instructors should strive for consistency in written examinations