bwright

How to convince your friends NOT to misuse raw scores: 

How to convince your friends NOT to misuse raw scores Benjamin D. Wright Institute for Objective Measurement & MESA Psychometric Laboratory bd-wright@uchicago.edu

THE TROUBLE WITH RAW SCORES: 

THE TROUBLE WITH RAW SCORES The BROKEN BUCKET of Missing Data The CRUMBLING CATEGORIES of Lumpy Ratings The DIRTY DATA of Unpredictable Responses The PERVERSE PRECISION of Extreme Scores The RUBBER RULER of Irregular Intervals and Squashed Extremes

The BROKEN BUCKET of Missing Data : 

The BROKEN BUCKET of Missing Data Compare Two Patients on an 8-item, 7-category Functional Independence Measure: Patient A: 2 3 3 m m m m 4 = 12 Patient B: 1 2 2 2 2 2 2 3 = 16 Which Patient is more Able? Patient B has the higher score 16 > 12 on all 8 items, BUT On the 4 Items A and B have in common Patient A: 2 3 3 m m m m 4 = 12 Patient B: 1 2 2 m m m m 3 = 8 Patient A has the higher score of 12 > 8 Are you sure you want to misuse missing-data-leaking raw scores for missing-data-impervious measures?

The CRUMBLING CATEGORIES of Lumpy Ratings: 

The CRUMBLING CATEGORIES of Lumpy Ratings Rating Forms Offer Equally Spaced Categories 1 . 2 . 3 . 4 . 5 . 6 . 7 But Raters Reply with Unequally Spaced Responses 1 . . . 2 . . 3 4 . 5 . . 6 . . . . . 7 The Measure Distance of One More Point from Category 1 to 2 can be FOUR times BIGGER than The Measure Distance of One More Point from Category 3 to 4 !! Are you sure you want to mistake Lumpy Ratings for Equal Interval Measures?

The DIRTY DATA of Unpredictable Responses: 

The DIRTY DATA of Unpredictable Responses When Item Responses are Arranged from Easy Items to Hard Items you Expect Response Patterns like: 7 7 6 6 6 5 5 4 = 46 and 4 4 3 3 2 1 1 1 = 19 BUT suppose you get: 7 7 6 6 {1}5 5 4 = 41 ? or 4 4 3 3 {7}1 1 1 = 24 ? What then? Raw Scores are Blind to Unpredictable Responses. Only Quality Control of Well-Constructed Measures Tells you about Response Surprises Are you sure you want to suffer raw-score-dirty-data blindness instead of enjoying data-vigilant measures?

The PERVERSE PRECISION of Extreme Scores: 

The PERVERSE PRECISION of Extreme Scores The Statistical Precision of a Raw Score is MAXIMUM at exactly the place where the Information a Raw Score Provides is MINIMUM When a person gets the lowest possible score, their raw score precision is perfect. We know exactly the score their low ability implies. But we have no idea how far Below that Score their ability might be! It is the same with the highest possible score. We know exactly the score their high ability implies. But we have no idea how far Above that Score their ability might be! They are Off-Our-Scale and Our Precision for their Unknown Measure is ZERO! Are you sure you want to mistake imprecise raw scores for precise measures?

The RUBBER RULER of Irregular Intervals and Squashed Extremes: 

The RUBBER RULER of Irregular Intervals and Squashed Extremes When our items bunch in clumps of equally difficult items then a count of one more right answer within a clump implies only a little increase in our ability. But when we leap ahead and our next right answer is in a distinctly harder clump, then we see that this one more right implies a large increase in our ability. As for the ends of the test where one more right is from 0 to 1 or from all but one to all. Then the implied change in our ability is infinite.

WHAT ARE VARIABLES?: 

WHAT ARE VARIABLES? Length and weight may be real variables. But we construct their units of measure. Inches and ounces are our creations - Our own imaginative constructions. A variable is an amount of something which we can always picture as a distance >From Less -------------------------> To More We can arrange to experience evidence of this "something". But its measurement line and its units of measurement are up to us to construct.

EVIDENCE OF A VARIABLE?: 

EVIDENCE OF A VARIABLE? The variable and its evidence could be: length benchmarks exceeded health symptoms absent ability problems solved skill tasks completed attitude assertions condoned We can arrange to provoke occurrences of evidence and count how many pieces occur. But these counts are not measures.

REQUIREMENTS FOR MEASUREMENT: 

REQUIREMENTS FOR MEASUREMENT Pieces of evidence must be concrete to be observed. This necessary reality keeps them uneven in size. To measure we need an even abstraction, a line marked out in abstractly equal units. Pieces of evidence are unstable. They appear and disappear by accident. They are only probable signs of the variable which they are designed to manifest. To measure, we must find a way to connect the pieces of evidence we can arrange to observe to the probabilities of the measures we want.

WHAT IS MEASUREMENT?: 

WHAT IS MEASUREMENT? DISTANCE (Length) was our First Variable COUNTING Steps and Fingers was our First Measuring Operation The Trouble with Counting is its UNEQUAL UNITS How many apples fill a basket? How many oranges squeeze a glass? You may not believe it. But can we mix apples and oranges? We do it all of the time, by WEIGHING them!

CONSTRUCTING MEASURES: 

CONSTRUCTING MEASURES But weighing is a constructed abstraction.There are no tangible equal units. We have to invent them. Equal feet are abstracted from real feet. Equal pounds are abstract real weights We construct our instrumentation of the variables: length and weight to approximate units equal enough to serve our practical purposes We measure so that we can use the past to plan and navigate the future. But the future is by definition UNCERTAIN

HANDLING UNCERTAINTY: 

HANDLING UNCERTAINTY Imagine two batters: Smith bats 400 and Jones bats 200 So which one will hit at their next batter-up? No way to know ahead of time. Even Smith has only a 4 out of 10 record. We can’t wait to find out. So which one shall we send to the plate? Smith's odds for a hit are 2/3; Jones' odds for a hit are only 1/4 Smith odds for a hit are 8/3 times better than Jones'. Even though we know nothing for sure, Does any doubt remain as to who to send to bat? That's how we handle uncertainty. We use past experience to estimate PROBABILITIES and use these probabilities to forsee the future.

COUNTING ABSTRACT UNITS: 

COUNTING ABSTRACT UNITS To finish this job we have to construct a reproducible transition from counting concrete events, like right answers, observed or reported symptoms, relative agreements, frequency or importance categories to counting abstract units of equal size and wide generality. How can we do this?

INVERSE PROBABILITY: 

INVERSE PROBABILITY To deal with the uncertainty we ask Bernoulli, Bayes and Laplace and interpret our observation X as evidence of its occurence probability Px To construct unit equality we ask Campbell, Thurstone, Rasch and Luce & Tukey and define Px to satisfy the equation: log[Pnix/(1-Pnix)] = Bn - Di Pnix is the probability of a successful response Xni being produced by person n to item i Bn is the ability of person n Di is the difficulty of item I The construction of equal size and hence additive units is called CONJOINT ADDITIVITY

Slide17: 

For situations where Xni occurs in incremental steps such as Xni = 0,1,2,3,,,M This simple solution generalizes to log[Pnix/Pnix-1] = Bn - Di - Fix For situations where Xnijk = 0,M occurs as the result of a Rater j rating the performance of a Person n on a Task k This MEASUREMENT MODEL becomes log[Pnijkx/Pnijkx-1] = Bn - Di - Cj - Ak - Fix CONJOINT ADDITIVITY

CLOSING THE DEAL: 

CLOSING THE DEAL Raw scores cause problems with: missing data lumpy ratings unpredictable responses extreme scores irregular intervals and squashed extremes Rasch measurement provides a solution to these problems by: abstracting units of measurement using probabilities to predict futures constructing equal-sized intervals