Reliability and Validity

Reliability and Validity: 

Reliability and Validity

THE GREAT TRIAD : 

Reliability – Does the instrument measure accurately and consistently? Validity- Does the instrument measure what it says it measures? Objectivity- Is the instrument immune to the personal attitudes and opinions of the researcher? THE GREAT TRIAD

RELIABILITY : 

RELIABILITY The degree to which test scores are free from errors of measurement. The degree of consistency with which an instrument measures what it is measuring.

RELIABILITY AND ERROR : 

Error Variance Systematic Error – there is a system (methods are planned, orderly, and methodical) Unsystematic Error – there is a lack of a system (occurrences are presumed to be random) RELIABILITY AND ERROR

3 IMPORTANT MEASURES OF RELIABILITY : 

1. Coefficient of Stability How stable is it over time? 2. Coefficient of Internal Consistency Is the test internally consistent? Coefficient of Equivalence Is the test the same/different than others? 3 IMPORTANT MEASURES OF RELIABILITY

MEASURES OF RELIABILITY: 

Test-Retest Reliability (Stability) Alternate, Parallel, or Equivalent Forms Reliability (Equivalence) Split-Half or Odd-Even Reliability (Consistency) Cronbach’s Coefficient Alpha Kuder -Richardson Interrater Reliability MEASURES OF RELIABILITY

Standard Error of Measurement: 

AKA: SEM Every score has 2 parts: True Score + Error The SEM estimates the error and places a band around the given score. The true score will fall within the range. This is referred to as a confidence band. Standard Error of Measurement

SEM Practice: 

If the SEM is 6 and the person’s score was 90, we would say that his true score would be between 84 and 96. 90 – 6 = 84, 90 + 6 = 96. If the score was 112 and the SEM was 5, what would the Confidence band be? 107 – 117. SEM Practice

Concerns: 

High coefficients of stability does not mean that internal consistency is high. Reliability coefficients apply to data and not to instruments. Reliability estimates consistency Concerns

VALIDITY: 

VALIDITY The degree to which all of the accumulated evidence supports the intended interpretation of test scores for the intended purpose The ability of research instruments to measure what they say they measure

TYPES OF VALIDITY: 

Content Validity Face Validity Criterion-Related Validity Concurrent Validity Predictive Validity TYPES OF VALIDITY

TYPES OF VALIDITY : 

TYPES OF VALIDITY Construct Validity Factor Analysis Meta-analysis Convergent Validity Discriminant Validity

Internal Invalidity: 

Ask the question, “Are the measurements influenced only by the treatment or are there other influences which change it?” Concerned with the DV Internal Invalidity

Internal Invalidity: 

History Maturation Testing Instrumentation Regression Selection Interaction Mortality John Henry Effect Internal Invalidity

Additional Problems: 

Hawthorne Effect – If subjects know they are part of an experiment – or if they are given more attention because of the experiment – their performance sometimes improves. Halo Effect – The halo effect occurs when a trait that is not being evaluated influences a researcher’s rating on another trait. Rosenthal Effect – The experimenter’s beliefs about the individual may cause the individual to be treated in a special way so that the individual begins to fulfill the experimenter’s expectations. Reactive Effect – The very presence of the researcher can influence behavior. Additional Problems

External Invalidity: 

Ask the question, “How confidently can I generalize my experimental findings to the world?” Concerned with the IV External Invalidity

External Invalidity: 

Effects of Testing Treatment and Subject Testing and Subject Multiple Treatments External Invalidity

IMPORTANT!!!: 

A Test Must be Reliable in Order to be Valid! A Test Can be Reliable Even if it Isn’t Valid! NCE hint: Validity is more important than reliability. IMPORTANT!!!

Observation Obstacles: 

Personal Interest Early Decision Personal Characteristics (Being human) Observation Obstacles

Measurement Types: 

Nominal Measurement Variables which are categorized into discrete groups. Ordinal Measurement Variables which are rank ordered. Interval Measurement In order to measure distance between data points, we need a scale of equal, fixed gradation. No meaningful zero! Ratio Measurement The same as interval measurements except... With a meaningful zero! Measurement Types

PowerPoint Presentation: 

Name Attitude Score Test Score Test Rank Test Grade Gender Chris 80 92 2 A M Macy 55 81 4 B F Mike 90 96 1 A M Tina 84 85 3 B F

PowerPoint Presentation: 

Name Attitude Score Test Score Test Rank Test Grade Gender Chris 80 92 2 A M Macy 55 81 4 B F Mike 90 96 1 A M Tina 84 85 3 B F