Reliability and Validity: Reliability and Validity
THE GREAT TRIAD : Reliability – Does the instrument measure accurately and consistently? Validity- Does the instrument measure what it says it measures? Objectivity- Is the instrument immune to the personal attitudes and opinions of the researcher? THE GREAT TRIAD
RELIABILITY : RELIABILITY The degree to which test scores are free from errors of measurement. The degree of consistency with which an instrument measures what it is measuring.
RELIABILITY AND ERROR : Error Variance Systematic Error – there is a system (methods are planned, orderly, and methodical) Unsystematic Error – there is a lack of a system (occurrences are presumed to be random) RELIABILITY AND ERROR
3 IMPORTANT MEASURES OF RELIABILITY : 1. Coefficient of Stability How stable is it over time? 2. Coefficient of Internal Consistency Is the test internally consistent? Coefficient of Equivalence Is the test the same/different than others? 3 IMPORTANT MEASURES OF RELIABILITY
MEASURES OF RELIABILITY: Test-Retest Reliability (Stability) Alternate, Parallel, or Equivalent Forms Reliability (Equivalence) Split-Half or Odd-Even Reliability (Consistency) Cronbach’s Coefficient Alpha Kuder -Richardson Interrater Reliability MEASURES OF RELIABILITY
Standard Error of Measurement: AKA: SEM Every score has 2 parts: True Score + Error The SEM estimates the error and places a band around the given score. The true score will fall within the range. This is referred to as a confidence band. Standard Error of Measurement
SEM Practice: If the SEM is 6 and the person’s score was 90, we would say that his true score would be between 84 and 96. 90 – 6 = 84, 90 + 6 = 96. If the score was 112 and the SEM was 5, what would the Confidence band be? 107 – 117. SEM Practice
Concerns: High coefficients of stability does not mean that internal consistency is high. Reliability coefficients apply to data and not to instruments. Reliability estimates consistency Concerns
VALIDITY: VALIDITY The degree to which all of the accumulated evidence supports the intended interpretation of test scores for the intended purpose The ability of research instruments to measure what they say they measure
TYPES OF VALIDITY: Content Validity Face Validity Criterion-Related Validity Concurrent Validity Predictive Validity TYPES OF VALIDITY
TYPES OF VALIDITY : TYPES OF VALIDITY Construct Validity Factor Analysis Meta-analysis Convergent Validity Discriminant Validity
Internal Invalidity: Ask the question, “Are the measurements influenced only by the treatment or are there other influences which change it?” Concerned with the DV Internal Invalidity
Internal Invalidity: History Maturation Testing Instrumentation Regression Selection Interaction Mortality John Henry Effect Internal Invalidity
Additional Problems: Hawthorne Effect – If subjects know they are part of an experiment – or if they are given more attention because of the experiment – their performance sometimes improves. Halo Effect – The halo effect occurs when a trait that is not being evaluated influences a researcher’s rating on another trait. Rosenthal Effect – The experimenter’s beliefs about the individual may cause the individual to be treated in a special way so that the individual begins to fulfill the experimenter’s expectations. Reactive Effect – The very presence of the researcher can influence behavior. Additional Problems
External Invalidity: Ask the question, “How confidently can I generalize my experimental findings to the world?” Concerned with the IV External Invalidity
External Invalidity: Effects of Testing Treatment and Subject Testing and Subject Multiple Treatments External Invalidity
IMPORTANT!!!: A Test Must be Reliable in Order to be Valid! A Test Can be Reliable Even if it Isn’t Valid! NCE hint: Validity is more important than reliability. IMPORTANT!!!
Observation Obstacles: Personal Interest Early Decision Personal Characteristics (Being human) Observation Obstacles
Measurement Types: Nominal Measurement Variables which are categorized into discrete groups. Ordinal Measurement Variables which are rank ordered. Interval Measurement In order to measure distance between data points, we need a scale of equal, fixed gradation. No meaningful zero! Ratio Measurement The same as interval measurements except... With a meaningful zero! Measurement Types
PowerPoint Presentation: Name Attitude Score Test Score Test Rank Test Grade Gender Chris 80 92 2 A M Macy 55 81 4 B F Mike 90 96 1 A M Tina 84 85 3 B F
PowerPoint Presentation: Name Attitude Score Test Score Test Rank Test Grade Gender Chris 80 92 2 A M Macy 55 81 4 B F Mike 90 96 1 A M Tina 84 85 3 B F