The Equivalence of Internet and Paper-and-pencil Personality Assessments: The Equivalence of Internet and Paper-and-pencil Personality Assessments Chuah Siang Chee Fritz Drasgow Brent Roberts Society for Industrial and Organizational Psychology, Toronto, April 12-April 14, 2002


Internet Assessment: Internet Assessment Access from more locations and to more people Convenience for the user and test administrator User access from Internet enabled computers Electronic data entry Relative speed and low cost for large test administrations


Database Driven Website: Database Driven Website


Are Internet Assessments Equivalent to Other Approaches? : Are Internet Assessments Equivalent to Other Approaches? Measurement Equivalence Mean-Level Differential Item Functioning (DIF) Examinees of equal ability but from separate populations differ in their probability of giving a particular answer.


Previous research on mean-level differences across modality: Previous research on mean-level differences across modality Mean-level Social Desirability Distortions Meta-analysis of non-cognitive tests across mediums Richman, Kiesler, Weisband and Drasgow (1999) Anonymity Lower Social Anxiety and Social Desirability with Internet Johnson (1999) Pasveer andamp; Ellard (1998) Self-Trust – No differences Davis (1999) Ruminative Responses – No differences


Problems with previous research on mean levels: Problems with previous research on mean levels Comparison groups were not equivalent. Different recruiting techniques for groups Different samples sizes Unknown internet structure to assessment


Previous research on DIF: Previous research on DIF No published research with DIF analysis


The Present Study: The Present Study Examined mean-level differences on the Big Five across testing medium Equivalent groups across modalities Paper-and-Pencil Traditional administration under experimenter supervision. Computer Laboratory Group administration of WWW questionnaires in laboratory under experimenter supervision. Internet E-mail notification of WWW questionnaire. Administered over WWW for 1 week.


The Present Study …: The Present Study … Random Assignment to Conditions No Self-selection effects Identification of Subjects Unique Password – security/integrity data Academic – course credit Industry – aberrant responses, linking information Multiple-access for Internet Questionnaire No Duplicate Responses


DIF Analyses: DIF Analyses SIBTEST (Stout andamp; Roussos, 1995) SIB Statistic β andamp; Mantel-Haenszel Iterlink DIF (Steve Stark) Lord’s Chi-Square


Big 5 Personality Scale: Big 5 Personality Scale Trait Descriptive Adjective (Goldberg) Neuroticism, Extroversion, Openness, Agreeableness, Conscientiousness 5 point Likert Scale (Strongly Disagree – Strongly Agree) 'How well do the adjectives describe you?'


Big 5 Personality Items: Big 5 Personality Items * Reverse Scored Items * Reverse Scored Items


Sample: Sample Condition N Paper-and-pencil 266 Computer Lab 222 Internet 240 Total 728 Undergraduates Psychology Students participating in the study for course credit


SIB Statistic p<.05: SIB Statistic pandlt;.05


Mantel-Haenszel p<.05: Mantel-Haenszel pandlt;.05


Lord’s Chi-Square p<.05: Lord’s Chi-Square pandlt;.05


Expected # of “False Alarms”: Expected # of 'False Alarms' 95% Confidence Interval 5% Comparisons ± (α X SD) α X SD = 1.96 X (30 X .05 X .95)½ = 2. 34 α X SD = 1.96 X (123 X .05 X .95)½ = 4.74 Per Dimension 2.05 ± 2.735 = (0, 4.39) All 4 Dimensions 6.15 ± 4.74 = (1.41, 10.89)


Summary Analysis p<.05: Summary Analysis pandlt;.05


SIB Statistic Bonferroni Corr.: SIB Statistic Bonferroni Corr.


Mantel-Haenszel Bonferroni : Mantel-Haenszel Bonferroni


Lord’s Chi-Square Bonferroni : Lord’s Chi-Square Bonferroni


Mean-Level Comparisons: Mean-Level Comparisons DIF analyses established that Personality Items are functioning equivalently across media Is there a shift in Trait Levels as a consequence of medium? mean-levels comparisons using ability/theta estimated by Bilog


Mean Trait Levels Effect Size: Mean Trait Levels Effect Size Paper-and-Pencil Vs Both Computer Administrations Emotional Stability d’ = .19 Emotional Stability Extroversion Agreeableness Conscientiousness pandlt;.033


Conclusion: Conclusion Personality Tests are Equivalent Features of Internet Questionnaire Important


Slide25: