Camara_Mattern

Lessons Learned from Industry: Achieving Diversity & Efficacy in College Success : 

Lessons Learned from Industry: Achieving Diversity & Efficacy in College Success ETS - College Board Invitational Conference Washington, DC Wayne Camara & Krista Mattern September 8, 2008

Job Analysis : 

Job Analysis Organizations use job analysis to determine what work outcomes are desired. Sample individual outcomes (productivity, job performance, retention) and organizational outcomes (efficiency, quality, innovation, team work) Identify performance components (pc) pc = {Declarative knowledge x Procedural knowledge x Motivation} Knowledge x Cognitive skills x Level of effort Goals Interpersonal “ Persistence “ Ability, Interests, Education, Experience Importance & Prob. Of Outcomes Predictors

Job Analysis – College Success : 

Job Analysis – College Success Identify desired performance outcomes for individuals and organizations (college success) (GPA, return, graduate, life after college – grad school, certification) Each outcome likely has different predictors Identify performance tasks associated with outcomes (persistence, academic ability, health, engagement) Identify or develop performance measures (GPA, advisory ratings, self report data, dB of student engagement)

Predicting Performance : 

Predicting Performance Goldstein, Zedeck, & Schneider (1993)

Slide 5: 

Group differences are not unique to tests: They are present across most educational measures

Large Mean Differences Persist on Cognitive Ability Tests by Race/Ethnicity Remain : 

Large Mean Differences Persist on Cognitive Ability Tests by Race/Ethnicity Remain SAT CR+M

Slide 7: 

Source: U.S. Census Bureau College-Going Rates of High School Graduates Aged 18 to 24 by Ethnic Group, 1999-2006

Disparities Exist in HS Graduation, HS Drop Out and College Ready : 

Disparities Exist in HS Graduation, HS Drop Out and College Ready Source: Manhattan Institute, Public HS Graduation and College-Readiness Rates: 1991-2002, http://Manhattan-institute.org/html/ewp_08.htm;* Condition of Education, 2007 Table 23-2

Graduation Rates in 2004 by ethnicity : 

Graduation Rates in 2004 by ethnicity Published 3/7/2007 Title Awards conferred by Title IV institutions, by race/ethnicity, level of award, and gender: United States, academic year 2004–05 (recalculated to eliminate students who with other or no ethnicity reported). http://nces.ed.gov/ipeds/factsheets/pdf/fct_awards_conferred_03072007_5.pdf; Public HS graduation rates: WICHE 3/2008, http://www.wiche.edu/policy/knocking/1992-2022/index.asp

Rationale for looking beyond Grades and Tests : 

Rationale for looking beyond Grades and Tests What is college success? Is it more than grades and GPA? (Camara & Kimmel) Develop measures that predict your goal or desired outcome. Employers test multiple measures: openness, conscientiousness, extraversion, agreeableness, neuroticism Military use today (GED). Can do does not equal will do.

Slide 12: 

Predictors of College Success Tests Measure Colleges Collect in some form (applications, transcripts) Not Collected in Standardized form

Research collaboration with Michigan State University : 

Identify a broader domain of college student performance: Review university mission statements and department objectives Interview with university staff responsible for student life Review of the education literature on student outcomes Our systematic search (A JOB ANALYSIS OF UNDERGRADUATE STUDENTS) resulted in 12 dimensions of student performance… Validate items with successful juniors – they are the experts. Research collaboration with Michigan State University

12 Dimensions of Student Performance : 

12 Dimensions of Student Performance Broadening the Performance Domain in the Prediction of Academic Success (Schmitt, Oswald, & Gillespie, 2004) Knowledge, learning, mastery of general principles Continuous learning, intellectual interest and curiosity Artistic and cultural appreciation Multicultural appreciation Leadership Interpersonal skills Social responsibility, citizenship and involvement Physical and psychological health Career orientation Adaptability and life skills Perseverance Ethics and integrity

Two “Noncognitive” Measures : 

Two “Noncognitive” Measures Situational judgment inventory A situation is presented along with several alternative courses of action. The respondent is asked to indicate what she/he would be most likely and least likely to do. Biodata Short, multiple choice reports of past experience/background and interests/preferences.

Sample SJI Item for Leadership : 

Sample SJI Item for Leadership You are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything. -1 Look at them until someone eventually says something Start the conversation yourself by introducing yourself +1 Get to know everyone first and see what they are thinking about the project to make sure the project’s goals are clear to everyone Try to start working on the project by asking everyone’s opinion about the nature of the project You would take the leadership role by assigning people to do things or ask questions to get things rolling

Sample Biodata Items for Leadership : 

Sample Biodata Items for Leadership The number of high school clubs and organized activities (such as band, sports, newspapers, etc.) in which I took a leadership role was: 4 or more 3 2 1 I did not take a leadership role How often do you talk your friends into doing what you want to do during the evening? most of the time sometimes (about half the time) occasionally (about as often as others in my group seldom or infrequently never

Study 1: Develop and refine the measures : 

Study 1: Develop and refine the measures 644 MSU freshmen completed one of the two parallel forms of the biodata and SJI instruments at the beginning of the academic year. Results indicated significant incremental validity for some of the scales above and beyond the validity of SAT/ACT scores and existing measures of personality in predicting college GPA. The biodata and SJI demonstrated the greatest incremental validity when absenteeism, students’ self ratings, and peer-ratings of performance were examined ( .19, .22, and .14, respectively).

Study 2 Examine Validity & Subgroup Differences: 10 Participating Institutions & 2,700 Freshmen : 

Study 2 Examine Validity & Subgroup Differences: 10 Participating Institutions & 2,700 Freshmen HBCU N Winston-Salem (public) 229 Spelman College (private) 254 Big Ten (public) N University of Iowa 335 Michigan State University 546 Ohio State University 304 University of Michigan 297 Indiana University 170 Other Institutions N University of Chicago (private) 168 Cal State – Fullerton (public) 223 Virginia Tech (public) 237

Predicting FYGPA: Total Sample across 10 Institutions (N = 2443) : 

Predicting FYGPA: Total Sample across 10 Institutions (N = 2443) Non cognitive measures contribute little beyond tests and grades in predicting academic outcomes

Predicting Class Absenteeism: Total Sample across 10 Institutions (N = 899) : 

Predicting Class Absenteeism: Total Sample across 10 Institutions (N = 899) However, non cognitive measures will predict non cognitive outcomes – better than tests or grades (graduation, attendance, leadership, engagement)

Percent of Students Selected:Two Composites and Three Selection Strategies : 

Percent of Students Selected:Two Composites and Three Selection Strategies Top 85% Top 50% Top 15% Group AB AB+ AB AB+ AB AB+ Hispanic 4.4  4.6 4.1  4.9 3.9  5.5 (+.2) (+.8) (+1.6) Asian 7.6  7.7 9.9  9.5 17.5  12.9 (+.1) (-.4) (-4.6) African-American 17.9  19.8 9.6  13.6 1.3  7.2 (+1.9) (+4.0) (+5.9) White 70.2  67.9 76.4  71.9 77.2  74.4 (-2.3) (-4.5) (-2.8) AB = equally weighted composite of HSGPA and SAT/ACT. AB+ = equally weighted composite of HSGPA, SAT/ACT, Biodata, and SJI. Less selective Moderately selective Very selective

Correlations of Non-cognitive Measures with Cumulative GPA and Graduation : 

Correlations of Non-cognitive Measures with Cumulative GPA and Graduation Note. Bold values are significant at p< .01. N ranges from 1560 to 1798 across variables. Graduation is dichotomously scored (1, 0).

Study 3: Purpose & Research Questions : 

Study 3: Purpose & Research Questions 15 institutions (n = 4,164 for SJI and 7,645 for biodata) Purpose: evaluating the utility of the biodata and situational judgment measures in as close to a real admissions situation as is possible Administer new measures to college applicants rather than college freshmen. On an annual basis, collect class absenteeism, self rated performance of the noncognitve dimensions, and commitment to the university from enrolled students; institutions will provide course grades and retention.

Incremental Validity of Biodata Measures : 

Incremental Validity of Biodata Measures To preserve N in these regressions, the SJI was not included because of a relatively low response rate to this measure. It is worth noting that small sample sizes, such as those observed in these analyses, can seriously limit the ability to detect significant relationships due to decreased statistical power.

Next Steps : 

Next Steps In need of a demonstration project – Implement with Research across a few colleges! Encourage applicants to complete on-line as part of admissions and only use data as a “plus factor.” Provide incentives for applicants to complete the new measures and institutions to track student success over time. Likely outcomes will be more diversity, broader talent, greater retention, and standardized – defensible measures to evaluate applicants fairly and objectively. Increased efficiency and judgmental decisions based on data and comparability. For more information, go to http://www/iopsych.msu/cbstudy