EvaluatingJobApplica nts093004

Evaluating Job Applications and Letters of Recommendation : Evaluating Job Applications and Letters of Recommendation The University of Montana PArtnership for Comprehensive Equity PACE@mso.umt.edu (406) 243-PACE Diana I. Lurie Project PACE


Reviewing ApplicationsAdapted from “Reviewing Applicants: Research on Bias and Assumptions The University of Wisconsin, Madison : Reviewing Applications Adapted from “Reviewing Applicants: Research on Bias and Assumptions The University of Wisconsin, Madison Gender of the evaluator: not significant Both men and women apply and share the same assumptions about gender. Statistical discrimination Biases that can influence the evaluation of applications Tips for reviewing applications


Statistical Discrimination : Statistical Discrimination The application of generalizations that may or may not be accurate to the evaluation of individuals Evaluators overestimate the heights of male subjects and underestimate the heights of female subjects when shown photographs of people of the same height that contain a reference point, i.e. a doorway (Biernet et al, 1992). Evaluators rated the athletic ability of African American men higher than that of Caucasian men when shown photographs of men with similar athletic abilities (Biernet et al, 1992).


Biases that can influence the evaluation of applications : Biases that can influence the evaluation of applications Busy evaluators who were distracted by other tasks and under time constraints gave women lower ratings than men for the same written evaluation of job performance (Martell, 1991) This bias decreased when evaluators gave all their time and attention to these applications. Take Home Message: evaluators are more likely to rely on underlying assumptions and biases when they do not have sufficient time to devote to evaluations.


Biases that can influence the evaluation of applications : Biases that can influence the evaluation of applications Perceived discrepancies between the female gender role and leadership roles leads to attitudes that are less positive toward female leaders compared to male leaders (Eagly and Karau, 2002; Ridgeway, 2001). Female leaders can be viewed less positively than male leaders.


Biases that can influence your search : Biases that can influence your search The work, research, and ideas of women or minorities may be undervalued or unfairly attributed to a mentor or collaborator despite evidence to the contrary. Biases from letters of recommendation and attribution of success to luck vs. skill. Assumptions about potential family commitments and their impact on the candidate’s career may negatively influence evaluation of candidate. Statistical discrimination The ability of females or minorities to run a research group and successfully supervise students and staff may be underestimated. Assumptions regarding leadership abilities.


Tips for Reviewing Applications : Tips for Reviewing Applications Consistently apply all criteria to all candidates. Spend sufficient time evaluating each applicant (15-20 minutes). Don’t depend too heavily on only one element of the application (i.e. letters of recommendation or prestige of university). Clearly define every decision you make for rejecting or retaining a candidate.


Evaluating Letters of RecommendationAdapted from Trix and Psendka (2003) “Exploring the colors of glass: letters of recommendation for female and male medical faculty” Discourse and Society, 14(2): 191-220.  : Evaluating Letters of Recommendation Adapted from Trix and Psendka (2003) “Exploring the colors of glass: letters of recommendation for female and male medical faculty” Discourse and Society, 14(2): 191-220. Evaluation of over 300 letters of recommendation for positions at a major medical university Length of letters Letters of minimal assurance Doubt raisers Grindstone adjectives Repetition Possessive phrases Tips for reviewing letters of recommendation


Study Design : Study Design Over 300 letters of recommendation for medical faculty at a large American medical school in the mid-1990’s were evaluated. Authors received a request by a member of the Executive Committee for Hiring and Promotion of the medical school to determine if letters of recommendation written for female applicants were systematically different to those written for male applicants.


Data Collection : Data Collection Data for study were a naturalistic set of all letters of recommendation for successful applicants for faculty positions at a large American medical school from 1992-1995. 312 letters for applicants for 103 faculty positions were included: approximately 3 letters per applicant. Only letters of successful applicants were available for study. Positions were both clinical and research at the assistant professor level but included: Adjunct positions 8 associate professorships 1 full professorship.


Medical Specialties : Medical Specialties 37 different specialties including: Surgery Oncology Neurology Internal medicine Ob-gyn Urology Radiology Pathology Psychiatry Pediatrics Family medicine Anatomy Basic science


Applicants : Applicants 89 (29%) letters were for women. 222 (71%) of the letters were for men. One letter was for a couple.


Letter Writers : Letter Writers 85% male, 12% female, 3% unknown (foreign names that could not be easily categorized by the office that blacked out the names. US letters (local geographic area comprised of 3 adjoining states): 215 US (not local): 79 Canada, Britain, Australia, South Africa, Israel, Europe etc: 17 In general, letters from Europe tended to be shorter. Gatekeepers (to whom the letters of recommendation were addressed) were 96% male and 4% female. In general, letters were written from male physicians to male physicians. 13% of all letters addressed the gatekeepers by their first name. This reinforced the in-group nature of the letters.


General Findings : General Findings Length Average length of the letters was 246 words. Females: 227 words Males: 253 words Letters for males and females similar at the mid-range of length (11-50 lines). 8% of letters for men were over 50 lines long. 2% of letters for females were over 50 lines long. 10% of letters for women were 10 lines or shorter. 6% of letter for men were 10 lines or shorter.


Letters of minimal assurance : Letters of minimal assurance Letters of recommendation should include: Commitment and relationship of recommender with the applicant Some specificity of focus and record of the applicant Some evaluation or comparison of traits and accomplishments of the applicant Where at least one of these three areas are not represented, letters considered to be letters of minimal assurance rather than letters of recommendation.


Letters of minimal assurance : Letters of minimal assurance Dear Dr. Alfred Koop: It gives me great pleasure in writing this recommendation letter for Dr Sarah Gray. I have known Sarah as a resident and as staff of Mrahonod Metropolitan Hospital. She is knowledgeable, pleasant, and easy to get along with. I have no hesitation in recommending her for a faculty position at Missoula County Hospital. I will be happy to answer any further questions in this regard. Charles Lewis, MD Chairman, Dept. of Psychiatry Letter shows commitment and relationship of recommender with applicant, but no specificity of focus or record, and no evaluation of traits or accomplishments.


Letters of Minimal Assurance : Letters of Minimal Assurance 15% of the letters for female applicants 6% of the letters for male applicants Statistically significant (p-value of .021)


Short letter of recommendation : Short letter of recommendation Re: Appointment of Sarah Gray, MD: I am pleased to recommend Dr Sarah Gray for faculty appointment as Clinical Assistant Professor. I have known Dr. Gray for 8 years. She worked in research with me for 1 year and did fellowship training in our program for 2 years. She is a very good internist and endocrinologist. She is honest and reliable and of highest moral quality. She has good judgment in patient care and is very thoughtful and considerate towards those she serves. She is a good clinical teacher and should serve the department well in the capacity of instructing students and residents. From: Charles Lewis, MD, Director, Endocrine and Hypertension division. All the features of commitment of recommender, specificity and evaluative comments regarding the applicant are included. Applicant is described as an active person in her own right.


Doubt Raisers : Doubt Raisers Negative language While Sarah has not done a lot of bench research, She has a somewhat challenging personality Hedges It appears that her health and personal life are stable Potentially Negative As an independent worker she requires only the minimum of supervision Unexplained Now that she has chosen to leave the laboratory Faint Praise She worked hard on the projects that she accepted. Irrelevancy She is quite close to my wife Although his publications are not numerous While not the best student I have had, He appears to be a highly motivated colleague Bright,enthusiastic, he responds well to feedback I have every confidence that Bill will become better than average. He is very active in church


Doubt Raisers : Doubt Raisers 24% of letters written for females had at least 1 doubt raiser. 12% of letters written for males had at least 1 doubt raiser. Statistically significant with p-value p=0.01. Average of 1.7 doubt raisers per letter for females. Average of 1.3 doubt raisers per letter for males.


Grindstone Adjectives : Grindstone Adjectives “She is an extremely conscientious and meticulous researcher who devotes her time to laboratory work and the training of graduate students…..” “She is a superb experimentalist-very well organized, thorough and careful in her approach to research.” “I have found William to be hard-working, thorough, and conscientious in providing all aspects of patient care.”


Grindstone Adjectives : Grindstone Adjectives Nothing wrong with these qualities. Relates effort with ability in professional areas Implies that women are hardworking because they must compensate for a lack of ability. 34% of letters for female applicants included grindstone adjectives. 23% of letters for male applicants included grindstone adjectives.


Repetition : Repetition Standout adjectives Excellent Superb Outstanding Unique Exceptional Unparalleled Letters for women that had at least one of these terms had an average of 1.5 terms Letters for men that had at least one of these terms had an average of 2.0 terms. Standout adjectives are repeated in men’s letters to a greater extent.


Repetition : Repetition “Research”- Highly charged and desirable in an academic setting. 35% of letters for women that mentioned research at least once mentioned it multiple times. 62% of letters for men that mentioned research at least once mentioned it multiple times.


Possessive phrases : Possessive phrases Most common semantic categories of objects of possessive phrases for women: Her training Her teaching Her application Most common semantic categories of objects of possessive phrases for men: His research His skills and abilities His career By this measure, women are portrayed more as students and teachers, while men are portrayed more as researchers and professionals.


Conclusion : Conclusion Higher percentage of letters for women are short Higher percentage of letters for men are very long. Letters for women are lacking all the basic features to a greater degree. Letters for women include doubt raisers at double the rate for men. Greater frequency of reference to “research” in letters for men. Differences in semantic grouping, i.e. “her” teaching and “his” research.


Tips for Reviewing Letters of Recommendation : Tips for Reviewing Letters of Recommendation The length of the letter does not always indicate whether the letter is generally positive or negative. Check letters for basic features; letters for women tend to lack all the basic features to a greater degree. Be aware that letters for women tend to include twice as many doubt raisers as those for men. Be alert for grindstone adjectives and their frequency; letters for women tend to have more grindstone adjectives. Semantic groupings tends to portray female applicants more as students and teachers while men tend to be portrayed more as researchers and professionals. TAKE THE TIME TO READ LETTERS CAREFULLY.


Project PACE functions as a catalyst to facilitate achievement of faculty and administration gender diversity goals in the sciences at The University of Montana Project PACE337 & 340 North Corbin Phone: (406) 243-PACE (7223)PACE@mso.umt.edu : Project PACE functions as a catalyst to facilitate achievement of faculty and administration gender diversity goals in the sciences at The University of Montana Project PACE 337 & 340 North Corbin Phone: (406) 243-PACE (7223) PACE@mso.umt.edu