Slide1: The potential and problems of peer evaluation in higher education and research
Hans-Dieter Daniel
Professorship for Social Psychology and Research on Higher Education, ETH Zurich Evaluation Office, University of Zurich
Slide2: “Peer review (known as refereeing in some academic fields) is a scholarly process used in the publication of manuscripts and in the awarding of funding for research. Publishers and funding agencies use peer review to select and to screen submissions. The process also forces authors to meet the standards of their discipline and thus achieve scientific objectivity. Publications and awards that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals in many fields”.
Source: Wikipedia, The Free Encyclopedia
Critics of peer review: Critics of peer review "Manuscript refereeing, one aspect of peer review and self-management in the sciences, has been shown to be almost wholly lacking in interreferee agreement on the recommendation to publish (r2 = .04), without validity in forecasting the subsequent usefulness of a work to scientists as reflected in citations of the work in other scientific papers (r2 = .00), and biased in more than a dozen ways." (Ross, 1980)
"Peer review fails miserably with respect to every technical criterion for establishing the reliability and validity of an assessment instrument." (Bornstein, 1991)
Reliability: Agreement of referees in journal peer review: Reliability: Agreement of referees in journal peer review
Fairness: Gender bias in grant peer review: Fairness: Gender bias in grant peer review Bias in favour of females Bias in favour of males
Predictive validity: Comparison of the citation rates for papers accepted and rejected by The Journal of Clinical Investigation but published elsewhere (source: Wilson, 1978): Predictive validity: Comparison of the citation rates for papers accepted and rejected by The Journal of Clinical Investigation but published elsewhere (source: Wilson, 1978)
Mean number of citations of articles previously published to approval or rejection by B.I.F. post-doctoral applicants: Mean number of citations of articles previously published to approval or rejection by B.I.F. post-doctoral applicants Notes. Application window: 1990-1995; publication window: 1986-1994; citation window: from year of publication to the end of 2001. We included only average numbers of citations per year that could be calculated with citation counts for more than ten articles (1986–1994); the years 1981 to 1985 and 1995 to 1996 could not be included.
Slide8: Folie Evaluationsverfahren Academic Departmental Review at the University of Zurich
University of Zurich: a member of LERU: University of Zurich: a member of LERU Members of the League are
Universiteit van Amsterdam Lunds universitet
University of Cambridge University of Milan
University of Edinburgh LMU Munich
Universität Freiburg University of Oxford
University of Geneva Université Pierre et Marie Curie, Paris 6
University of Heidelberg Université Paris-Sud 11
University of Helsinki Karolinska Institutet, Stockholm
Leiden University Université Louis Pasteur Strasbourg
Katholieke Universiteit Leuven Universiteit Utrecht
University College London Universität Zürich
Membership in the League of European Research Universities (LERU), which is by invitation, is periodically evaluated against a broad set of quantitative and qualitative criteria, such as research volume, impact and funding, strengths in PhD training, size and disciplinary breadth, and peer-recognised academic excellence.
Citation analysis in science and medicine of 40 major European universities, 1995-1999: Citation analysis in science and medicine of 40 major European universities, 1995-1999 Source: Centre for Science and Technology Studies (CWTS), Universiteit Leiden CPP: average number of Citations Per Publication, JCS: mean Journal Citation Score of the institute‘s ‘journal set‘, FCS: mean citation rate of all papers published in all journals of the field(s) in which the institute is active
Organizational Structure (extract): Independent Evaluation Office: Organizational Structure (extract): Independent Evaluation Office Evaluation Office Board of the University Minister of Education, and six personalities from science, culture, business, and politics Extended Executive Board Executive Board
President, Vice-Presidents, Managing Administration Director Senate Supervision / Strategic Management Executive Management
Evaluation Objectives: Evaluation Objectives Assess, assure, and improve the quality of academic work in research, teaching, and services as well as assure the quality of management and administration (referred to as “Full Review” at the University of Cambridge)
Provide decision aids to support medium and long-term strategic planning
Report to the public (accountability)
Evaluation Office: Evaluation Office Personnel
Director Prof. Dr. Hans-Dieter Daniel, ETH Zurich
Managing Director Dr. Thomas Rothenfluh
Project Managers Andrea Eichholzer, Judith Maag, Dr. des. Sandra Mittag, Vinciane Vouets, Peter Koch
Secretary Annette Häusermann
Projects
~113 evaluation projects within six years
Costs
0,14 % of the University’s budget (including third party funds)
(2004: env. 932 Million Swiss Francs)
First evaluation cycle: Project status: First evaluation cycle: Project status
Evaluation Process: Informed peer review: Surveys Students Alumni
Academic Staff
Personnel Customers Self-evaluation report Responses Unit under evaluation Evaluation Office Comprehen-sive Evaluation Report Responses Course participants Semester 1 Semester 2 Semester 4 Bibliometry Evaluation Process: Informed peer review Experts (Peers) Experts’ Report Site Visit Semester 3 Follow-up Board, Executive Board
Memorandum of understanding / Goal agreement
Self-Evaluation: Section Headings for Self-Evaluation and Guidance on Content: Self-Evaluation: Section Headings for Self-Evaluation and Guidance on Content Retrospective on last five years
History of the institute
Structure and organization
Financial resources: Budget and third-party funds
Human resources
Material resources (e.g., IT, library, equipment, spatial situation)
Management and administration
Teaching and study programs
Research
Promotion of young academics/scientists
Services
Internal quality assurance measures
Profile of strengths and weaknesses
Future perspectives
Brief portraits of the chairs, senior lecturers, research group leaders
Research (qualitative / quantitative); teaching; promotion of young academics / scientists; offices held / professional functions / services
Selection criteria for external reviewers : Selection criteria for external reviewers Each expert should have an outstanding international scientific reputation.
The composition of the team of experts as a whole should ensure that the following requirements can be met:
All core competencies of the unit under evaluation are covered
Experience in the area of structured promotion of young academics / scientists (participation in graduate schools, PhD programs, etc.)
Knowledge in the area of curriculum development
Evaluation experience
Experience in management / administration of sizeable university units (dean, vice president, assistant rector of a university)
Active professors (Emeritae / Emeriti only in exceptional cases)
Desirable:
Representation of men and women; good age mix
Familiarity with the Swiss university system
External Reviewers: Bias: External Reviewers: Bias Persons should not be selected as experts if there is any appearance of bias. Appearance of bias is given if a potential expert is in any way affected by participation in the evaluation or if there is any other reason that might render the impartiality of the evaluation suspect. The following can constitute such reasons:
Teaching contracts or visiting professorships at the unit under evaluation
Close scientific collaboration, e.g., joint projects or co-authorship of publications
Teacher / student relationship
Participation in ongoing or very recently concluded procedures for the appointment of professors in the environment surrounding the unit under evaluation
Simultaneous or previous participation in boards or committees advising the unit under evaluation (e.g., Scientific Advisory Board)
Personal or mutual economic interests; economic competition
Informed (Evidence-based) Peer Review: Informed (Evidence-based) Peer Review
Slide20: Lessons Learned from Departmental Reviews at the University of Zurich Strengths
Evaluations find acceptance. They forge identity in the units under evaluation.
They foster communication and transparency – within the university and with strategic and political authorities.
They uncover strengths and weaknesses (and the mere announcement of a pending evaluation can in part contribute to performance improvement).
They are indispensable for structure and development planning.
Weaknesses
Evaluation and implementing the results of the evaluation require a lot of work and are time-consuming for all participants.
If human and equipment resources are found to be insufficient, the funds required can not always be secured from the university or the state.
Meta-evaluation: Central Evaluation and Accreditation Agency Hannover (ZEvA) and Consortium of Universities in Northern Germany (Nordverbund) Overall assessment of the evaluation process by institute members and reviewers: Meta-evaluation: Central Evaluation and Accreditation Agency Hannover (ZEvA) and Consortium of Universities in Northern Germany (Nordverbund) Overall assessment of the evaluation process by institute members and reviewers
Meta-evaluation: Academic Advisory Council of Lower Saxony Quality assurance instruments for quality improvement of research and teaching (responses “very appropriate” and “appropriate”, in %): Meta-evaluation: Academic Advisory Council of Lower Saxony Quality assurance instruments for quality improvement of research and teaching (responses “very appropriate” and “appropriate”, in %) Source: Academic Advisory Council of Lower Saxony (unpublished meta-evaluation report)
Scenarios of future quality assurance in research (reviewers: n=54, faculty members: n=364; acceptance, in percent): Scenarios of future quality assurance in research (reviewers: n=54, faculty members: n=364; acceptance, in percent) Source: Academic Advisory Council of Lower Saxony (Wissenschaftliche Kommission Niedersachsen)
Publications: Publications Bornmann, L. & Daniel, H.-D. (2006, accepted for publication). Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics.
Bornmann, L., Mittag, S., & Daniel, H.-D. (2006, accepted for publication). Quality assurance in higher education - meta-evaluation of multi-stage evaluation procedures in Germany, Higher Education.
Bornmann, L. & Daniel, H.-D. (2005). Does the h-index for ranking of scientists really work? Scientometrics, 65(3), 391-392.
Bornmann, L. & Daniel, H.-D. (2005). Selection of research fellowship recipients by committee peer review. Analysis of reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics, 63(2), 297-320.
Bornmann, L. & Daniel, H.-D. (2005). Criteria used by a peer committee for selection of research fellows. A Boolean probit analysis. International Journal of Selection and Assessment, 13(4), 296-303.
Bornmann, L. & Daniel, H.-D. (2005). Committee peer review at an international research foundation. Predictive validity and fairness of selection decisions on post-graduate fellowship applications. Research Evaluation, 14(1), 15-20.
Daniel, H.-D. (2005). Publications as a measure of scientific advancement and of scientists' productivity. Learned Publishing, 18(2), 143–148.
Daniel, H.-D. (2004). Guardians of Science - Fairness and Reliability of Peer Review . Weinheim: Wiley-VCH. ISBN: 3-527-60220-8.