200703 ISMJC 1

ISM Journal Club2007. 3. 6.: 

ISM Journal Club 2007. 3. 6. An Index to Quantify an Individual's Scientific Research Output Hirsch 2005, PNAS, 102, 16569 Measures for Measures Lehmann et al. 2006, Nature, 444, 1003 Review by Minho Choi

I.: 

I. Measures (Hirsch h)

Measuring Scientists: 

Measuring Scientists It is best to read papers and talk to each other. Then you will clearly know the quality of scientists. But, who has the time? Even if distasteful, quantification is needed in a world of limited resources. Recruitment, advancement, grants, etc. How to quantify the cumulative impact and relevance of an individual’s scientific research output? Available data Np: Number of papers n: Duration of publication (years) Nc: Number of citations for each paper Impact factors of journals

Single-number Criteria: 

Single-number Criteria Total number of papers: Np Publication frequency: Np/n Total number of citations: Sum(Nc) Citations per paper: Sum(Nc)/Np Number of significant papers: N(Nc>y) Hirsch’s h-index

Pros and Cons: 

Pros and Cons Total number of papers and Publication frequency Pro: Measures productivity/industry Con: Does not measure impact Total number of citations Pro: Measures total impact Con: Inflated by a small number of big hits Con: Undue weight to review articles

Pros and Cons: 

Pros and Cons Citation per paper Pro: Allows comparison of scientists of different ages Con: Rewards low productivity and penalizes high productivity Number of significant papers Pro: Gives an idea of broad impact Con: y (threshold) is arbitrary

Hirsch’s h-Index: 

Hirsch’s h-Index A scientist has index h if h of his/her papers have at least h citations each and the other papers have at most h citations each. Measures the broad impact of an individual’s work. (h increases monotonically.) Avoids the disadvantages of other criteria. Gives a ballpark estimate of the total number of citations. Sum(Nc) = a h2 h = m n Scientist with higher h is likely to be more accomplished.

Example: 

Example

More Examples: 

More Examples

Citation History of Papers: 

Citation History of Papers Citation count usually increases briefly and then decreases slowly.

Citation History of Papers: 

Citation History of Papers But not always…

Index h of Some Prominent Physicists: 

Index h of Some Prominent Physicists E. Witten 110 A. J. Heeger 107 M. L. Cohen 94 A. C. Gossard 94 P. W. Anderson 91 S. Weinberg 88 M. E. Fisher 88 M. Cardona 86 P. G. deGennes 79 J. N. Bahcall 77 Z. Fisk 75 D. J. Scalapino 75 G. Parisi 73 S. G. Louie 70 R. Jackiw 69 F. Wilczek 68 C. Vafa 66 M. B. Maple 66 D. J. Gross 66 M. S. Dresselhaus 62 S. W. Hawking 62

Typical h: 

Typical h Tenure (associate professor): 12 Advancement to full professor: 18 Fellowship in American Physical Society: 15-20 National Academy of Sciences of USA: 45 Nobel prize: 22 - 79 Nobel prizes do not originate in one stroke of luck but in a body of scientific work.

Hirsch Slope m: 

Hirsch Slope m Slope m is a useful yardstick to compare scientists of different seniority. m ~ 1: successful scientists m ~ 2: outstanding scientists m > 3: truly unique individuals. Some prominent physicists Witten m ~ 3.9 Weinberg m ~ 1.8 Bahcall m ~ 1.8 Hawking m ~ 1.6 But, m is useful only when Np is large.

Caveats: 

Caveats Differences in h in different fields Scientists in non-mainstream areas Though h is a reliable indicator of high accomplishment, the converse is not always true. A scientist with a few seminal papers would have low h. A scientist with many coauthors would have high h. Self-citations

II.: 

II. Quality Testing

Quantifying the Quality: 

Quantifying the Quality In general, It is better to publish more than less. Citation count is a useful measure of a paper’s quality. Citation measures Which one is the best? How reliable are they (statistically)?

Comparing Measures: 

Comparing Measures “Best” measure is that which minimizes uncertainty and maximizes discrimination. To compare measures, we need a homogeneous set of authors. SPIRES (high-energy physics) Probability that a paper will receive k citations is a power-law distribution. P(k) ~ k- ( ~ 2.8)

Probability of Citation Counts: 

Probability of Citation Counts P ~ (k+1)- For k<50,  = 1.10 For k>50,  = 2.78

How to Compare Measures: 

How to Compare Measures Sort and bin authors according to the chosen measure. For each bin, calculate the conditional probability that a paper by an author in the bin n will have k citations. Using the full citation records for all authors in the given bin. Calculate the average probability that an author initially assigned to bin n should instead be assigned to bin m. The m assignment is more reliable than the n assignment, because it is based on an author’s full citation record. Plot m vs n. A perfect measure would place all weight in the diagonal entries. Tight correlation means less uncertainty (higher accuracy).

The Winner is …: 

The Winner is … Publication frequency is barely better than the (meaningless) alphabetization. Hirsch h-index is better but the uncertainty is large, and there is a bias. Mean number of citation is superior, in both accuracy and precision. Median is even better.

Publication Frequency is BAD: 

Publication Frequency is BAD Publication frequency (e.g., average number of papers published by an author per year) has a large uncertainty and not useful. It measures industry rather than ability. Bad, but it is the most widely used measure! (including KASI) Introducing impact factor is unlikely to improve the situation. Impact factor of a journal is determined by a small number of highly cited papers.

So… Should We Use Mean Citation?: 

So… Should We Use Mean Citation? Need a large number of papers to make reliable assignments. (Np > 50!) IMHO: For lowly scientists like us, h may be better. Misuses Institutions have a misguided sense of the fairness of decisions reached by algorithm, and unable to measure what they want to maximize (quality). Institutions will maximize what they can measure.

In Practice…: 

In Practice… Citation-based measures are unusable in year-by-year assessments. Measures are meaningful only when the sample is large. Should wait for a long time to count citations. Citation history of papers can be very different. In the meantime… We must actually read the papers!

My Suggestions: 

My Suggestions Please read papers and talk to each other. The institute should be helping researchers to build a body of consistent work rather than hoping for a one-shot killer paper. Do not get misguided by small-number statistics. Yearly assessment (paycheck) should be separated from quality assessment (leadership). Quality measure should be introduced into the assessment process. (impact factor?)