Monday, September 17, 2012

A New Way to Measure a Scientist's Impact

Publish or perish. Most scientists know this mantra well. Major academic career milestones, such as earning a postdoctoral position, tenure, or a major grant, depend heavily on a physicist's ability to produce highly-cited research. But are they being evaluated fairly?

Perhaps the simplest way to measure a scientist's research output is the number of citations her papers receive. If more scientists are talking about your research, you'd naturally conclude that it's having a greater impact.

Other factors, however, complicate the matter. Sometimes, for instance, one lucky undergraduate researcher will co-author an influential paper with hundreds of collaborators. If the paper receives significant attention, the undergraduate's citation count will soar. Is that a fair representation of the student's likely modest contribution? Probably not.

To combat factors such as the aforementioned "shot noise" problem, physicist Jorge Hirsch developed a new method of measuring research impact in 2005. The paper that unveiled this widely regarded method — called the h-index or Hirsch index — has well over 2,000 citations. By rudimentary standards, it has certainly made an impact. Nonetheless, more challenges have lingered.

Image courtesy futureatlas.com via flickr.


That's why information scientists Alberto Pepe (Harvard) and Frak Kurtz (Smithsonian Observatory) have recently developed a newer, more refined way to measure scientific impact: the tori index. So why might this measure more accurately reflect a researcher's impact?

The h-index, while popular and useful, neglects two important facts: the variability of citations across different fields and self-citations. Tori, which stands for Total Research Impact, largely corrects both of these problems.

For the self-citation problem, the tori index simply doesn't count citations that include one of the original authors. Simple enough.

Furthermore, your tori value decreases as the person who cites you uses more references (citations of other research papers) in their article. This evens the playing field across sub-disciplines in physics because some fields simply use more citations in their papers. The Hirsch index can't compete here.

Physicists studying particles, for instance, generally receive many more citations than scientists working on cosmic rays. The tori value accounts for these two fields' citation practices, enabling more accurate cross-field comparisons.

This correction also allows for comparisons of researchers from different historical periods with different citation practices. Because citations have significantly increased over the past few decades, the popular h-index may not allow for accurate comparisons of a 1960's physicist vs. a modern day physicist, for example. Tori, however, largely eliminates this bias as seen in the plots below.

Old Measure

The age-adjusted h-index (called the m-index) plotted against the start of a researcher's career. Clearly, researchers who started working later in the past century had higher index numbers, but they may not have necessarily made more of an impact.

New Measure

The age-adjusted tori index (called riq) plotted against a researcher's first publication. As you can see, the tori index better accounts for historical trends, flattening out the line over time. In this case, a flatter fit line represents a more fair representation across historical periods.

If you're a researcher, you can even find your tori and riq numbers online. Simply go to Harvard/NASA's Astrophysics Data System. Type in your name; select all of your articles; then click "Get metrics summary for articles," at the bottom of the screen.

I looked up my scores (based on two undergraduate, co-authored research papers), and my index values are rightfully low. Apparently, I've got a lot of work to do if I want to re-start my research career.

For more information, see the relevant arXiv article by Alberto Pepe and Michael Kurtz.

-------------------------------------------------------------------------------------

If you want to keep up with Hyperspace, AKA Brian, you can follow him on Twitter.

No comments:

Post a Comment