Kidney News May 2012, 4#5 : Page 10

10 | ASN Kidney News | May 2012 Kidney 2.0 Understanding Research Impact U By Kristi L. Holmes and Cathy C. Sarli nderstanding the true value of a scholar’s research and output is no small feat. Although it’s fairly straightforward to track the number of publications or total dollar amount of awarded funding, it can be a greater chal-lenge to assess the reach of scholarly eff orts and determine how others are utilizing the research results. Metrics for assessing research performance, quality, and impact cover a wide range of the scholarly ecosys-tem and are used for a variety of purposes: individual career planning, promotion, and tenure; benchmarking to track group or institutional performance; marketing and strategic planning purposes; and re-porting research outcomes to the public. No single metric is suffi cient for measur-ing performance, quality, or impact by an author; indeed, the discovery of a scholar’s most impactful work may be gleaned only through qualitative forms of assessment that do not rely solely on publication data. Journal-level metrics Journals are also assessed by diff erent crite-ria. Th e impact factor, listed in Th omson Reuters’ Journal Citation Reports, assigns journals a numeric score based on the fre-quency with which the average article in the journal is cited over a set period of time (3). Whereas the impact factor tracks straight citations, the Eigenfactor score (http:// www.eigenfactor.org) is derived from a for-mula based on citations from a journal over a 5-year period, with citations from highly ranked journals given more weight. Journal self-citations are not included in the Eigen-factor score, unlike the impact factor score. A caveat of note for journal-level met-rics: specialized journals or those published by societies may disseminate your work more effi ciently to colleagues in your fi eld than a “high-impact” general-interest sci-ence journal. Reaching your intended au-dience is the surest way to enhance the vis-ibility and impact of your research. Article-level metrics Citation counts are perhaps the most fre-quently used metric at the article level. A citation is a reference to a specifi c publi-cation. Th e inherent assumption is that signifi cant articles will have high numbers of citations. Further analysis is required to discover why select publications garner a higher citation rate than others. Many da-tabases provide tools for authors to track citations to their work, with some off ering citation maps that can be downloaded for reporting purposes. A growing article-level metric is based on the usage of a publication; several jour-nals and third-party service providers are making it possible to assess the Web-based use and subsequent dissemination of indi-vidual articles. Th e Public Library of Sci-ence (PLoS) journals off er perhaps the most highly developed publisher-based platform for this type of tracking. Articles published in PLoS journals include an article-level metrics tab that shows such details as arti-cle usage statistics (e.g., HTML page views, PDF and XML downloads, and accesses from PubMed Central; number of users via Mendeley; and number of Facebook mentions); citations from the scholarly lit-erature (currently from CrossRef, PubMed Central, SciVerse Scopus, and Web of Sci-ence); social bookmarks from CiteULike and Connotea; PLoS reader evaluation (i.e., readers’ feedback on the article in the form of comments, notes, and star ratings); and discussion of the article in blogs (2). Th ese alternative metrics (or “altmetrics” as they are commonly known) for articles and even datasets and presentations are becom-ing easier to track via Web services such as Total-Impact (http://total-impact.org) and Altmetric (http://www.postgenomic.com), who off ers explorer and browser-based bookmarklet applications. strategies for enhancing research impact. Th e site off ers reporting templates, a glos-sary, and examples of relevant indicators of impact across the research process as well as a sample of a completed report. Th e use of publication data in tandem with the Beck-er Model provides a more robust overview of the impact of research to accomplish a host of higher-order activities that are criti-cal in today’s biomedical research world, including the following: • Justifi cation of future requests for funding • Quantifi cation of return on research in-vestment • Discovery of how research fi ndings are being used • Promotion and tenure activities • Identifi cation of possible collaborators • Demonstration that research fi ndings are resulting in meaningful health outcomes • Discovery of community benefi t as a result of research fi ndings 1. Hirsch JE. An index to quantify an individual’s scientifi c research out-put. Proc Natl Acad Sci U S A 2005; 102:16569–16572. 2. Public Library of Science. Article-level metrics. http://article-level-metrics.plos.org. 3. Th omson Reuters. Th e Th omson Reu-ters impact factor. http://thomsonreuters. com/products_services/science/free/es-says/impact_factor. 4. Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Libr Assoc 2010; 98:17–23. 5. Bernard Becker Medical Library. As-sessing the impact of research. Th e Becker Medical Library Model for as-sessment of research impact. https:// becker.wustl.edu/impact/assessment/ index.html. Kristi L. Holmes and Cathy C. Sarli are af-fi liated with the Bernard Becker Medical Li-brary, School of Medicine, Washington Uni-versity, St. Louis. Author-level metrics Author-level metrics allow individuals to track their scholarly output and serve as a refl ection of a researcher’s productivity. Enumerating such things as the number of publications in the scholarly literature, number of books published, and number and amount of funding awards can all serve to understand the eff orts of individual re-searchers. One commonly used metric is the Hirsch index (h index). Th e h index, developed by Jorge E. Hirsch, PhD, in 2005 (1), off ers a numeric index to meas-ure the productivity and impact of a given researcher. Th e h index is a quantitative metric based on analysis of publication data, using publications and citations to provide “an estimate of the importance, signifi cance, and broad impact of a scien-tist’s cumulative research contributions” (1). According to Hirsch, the h index is defi ned as follows: “A scientist has index h if h of his or her N p papers have at least h citations each and the other ( Np – h ) pa-pers have ≤ h citations each” (1), where Np is total number of papers published. As an example, an h index of 10 means that among all publications by one author, 10 of these publications have received at least 10 citations each. Th e h index is but one metric for author-level assessment. Going beyond the metrics: the Becker model It is tempting to use these metrics as an objective way to assign value or worth to a researcher’s output or to an individual jour-nal. Although these metrics can be helpful in understanding research eff orts, they can-not be evaluated in a vacuum. To under-stand the true impact of research, metrics derived from publication data must be supplemented with indicators that dem-onstrate tangible outcomes such as clinical implementation, benefi t to the commu-nity, infl uence on legislation or policy, and economic benefi t. Publication data alone do not provide a full narrative of research impact, nor are they predictive of meaning-ful health outcomes. Th e Becker Medical Library Model for Assessment of Research Impact (4,5) serves as a framework to quantify and document research impact based on research outputs and activities. It includes resources for lo-cating evidence of research impact and ASN Career Center Looking for that perfect ﬁ t? Post your resume online. Whether or not you’re actively seeking work, posting your resume with ASN provides you access to the best job offers in kidney medicine and research. Access the newest jobs available, those at the institutions and locations that most interest you, and create job alerts so you never miss a matching job opportunity. Get started today. Member Beneﬁ ts | The ASN Advantage careers.asn-online.org Find the right job faster with the

Understanding Research Impact

Kristi L. Holmes And Cathy C. Sarli

Understanding the true value of a scholar’s research and output is no small feat. Although it’s fairly straightforward to track the number of publications or total dollar amount of awarded funding, it can be a greater challenge to assess the reach of scholarly efforts and determine how others are utilizing the research results. Metrics for assessing research performance, quality, and impact cover a wide range of the scholarly ecosystem and are used for a variety of purposes: individual career planning, promotion, and tenure; benchmarking to track group or institutional performance; marketing and strategic planning purposes; and reporting research outcomes to the public. Author-level metrics Author-level metrics allow individuals to track their scholarly output and serve as a reflection of a researcher's productivity. Enumerating such things as the number of publications in the scholarly literature, number of books published, and number and amount of funding awards can all serve to understand the efforts of individual researchers. One commonly used metric is the Hirsch index (h index). The hindex, developed by Jorge E. Hirsch, PhD, in 2005 (1), offers a numeric index to measure the productivity and impact of a given researcher. The hindex is a quantitative metric based on analysis of publication data, using publications and citations to provide an estimate of the importance, significance, and broad impact of a scientist's cumulative research contributions" (1).According to Hirsch, the h index is defined as follows: "A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np.H) papers have .h citations each" (1), where Np is total number of papers published. As an example, an h index of 10 means that among all publications by one author, 10 of these publications have received at least 10 citations each. The h index is but one metric for author-level assessment. No single metric is sufficient for measuring performance, quality, or impact by an author; indeed, the discovery of a scholar’s most impactful work may be gleaned only through qualitative forms of assessment that do not rely solely on publication data. Article-level metrics Citation counts are perhaps the most frequently used metric at the article level. A citation is a reference to a specific publication. The inherent assumption is that significant articles will have high numbers of citations. Further analysis is required to discover why select publications garner a higher citation rate than others. Many databases provide tools for authors to track citations to their work, with some offering citation maps that can be downloaded for reporting purposes. A growing article-level metric is based on the usage of a publication; several journals and third-party service providers are making it possible to assess the Web-based use and subsequent dissemination of individual articles. The Public Library of Science (PLoS) journals off er perhaps the most highly developed publisher-based platform for this type of tracking. Articles published in PloS journals include an article-level metrics tab that shows such details as article usage statistics (e.g., HTML page views, PDF and XML downloads, and accesses from PubMed Central; number of users via Mendeley; and number of Facebook mentions); citations from the scholarly literature (currently from CrossRef, PubMed Central, SciVerse Scopus, and Web of Science); social bookmarks from CiteULike and Connotea; PloS reader evaluatio (i.e., readers’ feedback on the article in the form of comments, notes, and star ratings); and discussion of the article in blogs (2). These alternative metrics (or “altmetrics” as they are commonly known) for articles and even datasets and presentations are becoming easier to track via Web services such as Total-Impact (http://total-impact.org) and Altmetric (http://www.postgenomic.com), who offers explorer and browser-based bookmarklet applications. Journal-level metrics Journals are also assessed by different criteria. The impact factor, listed in Thomson Reuters’ Journal Citation Reports, assigns journals a numeric score based on the frequency with which the average article in the journal is cited over a set period of time (3). Whereas the impact factor tracks straight citations, the Eigenfactor score (http://www.eigenfactor.org) is derived from a formula based on citations from a journal over a 5-year period, with citations from highly ranked journals given more weight. Journal self-citations are not included in the Eigenfactor score, unlike the impact factor score. A caveat of note for journal-level metrics: specialized journals or those published by societies may disseminate your work more efficiently to colleagues in your field than a “high-impact” general-interest science journal. Reaching your intended audience is the surest way to enhance the visibility and impact of your research. Going beyond the metrics: the Becker model It is tempting to use these metrics as an objective way to assign value or worth to a researcher’s output or to an individual journal. Although these metrics can be helpful in understanding research efforts, they cannot be evaluated in a vacuum. Tounderstand the true impact of research, metrics derived from publication data must be supplemented with indicators that demonstrate tangible outcomes such as clinical implementation, benefit to the community, influence on legislation or policy, and economic benefit. Publication data alone do not provide a full narrative of research impact, nor are they predictive of meaningful health outcomes. The Becker Medical Library Model for Assessment of Research Impact (4,5) serves as a framework to quantify and document research impact based on research outputs and activities. It includes resources for locating evidence of research impact and strategies for enhancing research impact. The site offers reporting templates, a glossary, and examples of relevant indicators of impact across the research process as well as a sample of a completed report. The use of publication data in tandem with the Becker Model provides a more robust overview of the impact of research to accomplish a host of higher-order activities that are critical in today’s biomedical research world, including the following: • Justification of future requests for funding • Quantification of return on research investment • Discovery of how research findings are being used • Promotion and tenure activities • Identification of possible collaborators • Demonstration that research findings are resulting in meaningful health outcomes • Discovery of community benefit as a result of research findings 1. Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A 2005; 102:16569–16572. 2. Public Library of Science. Articlelevel metrics. Http://article-levelmetrics. Plos.org. 3. Thomson Reuters. The Thomson Reuters impact factor. Http://thomsonreuters. Com/products_services/science/free/essays/ impact_factor. 4. Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Libr Assoc 2010; 98:17–23. 5. Bernard Becker Medical Library. Assessing the impact of research. The Becker Medical Library Model for assessment of research impact. Https:// becker.wustl.edu/impact/assessment/index.html. Kristi L. Holmes and Cathy C. Sarli are affiliated with the Bernard Becker Medical Library, School of Medicine, Washington University, St. Louis.

Previous Page Next Page

Publication List
Using a screen reader? Click Here