Evaluating the individual researcher – adding an altmetric perspective

Introduction

ACUMEN was an EU funded research project aimed at “understanding the ways in which researchers are evaluated by their peers and by institutions, and at assessing how the science system can be improved and enhanced” (1). This project was formed to answer an FP7 call that requested “studying and proposing alternative and broader ways of measuring the productivity and performance of individual researchers including new and improved bibliometric indicators and evaluation criteria for research careers, project evaluations, and scientific publications” (2). FP7 was the Seventh Framework Program of the European Union for the funding of research and technological development in Europe. The ACUMEN Consortium was comprised of nine institutions. The main outputs of the project are the ACUMEN portfolio and the Guidelines for Good Evaluation Practices (both available here).

In the following article we will provide a brief introduction to the portfolio concept and then concentrate on how altmetrics are utilized in the portfolio.

The ACUMEN Portfolio

The ACUMEN portfolio allows the researcher to present herself through a brief narrative in which she highlights her past achievements and future goals. This narrative is backed up by structured information available in the sub-portfolios: the expertise, the output and the influence sub-portfolios. For each factor in the sub-portfolios evidence is provided to support the claims. For example, if the researcher claims to have specific methodological expertise, he backs up this claim by providing references to works where this method was applied.

A more detailed description of the three sub-portfolios:

• In the expertise sub-portfolio there are factors for scientific/scholarly expertise, technological expertise, teaching expertise, knowledge transfer, communication skills and organizational expertise.
• The output sub-portfolio is comprised of factors for scholarly outputs, teaching outputs, outputs communicated to the general public including online presence and online contributions, datasets, software and tools created by the researcher, patents and grants received.
• The influence sub-portfolio provides information on citations and various citation-based indicators, scholarly prizes, prizes for teaching, membership in program committees and editorial boards, invited talks, advice given based on subject expertise, economic influence in terms of income, spin-offs, consultancies and patents, textbook sales, download counts of publications and datasets, followers on various social media platforms, Mendeley readership counts, tweets and blog posts about the researcher’s work, views of online presentations, online syllabi mentions and popular articles written about the portfolio owner.

Thus the portfolio provides a holistic view of the researcher’s achievements, expertise and influence. Most of the factors have detailed sub-factors, and information that the portfolio owner is interested in conveying that does not match any of the above-mentioned factors can be provided in the “other” factor of each of the sub-portfolios. Since time spent in academia is crucial for fair evaluations, the ACUMEN project introduced the “academic age”, which is the time from the date the PhD was awarded with allowances for having children, illness and part-time work.

As said above, for each factor/sub-factor evidence is provided to back up the claims. The evidence is not everything that can possibly be listed, but only the “best” evidence for each factor and not more than three items. “Best” is subjectively decided by the researcher creating the portfolio. “Best” is for the specific factor; for example, in the output sub-portfolio the portfolio owner is requested to list his top three journal papers and in the influence sub-portfolio his top three most cited papers. It is possible that a different set of papers is provided for the two factors, in case he considers one of his recent works which has not accrued citations yet to be among his best works, or if he considers one of his less cited works to be among his best contributions.

Altmetrics in the Portfolio

As can be seen from the description of the sub-portfolios, online and social media presence and altmetrics are well represented. In the portfolio, online presence is viewed as an output; the researcher is asked to list accounts in social media used for academic purposes, academic network accounts, digital repository accounts and websites that were created or used for dissemination. These include academic social media sites such as ResearchGate and Academia.edu, sites where research outputs can be published such as SlideShare, figshare, YouTube or Vimeo, and also blogs and Twitter accounts. She is also asked to indicate her activity level (e.g. average number of posts per year or month) on these sites.

Altmetrics are even more emphasized in the influence sub-portfolio. The researcher is asked for the number of followers on social media sites, where scholarly information is published or discussed. Examples of such sites are academia.edu, ResearchGate, Twitter and blog(s) maintained by the portfolio owner. The guidelines for filling in the portfolio explain that these numbers should only be provided if viewed substantial.

The researcher is asked to provide details of a maximum of three articles that were tweeted or reviewed in blogs. It was shown recently (3) that articles that are reviewed in science blog posts close to their publication date have a good chance of being cited within three years, and receive more citations than the median number of citations for articles published in the given journal and the given year that were not reviewed in science blogs. Significant associations were also found between higher number of tweets and blog mentions and higher number of citations (4).

For the portfolio the researcher is requested to list download counts for a maximum of three publications. Some publishers provide this information, and download counts are also available for example from academia.edu and ResearchGate. The ACUMEN team is aware that influence cannot be measured through publications only; therefore download counts of the top three most downloaded datasets and software are also requested.

Mendeley readership counts are currently viewed as the most promising altmetric indicator (5). Mendeley has impressive coverage, for example 93% and 94% of the articles published in 2007 in Science and Nature respectively are on Mendeley (6). Similarly, extremely high coverage (97%) was found for articles published in JASIST (Journal of the American Society for Information Science and Technology) between 2001 and 2011 (7). In (5) the coverage of Mendeley for 20,000 random publications was only 63%, but still Mendeley had by far the greatest coverage of all currently studied altmetric sources. In the ACUMEN portfolio, the user is requested to report the number of readers of up to three publications. Mendeley readership counts can possibly be useful in the Social Sciences and the Humanities, where the coverage of the citation databases (WOS and Scopus, but also Google Scholar to a smaller extent) is far from perfect. Mendeley readership counts may also reflect influence in other areas, especially for newly published items that have not received a large number of citations yet, because it takes much longer to cite an item than to be a “reader” of the item. On the other hand, it should be taken into account that it may take some time for a research result to prove its significance; receiving attention in a very early stage does not necessarily mean that the impact is stable over longer time periods. In addition, populations that do not publish in the scholarly system (e.g. students) may also be interested and influenced by scholarly work without being authors (and citers). Mendeley readership counts capture the influence of scholarly work on non-publishing, interested individuals as well. This is supported by correlations of around 0.5 in several works between readership counts and citations – indicating that Mendeley readership counts reflect impact that is different from the impact reflected by citation counts (8). It was shown (9) that PhD students, postgraduates and postdocs are the main readers of articles in Mendeley.

Educational impact can also be measured by altmetrics. Many universities have YouTube channels where they upload videos of lectures (e.g., the YaleCourses YouTube channel). Conferences also often upload videos of talks to the Web, and presentations can uploaded to Slideshare. Interest in the materials available on these sites can be measured by the number of downloads and/or the number of views. Finally, if works of the portfolio owner are referenced in online syllabi this indicates educational impact of her work. Download counts and views of the “top” items in these categories are reported in the portfolio. In addition, the researcher is encouraged to provide details of three interesting web mentions of her, or of her work not mentioned elsewhere. Thus the altmetric data appearing in the portfolio supplement information on the scientific impact and also reflect on the societal impact of the researcher and his work.

Discussion and conclusion

Altmetrics is an emerging subfield of informetrics. Currently there are no clear guidelines on how to interpret the altmetric data in the portfolio. This is problematic both for the person filling in the portfolio and for the evaluator receiving portfolios. The best advice ACUMEN can provide at the moment is to compare with other researchers in the same field and at the same career stage. Traditional bibliometrics rely mainly on citations, whereas there are a multitude of altmetric sources. This further complicates interpretation, since we do not know how to (and probably cannot and should not) compare between tweets, downloads, blog mentions and readership counts. We are also aware that some of the altmetric indicators can be manipulated quite easily.

The aim of the ACUMEN Portfolio is to provide a holistic picture of the researcher’s achievements and capabilities. To achieve this aim it is necessary to include as many facets of the achievements as possible. The ACUMEN team believes that altmetric data complement traditional bibliometric data; they indicate influence not necessarily captured by citations, and thus provide additional value.

The ACUMEN portfolio can also be used for self-assessment. The portfolio template is available here (10), and the readers are most welcome to create their own portfolio. But beware: preparing the portfolio is quite time consuming. Have fun!

References