Twenty-third Thing: Using Publishing Metrics Ethically

Not all metrics are equal, and numbers don’t always reflect the true impact of research. Thing 23, written by Kristijan Causovski, outlines the frameworks that underpin the ethical use of research metrics and why it’s so important to take an ethical approach.

What are publishing metrics? 

Researchers across all disciplines produce a wide variety of research outputs. These include: 

• Traditional Research Outputs e.g., journal articles, conference publications, books and book chapters. 
• Non-Traditional Research Outputs (NTROs) e.g., original creative works, live performance of creative works, recorded/rendered creative works, curated or produced substantial public exhibitions and events, research reports for an external body, and portfolios. 

Research output metrics, used to quantify engagement with outputs, include: 

• Traditional Bibliometrics e.g. journal impact factors, publication citation counts, author h-indexes. 
• Non-traditional Bibliometrics or Altmetrics e.g. online publication mentions across social media, blogs, news, policy documents, forums, encyclopedias and other multimedia platforms. 

Why is the ethical use of research output metrics so important? 

Researchers and academic staff are often asked to demonstrate the impact of their research in applications for academic promotions, grants and in many other contexts.  This gives rise to ethical considerations of what metrics to use and how to use them in order that outputs are framed contextually as part of a qualitative narrative.  

Several frameworks have emerged, dedicated to promoting the ethical use of metrics when assessing research impact and engagement across all disciplines. Three prominent frameworks are listed below. 

Frameworks for the ethical use of research output metrics

The Leiden Manifesto for Research Metrics (LM)

The LM was created in 2015 to provide 10 principles to editors and reviewers to help counter the misuse of bibliometrics, and “impact-factor obsession,” and guide their research evaluation. These principles are: 

• Quantitative evaluation should support qualitative, expert assessment. 

• Measure performance against the research missions of the institution, group or researcher. 

• Protect excellence in locally relevant research. 

• Keep data collection and analytical processes open, transparent, and simple. 

• Allow those evaluated to verify data and analysis. 

• Account for variation by field in publication and citation practices. 

• Base assessment of individual researchers on a qualitative judgement of their portfolio. 

• Avoid misplaced concreteness and false precision. 

• Recognise the systemic effects of assessment and indicators. 

• Scrutinise indicators regularly and update them. 

San Francisco Declaration on Research Assessment (DORA) 

DORA was born of a consensus amongst a group of editors and publishers of academic journals within the field of cell biology, that journal impact factors were insufficient and/or inappropriate in capturing the value of the research published in these journals.  

Emerging in 2012 and formalised in 2013, DORA is a global initiative encompassing all scholarly disciplines, that guides key stakeholders such as funders, publishers, professional societies, institutions, and researchers to develop, use and promote best practice in the evaluation of scholarly research. 

A total of 18 recommendations spanning all stakeholder groups have been developed, with several underlying key themes including: 

• The need to eliminate the overuse and reliance of journal-based metrics (e.g. journal impact factor) in considerations relating to research funding, appointment and promotions. 
• The need to prioritise the merits of the research itself over the journal in which it is published when assessing impact. 
• The need to take advantage of the benefits offered by online publication (e.g. easing irrelevant restrictions on the number of words, figures, and references in articles, and searching for new measures of significance and impact). 

Committee on Publication Ethics (COPE) 

Formed over 25 years ago, COPE predates both the LM and DORA.  Its principles encompass ethical considerations beyond the use of research output metrics, looking at publishing ethics more broadly. It seeks to educate and support editors, publishers, universities, research institutes, and all those involved in the publication process, to ensure that ethical practice becomes a cultural norm rather than an aspirational goal. 

COPE’s core practices were developed in 2017, outlining the importance for journals and publishers to ensure that their publication practices are solid, unambiguous and transparent in each of the following areas: 

• Allegations of misconduct
• Authorship and contributorship
• Complaints and appeals
  
• Conflicts of interest / competing interests
  
• Data and reproducibility
• Ethical oversight
• Intellectual property
• Journal management
• Peer review processes
• Post-publication discussions and corrections 

What impact have these ethical frameworks and committees had? 

DORA has been signed by 23,346 individuals and organisations in 161 countries to date. To date, the University of Melbourne is the only Australian university to have signed. 

As of 2022, COPE has over 13,000 members globally across all research disciplines. 

According to a blog on the LM website, findings from a Taiwanese article comparing the LM and DORA across their frameworks, citation impact and dissemination channels discovered that: 

• Journal editorial boards have focused on DORA; 
• Whereas articles in the scientometrics field of research and multidisciplinary journals cite the LM. 

Learn more 

• ERA (Excellence in Research for Australia) 2023 Submission Guidelines – defines and provides examples of: 
  • Traditional research outputs pp.29-33 
  • Non-traditional research outputs (NTROs) pp.33-37 
• University of Melbourne library guide on Research Impact – contains information on the multidisciplinary considerations around research impact. 
• Leiden Manifesto for Research Metrics (LM) – provides an overview of the Leiden Manifesto and its purpose. 
• Bibliometrics: The Leiden Manifesto for research metrics – lists and explains the 10 LM principles. 
• San Francisco Declaration on Research Assessment (DORA) – explains the motivation for the formation and purpose of DORA. 
• Committee on Publication Ethics (COPE) – highlights the purpose of COPE and links to various free and member-only resources.

About the Author 

Kristijan Causovski is the Liaison Librarian, Business and Economics, Scholarly Services, at the University of Melbourne. 

Cite this Thing

You are free to use and reuse the content on this post with attribution to the author. The citation for this Thing is:

CAUSOVSKI, KRISTIJAN (2024). Twenty-third Thing: Using Publishing Metrics Ethically. The University of Melbourne. Online resource. https://doi.org/10.26188/25347196

 

Featured imaged credit: Photo by Negative Space on Pexels