Establishing a Research Agenda for Scholarly Communication. 5

5. Value and Value Metrics in Scholarly Communications

Determining and measuring the effectiveness of and the value that is derived from scholarly communication is challenging and often subjective. John Houghton’s work on scholarly communication in Australia seeks to measure the economic and social returns to public sector investment in research and development and how those might rise with open access to published research findings. Analyses of citations in the published literature to the published literature lead to metrics such as the h-index, Eigenfactor, and the heavily-used impact factor. Extant measures may suffer from being tightly coupled to traditional processes while also inhibiting the application of other measures of value.

In the new digital environment, activities other than traditional or formal publication should be valued in the reward structure for scholarship. To this end, the Modern Language Association provides an example of examining current standards and emerging trends in publication requirements for tenure and promotion.

Illustrative Challenges
Citation analysis relies on a 50 year old assumption that the number of citations represents value, but in today’s environment this assumption is limiting. Other metrics could reflect the scholarly significance of new discoveries as they are developed and communicated. Effective metrics must be based on resources and practices that truly advance scholarly research. For example, it could be argued that journal articles have become totems to accrue and count for tenure and promotion but are not unique in their ability to advance scholarship and may be losing some effectiveness for this purpose.

"Open notebook science" and “open data” are examples of new research and communication practices that might be advancing scholarly research as much or more than what is possible through scholarly publication. The relationship between the reward system and indicators of the progress of knowledge is more tenuous. What resources and practices truly advance scholarly research? Even where robust indicators of the progress of knowledge exist, their relationship with the current reward system may be tenuous. How can the value and impact of communication practices be assessed and documented? How could these assessments be assimilated into the reward system?

Libraries should adopt a stronger role that more directly advances scholarly research beyond satisfying tenure and promotion practices. A starting point is work reported by King and Harley regarding formal vs. informal scholarly communications. Given that scholars are finding new ways to register, seek comment, refine, evaluate, and certify their work, how can those processes be tracked, recorded, measured, and reported as part of the value-chain in scholarship?

Should informal communications be captured and preserved by libraries, and if so, how? A useful analogy is to consider that presentations, preprints and letters and other informal communications are the conversations of science, while publications are the minutes. In some disciplines, journals are becoming less important to scholars than their professional meetings and informal networks where their accomplishments are recognized. How can librarians better characterize and, measure the contributions of these informal communications, and thereby make wise decisions about organized access to them?

Libraries also need to determine the value of their own services as contributions to the communication of scholarship. Which services, such as institutional repositories, should be evaluated, and what tools and measures exist for this purpose?

Research Possibilities
• Identify and evaluate the range of metrics currently used to measure the value and impact of scholarly publishing. There is citation analysis and its derivatives; but there are other measures being developed, including those that combine usage/readership and citation, such as those by the MESUR project at Los Alamos National Laboratory, and the UKSG Serials Group. A literature review to collect these efforts would provide a central reference point.
• Explore additional measurements that incorporate new kinds of indicators of value and covering a broader range of communication activities. New measures should address increased research efficiency and productivity, variations between disciplines, advancement of the process of research.

Other metrics may:
° characterize the value of Open Data and Open Notebook Science (disseminating source data, research methods, and negative experimental or clinical results) to advancing research and knowledge.
° correspond to technology transfer or other uses of new knowledge beyond generating further research, for instance, number of views, number of patents.
° show how informal communications are advancing the process of research

• Explore the relative value, importance, and significance of traditional journal and book publication compared to newer, informal forms of scholarly communication for a sample of representative scholars. This could build on studies by the CIC and Estabrook that indicated that in the humanities there is some acceptance of digital publications and new forms, while the scholarly monograph was still the standard for promotion and tenure.

[http://acrl.ala.org/scresearchagenda/index.php?title=Theme_5:_Value_and_Value_Metrics_of_Scholarly_Communications]