Reliability of Journal Impact Factor Rankings

Reliability of Journal Impact Factor Rankings
Darren C Greenwood (d.c.greenwood@leeds.ac.uk)

BMC Medical Research Methodology 2007, 7:48 doi:10.1186/147

ABSTRACT

Background
Journal impact factors and their ranks are used widely by journals, researchers, and research assessment exercises.

Methods

Based on citations to journals in research and experimental medicine in 2005, Bayesian Markov chain Monte Carlo methods were used to estimate the uncertainty associated with these journal performance indicators.

Results

Intervals representing plausible ranges of values for journal impact factor ranks indicated that most journals cannot be ranked with great precision. Only the top and bottom few journals could place any confidence in their rank position. Intervals were wider and overlapping for most journals.

Conclusion

Decisions placed on journal impact factors are potentially misleading where the uncertainty associated with the measure is ignored. This article proposes that caution should be exercised in the interpretation of journal impact factors and their ranks, and specifically that a measure of uncertainty should be routinely presented alongside the point estimate.

[http://www.biomedcentral.com/1471-2288/7/48]

Authority 2.0: The New Metrics of Scholarly Authority

The New Metrics of Scholarly Authority
By MICHAEL JENSEN

When the system of scholarly communications was dependent on the physical movement of information goods, we did business in an era of information scarcity. As we become dependent on the digital movement of information goods, we find ourselves entering an era of information abundance. In the process, we are witnessing a radical shift in how we establish authority, significance, and even scholarly validity. That has major implications for, in particular, the humanities and social sciences.

Scholarly authority is being influenced by many of the features that have collectively been dubbed Web 2.0 by Tim O'Reilly and others, and what I'll call Authority 2.0 in order to explore more fully the shifts that seem likely in the near future. While those trends are enabled by digital technology, I'm not concerned with technology per se — I learned years ago that technology doesn't drive change as much as our cultural response to technology does.

[snip]

That abundance changes greatly both the habits and business imperatives of the online environment. The lessons derived from successful Web 2.0 enterprises like Google, Flickr, YouTube, etc.include a general user impatience with any impediments ... [snip]. Abundance leads to immediate context and fact checking, which changes the "authority market" substantially. The ability to participate in most online experiences via comments, votes, or ratingsis now presumed, and when it's not available, it's missed.

[snip]

Web 2.0 is all about responding to abundance, which is a shift of profound significance.
[snip]
I think we're speeding — yes, speeding — toward a time when scholarship, and how we make it available, will be affected by information abundance ... .

But right now we're still living with the habits of information scarcity because that's what we have had for hundreds of years. Scholarly communication before the Internet required the intermediation of publishers. The costliness of publishing became an invisible constraint that drove nearly all of our decisions. It became the scholar's job to be a selector and interpreter of difficult-to-find primary and secondary sources; it was the scholarly publisher's job to identify the best scholars with the best perspective and the best access to scarce resources. Before risking half a year's salary on a book, the publisher would go to great lengths to validate the scholarship (with peer review), and to confirm the likely market for the publication. We evolved immensely complex, self-referential mechanisms to make the most of scarce, expensive resources. Consequently, scholarly authority was conferred upon those works that were well published by a respected publisher. It also could be inferred by a scholar's institutional affiliation (Yale or Harvard Universities vs. Acme State University). My father got his Ph.D. from Yale and had that implicit authority the rest of his professional life. Authority was also conferred by the hurdles jumped by the scholar, as seen in degrees and tenure status. And scholarly authority could accrue over time, by the number of references made to a scholar's work by other authors, thinkers, and writers — as well as by the other authors, thinkers, and writers that a scholar referenced. Fundamentally, scholarly authority was about exclusivity in a world of scarce resources.

Online scholarly publishing in Web 1.0 mimicked those fundamental conceptions. The presumption was that information scarcity still ruled. Most content was closed ... and the "authoritative" version was untouched by comments from the uninitiated. Authority was measured in the same way it was in the scarcity world of paper: by number of citations to or quotations from a book or article, the quality of journals in which an article was published, the institutional affiliation of the author, etc.
In contrast, in nonscholarly online arenas, new trends and approaches to authority have taken root in Web 2.0.

[snip]

Other Web 2.0 authority models are represented in a wide variety of sites. For example, there are the group-participation news and trend-spotting sites like Slashdot ... , which remixes, comments on, and links to other information Webwide; and digg, enabling up-or-down votes on whether a news story is worth giving value to. There is del.icio.us, a collection of favorite sites where descriptive tags denoting content raise the authority of a listed site. [snip]

[snip]

That engaged participation helps filter out crap and is a version of the "hive mind" form of conferred authority, where increasing populations simply add more richness to a resource. [snip]

S o what will be the next step? What is Authority 3.0, or Web 3.0? We will soon be awash in hundreds of billions of pages of content. How can we make sense of them? x Most technophile thinkers out there believe that Web 3.0 will be driven by artificial intelligences — automated computer-assisted systems that can make reasonable decisions on their own, to preselect, precluster, and prepare material based on established metrics, while also attending very closely to the user's individual actions, desires, and historic interests, and adapting to them.

[snip]

In the Web 3.0 world, we will also start seeing heavily computed reputation-and-authority metrics, based on many of the kinds of elements now used, as well as on elements that can be computed only in an information-rich, user-engaged environment. Given the inevitable advances in technology, remarkable things are likely to happen. In a world of unlimited computer processing, Authority 3.0 will probably include ...

• Prestige of the publisher (if any).


• Prestige of peer prereviewers (if any).


• Prestige of commenters and other participants.


• Percentage of a document quoted in other documents.


• Raw links to the document.


• Valued links, in which the values of the linker and all his or her other links are also considered.


• Obvious attention: discussions in blogspace, comments in posts, reclarification, and continued discussion.


• Nature of the language in comments: positive, negative, interconnective, expanded, clarified, reinterpreted.


• Quality of the context: What else is on the site that holds the document, and what's its authority status?


• Percentage of phrases that are valued by a disciplinary community.


• Quality of author's institutional affiliation(s).


• Significance of author's other work.


• Amount of author's participation in other valued projects, as commenter, editor, etc.


• Reference network: the significance rating of all the texts the author has touched, viewed, read.


• Length of time a document has existed.


• Inclusion of a document in lists of "best of," in syllabi, indexes, and other human-selected distillations.


• Types of tags assigned to it, the terms used, the authority of the taggers, the authority of the tagging system.

None of those measures could be computed reasonably by human beings. They differ from current models mostly by their feasible computability in a digital environment where all elements can be weighted and measured, and where digital interconnections provide computable context.

What are the implications for the future of scholarly communications and scholarly authority? First, consider the preconditions for scholarly success in Authority 3.0. They include the digital availability of a text for indexing (but not necessarily individual access — see Google for examples of journals that are indexed, but not otherwise available); the digital availability of the full text for referencing, quoting, linking, tagging; and the existence of metadata of some kind that identifies the document, categorizes it, contextualizes it, summarizes it, and perhaps provides key phrases from it, while also allowing others to enrich it with their own comments, tags, and contextualizing elements.

[snip]

Encourage your friends and colleagues to link to your online document. Encourage online back-and-forth with interested readers. Encourage free access to much or all of your scholarly work. Record and digitally archive all your scholarly activities. Recognize others' works via links, quotes, and other online tips of the hat. Take advantage of institutional repositories, as well as open-access publishers. [snip]

For universities, the challenge will be ensuring that scholars who are making more and more of their material available online will be fairly judged in hiring and promotion decisions. It will mean being open to the widening context in which scholarship is published, and it will mean that faculty members will have to take the time to learn about — and give credit for — the new authority metrics, instead of relying on scholarly publishers to establish the importance of material for them.

[snip]

I also don't know whether many, or most, scholarly publishers will be able to adapt to the challenge. But I think that those who completely lock their material behind subscription walls risk marginalizing themselves over the long term. They simply won't be counted in the new authority measures. They need to cooperate with some of the new search sites and online repositories, share their data with outside computing systems. They need to play a role in deciding not just what material will be made available online, but also how the public will be allowed to interact with the material.

That requires a whole new mind-set.

I hope it's clear that I'm not saying we're just around the corner from a revolutionary Web in which universities, scholarship, scholarly publishing, and even expertise are merely a function of swarm intelligences. That's a long way off. Many of the values of scholarship are not well served yet by the Web: contemplation, abstract synthesis, construction of argument.
Traditional models of authority will probably hold sway in the scholarly arena for 10 to 15 years, while we work out the ways in which scholarly engagement and significance can be measured in new kinds of participatory spaces.

But make no mistake: The new metrics of authority will be on the rise.

And 10 to 15 years isn't so very long in a scholarly career. Perhaps most important, if scholarly output is locked away behind fire walls, or on hard drives, or in print only, it risks becoming invisible to the automated Web crawlers, indexers, and authority-interpreters that are being developed. Scholarly invisibility is rarely the path to scholarly authority.

Michael Jensen is director of strategic Web communications for the National Academies. A version of this essay was originally presented as a speech given at Hong Kong University Press's 50th-anniversary celebration last fall.

The Chronicle ReviewVolume 53, Issue 41, Page B6
Copyright © 2007 by The Chronicle of Higher Education

[http://chronicle.com/weekly/v53/i41/41b00601.htm]

Scholarship in the Digital Age: Information, Infrastructure, and the Internet

Scholarship in the Digital Age: Information, Infrastructure, and the Internet

Christine L. Borgman

Scholars in all fields now have access to an unprecedented wealth of online information, tools, and services. The Internet lies at the core of an information infrastructure for distributed, data-intensive, and collaborative research. Although much attention has been paid to the new technologies making this possible, from digitized books to sensor networks, it is the underlying social and policy changes that will have the most lasting effect on the scholarly enterprise. In Scholarship in the Digital Age, Christine Borgman explores the technical, social, legal, and economic aspects of the kind of infrastructure that we should be building for scholarly research in the twenty-first century.

Borgman describes the roles that information technology plays at every stage in the life cycle of a research project and contrasts these new capabilities with the relatively stable system of scholarly communication, which remains based on publishing in journals, books, and conference proceedings. No framework for the impending "data deluge" exists comparable to that for publishing. Analyzing scholarly practices in the sciences, social sciences, and humanities, Borgman compares each discipline's approach to infrastructure issues. In the process, she challenges the many stakeholders in the scholarly infrastructure--scholars, publishers, libraries, funding agencies, and others--to look beyond their own domains to address the interaction of technical, legal, economic, social, political, and disciplinary concerns. Scholarship in the Digital Age will provoke a stimulating conversation among all who depend on a rich and robust scholarly environment.

Christine L. Borgman is Professor and Presidential Chair in Information Studies at the University of California, Los Angeles. She is the author of From Gutenberg to the Global Information Infrastructure: Access to Information in the Networked World (MIT Press, 2000).

Table of Contents

Detailed Contents - ix
[http://catdir.loc.gov/catdir/toc/ecip075/2006036057.html]
Preface - xvii
[http://mitpress.mit.edu/books/chapters/0262026198pref1.pdf]
Acknowledgments - xxi
1 Scholarship at a Crossroads - 1
2 Building the Scholarly Infrastructure -13
3 Embedded Everywhere - 33
4 The Continuity of Scholarly Communication -47
5 The Discontinuity of Scholarly Publishing - 75
6 Data: The Input and Output of Scholarship - 115
7 Building an Infrastructure for Information - 149
8 Disciplines, Documents, and Data - 179
9 The View from Here - 227
References - 267
Index - 321
[http://mitpress.mit.edu/books/chapters/0262026198index1.pdf]

Publisher Site
[http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=11333]

Interview

Inside Higher Ed / Nov. 14 2009 / ‘Scholarship in the Digital Age’

It’s hard to meet academics these days whose work hasn’t been changed by the Internet. But even if everyone knows that the world of scholarship has changed, it’s not always clear just how or the way those evolutions fit into the broad history of scholarship. Christine L. Borgman sets out to do just that in Scholarship in the Digital Age: Information, Infrastructure and the Internet, just published by MIT Press.

[http://www.insidehighered.com/news/2007/11/14/borgman]

Columbia University / Scholarly Communication Program / Research without Borders: The Changing World of Scholarly Communication / March 24 2009 / A-V / [Posted 04-10-09]

[http://scholcomm.columbia.edu/scholarship-digital-age-information-infrastructure-and-i]

Slides [In PDF]

[http://scholcomm.columbia.edu/files/BorgmanColumbia20090324.pdf]

Valuing Non-Traditional Vehicles of Scholarship

Scholarly Communications

in a

Digital World

A Convocation

January 27-28, 2005

VALUING NON-TRADITIONAL VEHICLES OF SCHOLARSHIP
A White Paper for the UNC-Chapel Hill Scholarly Communications Convocation

January 2005

by
Jeffrey Pomerantz, School of Information and Library Science
and
Bob Blouin, School of Pharmacy

This whitepaper, as the title suggests, concerns the issue of valuing non-traditional forms of scholarly output. By non-traditional, in the context of this convocation, we mean electronic as opposed to print publication, online as opposed to classroom teaching, the creation of electronic systems for collaboration as opposed to other forms of service to one’s professional community. We mean electronic here, but this discussion can and should be expanded to include scholarly output in other media.

The issue of valuing vehicles of scholarship – whatever the medium – rests on another, more fundamental question: what constitutes scholarship? There is of course no universal standard for evaluating scholarship or scholarly output: the university, the school, the department, the discipline, all are likely to have different standards. This may then lead to Justice Potter Stewart-like evaluations of excellence, where it cannot be defined, but one knows it when one sees it. Still, despite this inconsistency of standards, the scholarly community has been producing excellent scholarship for centuries, so clearly the lack of a universal standard for evaluating traditional scholarly output is not a major stumbling block. The problem is not to arrive at a universal standard for evaluating scholarly output, but to ensure that the different levels of the institution have standards that recognize and reward excellence in scholarship when it is present.

In the evaluation of scholarship, the value is in the ideas, and in the impact that those ideas have, and not in the medium in which those ideas are presented. We must recall that the scholarly journal as we know it today only dates back to the mid-1600s, and it undoubtedly took some time for this new medium to be accepted alongside meetings of learned societies as legitimate vehicles for scholarly communication. The credibility of any new medium must be earned, but once a new medium is established as being valuable, a community of users over time adapts to and embraces that medium. This is as true for scholarly uses as for any other purpose.

[snip]

As electronic and online forms of scholarly output come to take center stage in all aspects of scholarship, it is increasingly important for the university to be a favorable and nurturing environment for faculty who will be innovators in these areas. At the same time, it is increasingly important for faculty to innovate, to place high demands on the university for infrastructure to support non-traditional vehicles of scholarly output, and for recognition of the importance of this output. Many of the innovators on the faculty are likely to be younger faculty members: this upcoming generation of faculty are not only more comfortable with using and producing electronic materials, they also place more demands on the institution’s infrastructure. As senior faculty approach retirement and new positions are filled with younger faculty members, the demands on the university and the necessity of fostering a nurturing environment for innovation will only increase over time. In order to recruit and retain innovative faculty members, the university must provide such support.

[snip]

How, then, should the university change so as to provide support for non-traditional vehicles of scholarship? How can the university create standards for evaluating scholarship that value non-traditional forms of output? This change cannot come by simply demanding it; no one like being dictated to, and academics probably less than most. No, the scholarly community will be swayed by solid intellectual arguments and evidence of the value of new forms of scholarly output. When a solid argument can be made for the value of a new form of scholarly output, scholars will pay attention to output in that form. If excellent scholarship appears in new forms, it will be recognized – perhaps not immediately, but in time. Indeed, these arguments are already being made, evidence is already being presented. Early arguments against electronic scholarly journal publishing were that it lacked intellectual validity and diminished the rigor of the review process; this was in time shown to not be true, and there are now many top-quality electronic journals. Early efforts in online instruction were received with skepticism and the same criticisms of its intellectual validity; now research has demonstrated that online and classroom teaching each are useful for teaching students with different learning styles. The medium is not important; what is important are the ideas presented and the impact that those ideas have on the recipients and on society at large. The demonstration of the excellence of scholarship, in all its forms, is in this impact.

The impact of scholarship is central to the academic review process, and the academic review process is central to the retention and rewarding of faculty. And central to the academic review process is peer review. Peer review is where the rubber hits the road, so to speak, in terms of standards for evaluating scholarship. As scholars are favorably reviewed by their peers for producing non-traditional output, others perhaps less innovative will be emboldened to produce such output. [snip]

Valuing non-traditional forms of scholarship requires a critical evaluation of non-traditional forms of scholarship. Such a critical evaluation should start within institutions that have the most to lose by failing to recognize the importance. Those institutions may be the university, or schools or departments within the university. [snip]

[http://www.unc.edu/scholcomdig/whitepapers/pomerantz-blouin.html]

DisruptiveScholarshipModel

[http://www.public.iastate.edu/~gerrymck/TICER2005.ppt]

Invisible Hand(s): Quality Assurance in the Age of Author Self-Archiving

Friends/

Invisible Hand(s): Quality Assurance in the Age of Author Self-Archiving

JCOM 2 (3), September 2003

[http://jcom.sissa.it/archive/02/03/F020302/]

Abstract: There are forces, factors, and influences other than pending classical peer review that assure the quality of scholarship before formal publication.

[http://www.public.iastate.edu/~gerrymck/Hands.ppt]

/Gerry

The Promise of Authority in Social Scholarship

At the 2007 Charleston Conference held November 7-10, Laura B. Cohen, Web Support Librarian at the University of Albany, SUNY, joined Leigh Dodds, Chief Technology Officer of the scholarly publisher Ingenta, to speak on the theme of "Authoritative? What's That? And Who Says?"


November 8 2007 / Thursday, 2:00 – 2:50 PM / Concurrent Session 1

Web 2.0 makes it easier for anyone to publish information online, and search engines make content more easily findable. But how do users know what information is authoritative? Do they even understand what "authoritative" means? And who defines that something is "authoritative" in the first place?

In scholarly publishing, the peer review process is an indicator of quality. But as content is increasingly mashed-up, syndicated and blogged in many different locations, how do users differentiate between peer reviewed content, and \"user generated content\"? And is there a natural progression from the creative chaos of Wikipedia, through the “gentle expert oversight" of Citizendium to, ultimately, the closed rigorous approach of double-blind peer review.

[http://www.katina.info/conference/program.php]

And is there a natural progression from the creative chaos of Wikipedia, through the “gentle expert oversight" of Citizendium to, ultimately, the closed rigorous approach of double-blind peer review? Laura's PPT presentation titled"The Promise of Authority in Social Scholarship" is available on SlideShare

[http://www.slideshare.net/lcohen/the-promise-of-authority-in-social-scholarship/]

[http://liblogs.albany.edu/library20/2007/11/presentation_on_the_metrics_of.html]