A Principal Component Analysis of 39 Scientific Impact Measures

A Principal Component Analysis of 39 Scientific Impact Measures

Bollen J, Van de Sompel H, Hagberg A, Chute R, 2009 A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6): e6022. doi:10.1371/journal.pone.0006022

Background

The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact.

Methodology

We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data.

Conclusions

Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

Received: May 14, 2009; Accepted: May 26, 2009; Published: June 29, 2009

Excerpts

[snip]

A variety of impact measures can be derived from raw citation data. It is however highly common to assess scientific impact in terms of average journal citation rates. In particular, the Thomson Scientific Journal Impact Factor (JIF) [1] which is published yearly as part of the Journal Citation Reports (JCR) is based on this very principle; ... .

The JIF has achieved a dominant position among measures of scientific impact for two reasons. First, it is published as part of a well-known, commonly available citation database (Thomson Scientific's JCR). Second, it has a simple and intuitive definition. The JIF is now commonly used to measure the impact of journals and by extension the impact of the articles they have published, and by even further extension the authors of these articles, their departments, their universities and even entire countries. However, the JIF has a number of undesirable properties which have been extensively discussed in the literature [2], [3], [4], [5], [6]. This had led to a situation in which most experts agree that the JIF is a far from perfect measure of scientific impact but it is still generally used because of the lack of accepted alternatives.

The shortcomings of the JIF as a simple citation statistic have led to the introduction of other measures of scientific impact. Modifications of the JIF have been proposed to cover longer periods of time [7] and shorter periods of times (JCR's Citation Immediacy Index). Different distribution statistics have been proposed, e.g. Rousseau (2005) [8] and the JCR Citation Half-life (http://scientific.thomson.com/free/essay​s/citationanalysis/citationrates/ ). The H-index [9] was originally proposed to rank authors according to their rank-ordered citation distributions, but was extended to journals by Braun (2005) [10]. Randar (2007) [11] and Egghe (2006) [12] propose the g-index as a modification of the H-index.

[snip]

Since scientific literature is now mostly published and accessed online, a number of initiatives have attempted to measure scientific impact from usage log data. The web portals of scientific publishers, aggregator services and institutional library services now consistently record usage at a scale that exceeds the total number of citations in existence. In fact, Elsevier announced 1 billion fulltext downloads in 2006, compared to approximately 600 million citations in the entire Web of Science database. The resulting usage data allows scientific activity to be observed immediately upon publication, rather than to wait for citations to emerge in the published literature and to be included in citation databases such as the JCR; a process that with average publication delays can easily take several years. Shepherd (2007) [19] and Bollen (2008) [20] propose a Usage Impact Factor which consists of average usage rates for the articles published in a journal, similar to the citation-based JIF. Several authors have proposed similar measures based on usage statistics [21]. Parallel to the development of social network measures applied to citation networks, Bollen (2005, 2008) [22], [23] demonstrate the feasibility of a variety of social network measures calculated on the basis of usage networks extracted from the clickstream information contained in usage log data.

These developments have led to a plethora of new measures of scientific impact that can be derived from citation or usage log data, and/or rely on distribution statistics or more sophisticated social network analysis. However, which of these measures is most suitable for the measurement of scientific impact?

This question is difficult to answer for two reasons. First, impact measures can be calculated for various citation and usage data sets, and it is thus difficult to distinguish the true characteristics of a measure from the peculiarities of the data set from which it was calculated. Second, we do not have a universally accepted, golden standard of impact to calibrate any new measures to. In fact, we do not even have a workable definition of the notion of “scientific impact” itself, unless we revert to the tautology of defining it as the number of citations received by a publication. As most abstract concepts “scientific impact” may be understood and measured in many different ways. The issue thus becomes which impact measures best express its various aspects and interpretations.

Here we report on a Principal Component Analysis (PCA) [24] of the rankings produced by a total of 39 different, yet plausible measures of scholarly impact. 19 measures were calculated from the 2007 JCR citation data and 16 from the MESUR project's log usage data collection (http://www.mesur.org/). We included 4 measures of impact published by the Scimago (http://www.scimagojr.com/) group that were calculated from Scopus citation data. The resulting PCA shows the major dimensions along which the abstract notion of scientific impact can be understood and how clusters of measures correspond to similar aspects of scientific impact.

[snip]

PDF Available at

[http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0006022&representation=PDF]

Source

[http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0006022]

See Also

MESUR For Measure: MEtrics from Scholarly Usage of Resources

[http://scholarship20.blogspot.com/2008/12/mesur-for-measure-metrics-from.html]

Article-Level Metrics (At PLoS And Beyond)

"Article-Level Metrics at PLoS" takes the view that readers need some way to measure, or at least indicate, the 'worth' (or ‘impact’ etc) of a journal article.

With over a million articles published per year it’s impossible to read everything & so filtering tools are needed. Some journals have been experimenting with providing data on online usage, but PLoS is going further than this.

We are at the start of a program to provide citations, usage data, social bookmarking activity, media & blog coverage, commenting activity, 'star' ratings etc on all articles we publish.

Source ; Slides / Slide Notes / Audio / PDF Available At

[http://www.myplick.com/view/cf8qFak4Ymv/Article-Level-Metrics-at-PLoS-and-beyond]

Related / See Also

[http://everyone.plos.org/2009/05/27/article-level-metrics-at-plos/]

ELPUB 2009 Abstract / Presentation / Paper [06-11-09] (07-01-09)

PLoS One: background, future development, and article-level metrics / Peter Binfield

[http://conferences.aepic.it/index.php/elpub/elpub2009/paper/view/114]

"On Article-Level Metrics And Other Animals"

[http://network.nature.com/people/rpg/blog/2009/06/22/on-article-level-metrics-and-other-animals]

Article-Level Metrics At PLoS > Addition Of Usage Data

[http://scholarship20.blogspot.com/2009/09/article-level-metrics-at-plos-addition.html]

See Also

Article-Level Metrics And The Evolution Of Scientific Impact

[http://scholarship20.blogspot.com/2009/11/article-level-metrics-and-evolution-of.html]

Introduction to Webometrics: Quantitative Web Research for the Social Sciences

Introduction to Webometrics: Quantitative Web Research for the Social Sciences / Michael Thelwall‌ / Morgan & Claypool Publishers / 2009 / 116 pp. / doi:10.2200/S00176ED1V01Y200903ICR004 / ISBN-10: 159829993X ; ISBN-13: 978-1598299939

Abstract

Webometrics is concerned with measuring aspects of the web: web sites, web pages, parts of web pages, words in web pages, hyperlinks, web search engine results. The importance of the web itself as a communication medium and for hosting an increasingly wide array of documents, from journal articles to holiday brochures, needs no introduction. Given this huge and easily accessible source of information, there are limitless possibilities for measuring or counting on a huge scale (e.g., the number of web sites, the number of web pages, the number of blogs) or on a smaller scale (e.g., the number of web sites in Ireland, the number of web pages in the CNN web site, the number of blogs mentioning Barack Obama before the 2008 presidential campaign).

This book argues that it can be useful for social scientists to measure aspects of the web and explains how this can be achieved on both a small and large scale. The book is intended for social scientists with research topics that are wholly or partly online (e.g., social networks, news, political communication) and social scientists with offline research topics with an online reflection, even if this is not a core component (e.g., diaspora communities, consumer culture, linguistic change).

The book is also intended for library and information science students in the belief that the knowledge and techniques described will be useful for them to guide and aid other social scientists in their research. In addition, the techniques and issues are all directly relevant to library and information science research problems.

General Of Table of Contents

Introduction / Web Impact Assessment / Link Analysis / Blog Searching / Automatic Search Engine Searches: LexiURL Searcher / Web Crawling: SocSciBot / Search Engines and Data Reliability / Tracking User Actions Online / Advanced Techniques / Summary and Future Directions

Detailed Table Of Contents

1. Introduction
1.1 New Problems: Web-Based Phenomena
1.2 Old Problems: Offline Phenomena Reflected Online
1.3 History and Definition
1.4 Book Overview

2. Web Impact Assessment
2.1 Web Impact Assessment Via Web Mentions
2.2 Bespoke Web Citation Indexes
2.3 Content Analysis
2.3.1 Category Choices
2.3.2 Sampling Methods
2.3.3 Example
2.3.4 Validity
2.4 URL Analysis of the Spread of Results
2.5 Web Impact Reports
2.6 Web Citation Analysis—An Information Science Application
2.7 Advanced Web Impact Studies
2.8 Summary

3. Link Analysis
3.1 Background: Link Counts as a Type of Information
3.2 Types of Webometric Link Analysis
3.3 Link Impact Assessments
3.3.1 Interpreting the Results
3.3.2 Alternative Link Counting Methods
3.3.3 Case Study: Links to ZigZagMag.com
3.4 Content Analysis of Links
3.6 Colink Relationship Mapping
3.7 Link Impact Reports
3.8 Large-Scale Link Analysis with Multiple Site Groups
3.9 Link Differences Between Sectors—An Information Science Application
3.10 Summary

4. Blog Searching
4.1 Blog Search Engines
4.2 Date-Specific Searches
4.3 Trend Detection
4.4 Checking Trend Detection Results
4.5 Limitations of Blog Data
4.6 Advanced Blog Analysis Techniques
4.7 Summary

5. Automatic Search Engine Searches: LexiURL Searcher
5.1 Introduction to LexiURL Searcher
5.2 LexiURL Searcher Web Impact Reports
5.2.1 Web Impact Reports—Classic Interface Example
5.3 LexiURL Searcher Link Impact Reports
5.3.1 Link Impact Reports—Classic Interface Example
5.4 LexiURL Searcher for Network Diagrams
5.4.1 Rearranging, Saving, and Printing Network Diagrams
5.4.2 Network Diagram—Classic Interface Example
5.4.3 Colink Network Diagrams
5.5 LexiURL Searcher Additional Features

6. Web Crawling: SocSciBot
6.1 Web Crawlers
6.2 Overview of SocSciBot
6.3 Network Diagrams of Sets of Web Sites with SocSciBot
6.4 Other Uses for Web Crawls

7. Search Engines and Data Reliability
7.1 Search Engine Architecture
7.1.1 Duplicate and Near-Duplicate Elimination
7.2 Comparing Different Search Engines
7.3 Research Into Search Engine Results
7.4 Modeling the Web’s Link Structure

8. Tracking User Actions Online
8.1 Single-Site Web Analytics and Log File Analysis
8.2 Multiple-Site Web Analytics
8.3 Search Engine Log File Analysis

9. Advanced Techniques
9.1 Query Splitting
9.2 Virtual Memetics
9.3 Web Issue Analysis
9.4 Data Mining Social Network Sites
9.5 Social Network Analysis and Small Worlds
9.6 Folksonomy Tagging
9.7 API Programming and Mashup

10. Summary and Future Directions

Glossary /References / Author / Biography

Publication Web Page

[http://www.morganclaypool.com/doi/abs/10.2200/S00176ED1V01Y200903ICR004]

Access For Subscribers / Pay-Per-View For Non-Subscribers

Open Access No Longer Available [05-30-09]

Also Available In Paper

[http://www.amazon.com/Introduction-Webometrics-Quantitative-Synthesis-Information/dp/159829993X]

Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics

Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics / Nicola De Bellis / Scarecrow Press / March 2009 / 450 pp. / ISBN: 0-8108-6713-3 ; ISBN-13: 978-0-8108-6713-0 / $ 55 / Paper

DESCRIPTION

Can the methods of science be directed toward science itself? How did it happen that scientists, scientific documents, and their bibliographic links came to be regarded as mathematical variables in abstract models of scientific communication? What is the role of quantitative analyses of scientific and technical documentation in current science policy and management? Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics answers these questions through a comprehensive overview of theories, techniques, concepts, and applications in the interdisciplinary and steadily growing field of bibliometrics.

Since citation indexes came into the limelight during the mid-1960s, citation networks have become increasingly important for many different research fields. The book begins by investigating the empirical, philosophical, and mathematical foundations of bibliometrics, including its beginnings with the Science Citation Index, the theoretical framework behind it, and its mathematical underpinnings. It then examines the application of bibliometrics and citation analysis in the sciences and science studies, especially the sociology of science and science policy. Finally it provides a view of the future of bibliometrics, exploring in detail the ongoing extension of bibliometric methods to the structure and dynamics of the World Wide Web.

This book gives newcomers to the field of bibliometrics an accessible entry point to an entire research tradition otherwise scattered through a vast amount of journal literature. At the same time, it brings to the forefront the cross-disciplinary linkages between the various fields (sociology, philosophy, mathematics, politics) that intersect at the crossroads of citation analysis. Because of its discursive and interdisciplinary approach, the book is useful to those in every area of scholarship involved in the quantitative analysis of information exchanges, but also to science historians and general readers who simply wish to familiarize themselves with an important, albeit increasingly complex area of information science.

TABLE OF CONTENTS [World Cat]

Biblio/sciento/infor-metrics : terminological issues and early historical developments -- The empirical foundations of bibliometrics : the Science citation index -- The philosophical foundations of bibliometrics : Bernal, Merton, Price, Garfield, and Small -- The mathematical foundations of bibliometrics -- Maps and paradigms : bibliographic citations at the service of the history and sociology of science -- Impact factor and the evaluation of scientists : bibliographic citations at the service of science policy and management -- On the shoulders of dwarfs : citation as rhetorical device and the criticisms to the normative model -- Measuring scientific communication in the twentieth century : from bibliometrics to cybermetrics.

ABOUT THE AUTHOR

Nicola De Bellis is a medical librarian at the University of Modena and Reggio Emilia [Italy]

Source

[http://www.scarecrowpress.com/Catalog/SingleBook.shtml?command=Search&db=%5EDB/CATALOG.db&eqSKUdata=0810867133]

Flyer/Order Form

[http://www.scarecrowpress.com/Catalog/Flyer2.shtml?SKU=0810867133]

Web 2.0 and Scholarly Communication

Web 2.0 and Scholarly Communication

Mark Ware

We examine the ways in which Web 2.0 tools and services – including blogs, wikis, social bookmarking and tagging, social networking and data interoperability and re-use – are affecting scholarly communication, with examples and usage data where available.

We find that many of the tools have yet to live up to their early promise and the expectations that rode on them, and discuss the possible reasons for this.

• What is Web 2.0?


• Web 1.0 and scholarly communication


• Web 2.0 and Open Access


• Blogs


• Social bookmarking


• Social networking Podcasts


• Wikis


• Data


• Peer review


• Reasons for lack of uptake to date

Source and Fulltext

[http://mrkwr.files.wordpress.com/2009/05/ware-web-2-0-and-scholarly-communication-preprint.pdf]

See Also

[http://mrkwr.wordpress.com/2009/05/14/web-2-0-and-scholarly-communication/]

CfP: Communication Pedagogy in the Age of Social Media

CALL FOR PAPERS

Electronic Journal of Communication (EJC)

Special Issue

Communication Pedagogy in the Age of Social Media

Over the course of the last few years, social media technologies such as blogs, microblogs, digital videos, podcasts, wikis, and social networks, have seen a dramatic increase in adoption rates.

To date, Internet users have uploaded roughly 80 million videos to YouTube and launched approximately 133 million blogs worldwide. Because of their ability to connect people and to facilitate the exchange of information and web content, social media technologies not only provide a powerful new way to interact with one another, but they also present exciting new pedagogical opportunities.

Earlier this year, the New Media Consortium and the EDUCAUSE Learning Initiative released the 2008 Horizon Report, which seeks to identify new technologies capable of affecting the way we teach and learn. Among the critical challenges outlined by this year’s report is the need for universities to equip students with new media literacy skills and to develop curricula that “address not only traditional capabilities like developing an argument over the course of a long paper”, but also “how to create meaningful content with today’s tools.” (The New Media Consortium, 2008, p. 6).

Considering that these tools center around the ideas of collaboration, participation, and conversation, they should hold special interest to communication researchers and educators alike. As a result, this special issue seeks to examine the pedagogical applications of social media technologies, especially with regard to the communication classroom.

Examples of best practices in social media adoption in all areas of communication education are welcome, as are case studies or empirical research analyzing the effectiveness and/or effects of incorporating social media technologies into the communication classroom.

Research examining the role these technologies play in the social construction of a collective knowledge pool would also fit within the scope of this special issue.

The special issue is scheduled for publication in the first half of 2010. Deadline for completed manuscripts is June 15, 2009.

Submissions should be electronic (.doc or .rtf format) and must conform to the specifications of the Publication Manual of the American Psychological Association, 5th ed. Place author’s contact information in an email to the editor only, not on the title page of the submission.

Issue Editors:Corinne Weisgerber, Ph.D. and Shannan H. Butler, Ph.D. / St. Edward’s University

Send inquiries and submissions to: corinnew@stedwards.edu

Source

[http://www.cios.org/www/ejc/calls/socmedia.htm]

The Paradigms They Are A-Changin' > Open / Semantic / Social / Mobile

Colleagues/

The Paradigms They Are A-Changin' > The Future Of Research And Scholarship: Open / Semantic / Social / Mobile

There are four major themes that are and will become the context and framework of research and scholarship in the 21st Century: Open / Semantic / Social / Mobile:

Open >

Open Access / Open Data / Open Peer Review / Open Research

Semantic >

Audio / Interactivity / Supplemental Content / Video

Social >

Science Blogging / Social Bookmarking / Social Networking / Social Software

Mobile >

Mobile Access / Mobile Content / Mobile Data / Mobile Research

In scheduled presentation(s), we will briefly profile select developments related to these major themes and speculate on their potential evolution and impact on research and scholarship in the coming decade(s).

In June I Will Give Four (4) Invited Presentations On The Topic(s) In Ireland At

The National Library Of Ireland, Dublin (June 4 / 1:00 PM)

Lecture : The Paradigms They Are A-Changin': The Future of Research and Scholarship / Thursday / 4 June 2009 / 1.00 PM / Seminar Room / Admission Free / Booking Is Not Required

PPT Now Available At

[http://www.public.iastate.edu/~gerrymck/DublinParadigms.ppt] [06-22-09] [161]

The National University of Ireland - Galway (June 5 / 1:00 PM)

Library Seminar: The Future Of Research And Scholarship: Open / Semantic / Social / Mobile /Friday / 5 June / 1300 / Library Meeting Room, Basement Level, James Hardiman Library

Note: Sandwiches will be provided, so please contact Aoife Harrington (aoife.harrington@nuigalway.ie or ext. 2593) in advance for catering purposes.

PowerPoint Now Available At

[http://www.public.iastate.edu/~gerrymck/GalwayParadigms.ppt] [06-22-09] [93 Slides]

The University College Cork, Cork (June 8 / 10:00 AM)

Boole Lecture Theatre 2 / Co-Sponsored by Ionad Bairre

PowerPoint Now Available At

[http://www.public.iastate.edu/~gerrymck/CorkParadigms.ppt] [06-22-09] [122 Slides]

Audio/Video (1:24:04) Available At

[http://www.public.iastate.edu/~gerrymck/ParadigmsCork.m4v] (750.2 MB)

[QuickTime / iTunes]

>> Introduced By Crónán Ó Doibhlin / Sub-Librarian / Special Collections, Archives & Repository Services / George Boole Library / University College Cork

>> Thanks To Stephen Yearl / Library Digital Projects Officer ; Michael Tobin / Information Technology Analyst / Library Information Technology Services Services // Boole Library / College Cork For Providing (07-09/09)

The University of Limerick, Limerick (June 9 / 2:30 PM).

Symposium: Social Networking Services at Third-Level: Trends and Developments: "The Paradigms They Are A-Changin': The Future of Research and Scholarship (Open / Semantic / Social / Mobile)" / Tuesday / June 9th 2009 / 2.30 pm / Charles Parsons Lecture Theatre / All welcome

Contact Dr. Tríona Hourigan for further information.Tel: 00 353 (0) 61 234675

PowerPoint Now Available At

[http://www.public.iastate.edu/~gerrymck/LimerickParadigms.ppt] [06-22-09] [103 slides]

Combined / Master PowerPoint Presentation Available At

[http://www.public.iastate.edu/~gerrymck/Paradigms.ppt] [06-22-09] [270 Slides]

Thank You > Thank You > Thank You > Thank You >

I would like to thank the following individuals/organizations for their support, generosity, and hospitality during our Magical Visit To Ireland

National Library Of Ireland, Dublin

Joanna Finegan / Assistant Keeper / Education & Outreach / National Library of Ireland, Dublin

Sarah Shiel / Executive Education Assistant / Education & Outreach / National Library of Ireland, Dublin

National University of Ireland, Galway

John Cox / Acting Librarian / James Hardiman Library /National University of Ireland, Galway

Niall McSweeney / Head of Information / James Hardiman Library / National University of Ireland, Galway

University College Cork, Cork

Crónán Ó Doibhlin / Sub-Librarian / Special Collections, Archives & Repository Services / George Boole Library / University College Cork

Stephen Yearl / Library Digital Projects Officer / George Boole Library / University College Cork

Ionad Bairre / University College Cork

University of Limerick, Limerick

Mícheál Ó hAodha / Librarian, Science & Engineering (CSIS, ECE, MAE, MOE) / Lewis L. Glucksman Library / University of Limerick

Brian Fitzgerald / Vice-President Research / Frederick A Krehbiel II Chair in Innovation in Global Business & Technology/ University of Limerick

Institute for the Study of Knowledge in Society / University of Limerick

Iowa State University Library / Travel and Research Funds Committee

If you are interested in information about any / all of these presentations, please contact me at gerrymck@iastate.edu

Questionnaire

In order to ascertain the particular interests of potential attendees to my presentation titled "The Paradigms Are A-Changin' > The Future Of Research And Scholarship: Open > Semantic > Social > Mobile" , I would most appreciate your responses to a questionnaire requesting attendee preferences for a particular presentation

Please Note That All Responses Are Confidential And Will Only Be Used To Focus The Topics Of My Presentation.
I created a survey for potential attendees for each of the venues noted above and ask that potential attendees only complete the survey for the individual presentation they expect to attend:

National Library of Ireland, Dublin [http://tinyurl.com/lemfnl]

National University of Irealnd, Galway [http://tinyurl.com/l3vn5y]

University College Cork [http://tinyurl.com/m3bbkh]

University of Limerick [http://tinyurl.com/kn6kb4]

I will customize a prepared general presentation to reflect the specific preferences of probable attendees at a particular venue.

By July 1 2009 I will post the customized presentations as well as a Director's Cut that combines each with additional content.

NOTE: I have also created a survey for probable non-attendees:

[http://tinyurl.com/n6behd]

Thanks for your assistance and cooperation.

/Gerry

Related

1) Open 2) Semantic 3) Social 4) Mobile

[http://scholarship20.blogspot.com/2009/04/1-open-2-semantic-3-social-4-mobile.html]