Can Common Crawl Reliably Track Persistent Identifier (PID) Use Over Time

Henry S. Thompson; Jian Tong

doi:10.1145/3184558.3191636

Can Common Crawl Reliably Track Persistent Identifier (PID) Use Over Time

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We report here on the results of two studies using two and four monthly web crawls respectively from the Common Crawl (CC) initiative between 2014 and 2017, whose initial goal was to provide empirical evidence for the changing patterns of use of so-called persistent identifiers. This paper focusses on the tooling needed for dealing with CC data, and the problems we found with it. The first study is based on over 10¹² URIs from over 5x10⁹ pages crawled in April 2014 and April 2017, the second study adds a further 3x10⁹pages from the April 2015 and April 2016 crawls. We conclude with suggestions on specific actions needed to enable studies based on CC to give reliable longitudinal information.

Original language	English
Title of host publication	Companion Proceedings of the The Web Conference 2018
Place of Publication	Lyon, France
Publisher	International World Wide Web Conferences Steering Committee
Pages	1749-1755
Number of pages	7
ISBN (Electronic)	978-1-4503-5640-4
DOIs	https://doi.org/10.1145/3184558.3191636
Publication status	Published - 23 Apr 2018
Event	The Web Conference 2018 - Lyon, France Duration: 23 Apr 2018 → 27 Apr 2018 https://www2018.thewebconf.org/

Conference

Conference	The Web Conference 2018
Abbreviated title	TheWebConf 2018
Country/Territory	France
City	Lyon
Period	23/04/18 → 27/04/18
Internet address	https://www2018.thewebconf.org/

Keywords / Materials (for Non-textual outputs)

temporal web analytics
Digital Object Identifier
longitudinal web crawl analysis
Uniform Resource Identifier
persistent identifier
Common Crawl

Access to Document

10.1145/3184558.3191636Licence: Creative Commons: Attribution (CC-BY)

Can Common Crawl Reliably Track PIDFinal published version, 1.44 MBLicence: Creative Commons: Attribution (CC-BY)

https://dl.acm.org/doi/10.1145/3184558.3191636Licence: Creative Commons: Attribution (CC-BY)

Cite this

@inproceedings{f88a72b387c746aab18282e7ebf2424c,

title = "Can Common Crawl Reliably Track Persistent Identifier (PID) Use Over Time",

abstract = "We report here on the results of two studies using two and four monthly web crawls respectively from the Common Crawl (CC) initiative between 2014 and 2017, whose initial goal was to provide empirical evidence for the changing patterns of use of so-called persistent identifiers. This paper focusses on the tooling needed for dealing with CC data, and the problems we found with it. The first study is based on over 1012 URIs from over 5x109 pages crawled in April 2014 and April 2017, the second study adds a further 3x109 pages from the April 2015 and April 2016 crawls. We conclude with suggestions on specific actions needed to enable studies based on CC to give reliable longitudinal information. ",

keywords = "temporal web analytics, Digital Object Identifier, longitudinal web crawl analysis, Uniform Resource Identifier, persistent identifier, Common Crawl",

author = "Thompson, {Henry S.} and Jian Tong",

year = "2018",

month = apr,

day = "23",

doi = "10.1145/3184558.3191636",

language = "English",

pages = "1749--1755",

booktitle = "Companion Proceedings of the The Web Conference 2018",

publisher = "International World Wide Web Conferences Steering Committee",

note = "The Web Conference 2018, TheWebConf 2018 ; Conference date: 23-04-2018 Through 27-04-2018",

url = "https://www2018.thewebconf.org/",

}

TY - GEN

T1 - Can Common Crawl Reliably Track Persistent Identifier (PID) Use Over Time

AU - Thompson, Henry S.

AU - Tong, Jian

PY - 2018/4/23

Y1 - 2018/4/23

N2 - We report here on the results of two studies using two and four monthly web crawls respectively from the Common Crawl (CC) initiative between 2014 and 2017, whose initial goal was to provide empirical evidence for the changing patterns of use of so-called persistent identifiers. This paper focusses on the tooling needed for dealing with CC data, and the problems we found with it. The first study is based on over 1012 URIs from over 5x109 pages crawled in April 2014 and April 2017, the second study adds a further 3x109 pages from the April 2015 and April 2016 crawls. We conclude with suggestions on specific actions needed to enable studies based on CC to give reliable longitudinal information.

AB - We report here on the results of two studies using two and four monthly web crawls respectively from the Common Crawl (CC) initiative between 2014 and 2017, whose initial goal was to provide empirical evidence for the changing patterns of use of so-called persistent identifiers. This paper focusses on the tooling needed for dealing with CC data, and the problems we found with it. The first study is based on over 1012 URIs from over 5x109 pages crawled in April 2014 and April 2017, the second study adds a further 3x109 pages from the April 2015 and April 2016 crawls. We conclude with suggestions on specific actions needed to enable studies based on CC to give reliable longitudinal information.

KW - temporal web analytics

KW - Digital Object Identifier

KW - longitudinal web crawl analysis

KW - Uniform Resource Identifier

KW - persistent identifier

KW - Common Crawl

UR - https://doi.org/10.1145/3184558.3191636

U2 - 10.1145/3184558.3191636

DO - 10.1145/3184558.3191636

M3 - Conference contribution

SP - 1749

EP - 1755

BT - Companion Proceedings of the The Web Conference 2018

PB - International World Wide Web Conferences Steering Committee

CY - Lyon, France

T2 - The Web Conference 2018

Y2 - 23 April 2018 through 27 April 2018

ER -

Can Common Crawl Reliably Track Persistent Identifier (PID) Use Over Time

Abstract

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Other files and links

Fingerprint

Henry Thompson

Cite this

Can Common Crawl Reliably Track Persistent Identifier (PID) Use Over Time

Abstract

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Other files and links

Fingerprint

Profiles

Henry Thompson

Cite this