defoe: A Spark-based Toolbox for Analysing Digital Historical Textual Data

Rosa Filgueira Vicente; Michael Jackson; Anna Roubickova; Amrey Krause; Ruth Ahnert; Tessa Hauswedell; Julianne Nyhan; David Beavan; Timothy Hobson; Mariona Coll Ardanuy; Giovanni Colavizza; James Hetherington; Melissa Terras

doi:10.1109/eScience.2019.00033

defoe: A Spark-based Toolbox for Analysing Digital Historical Textual Data

Rosa Filgueira Vicente, Michael Jackson, Anna Roubickova, Amrey Krause, Ruth Ahnert, Tessa Hauswedell, Julianne Nyhan, David Beavan, Timothy Hobson, Mariona Coll Ardanuy, Giovanni Colavizza, James Hetherington, Melissa Terras

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.

Original language	English
Title of host publication	2019 IEEE 15th International Conference on e-Science (e-Science)
Publisher	Institute of Electrical and Electronics Engineers
Pages	235-242
Number of pages	8
ISBN (Electronic)	978-1-7281-2451-3
ISBN (Print)	978-1-7281-2452-0
DOIs	https://doi.org/10.1109/eScience.2019.00033
Publication status	Published - 19 Mar 2020
Event	2019 IEEE 15th International Conference on e-Science (e-Science) - San Diego, United States Duration: 24 Sept 2019 → 27 Sept 2019 https://escience2019.sdsc.edu/

Conference

Conference	2019 IEEE 15th International Conference on e-Science (e-Science)
Abbreviated title	e-Science 2019
Country/Territory	United States
City	San Diego
Period	24/09/19 → 27/09/19
Internet address	https://escience2019.sdsc.edu/

Keywords / Materials (for Non-textual outputs)

text mining
distributed queries
Apache Spark
High-Performance Computing
XML schemas
digital tools
digitised primary historical sources
humanities research

Access to Document

10.1109/eScience.2019.00033Licence: All Rights Reserved

eScience_2019_paper_28_easychairAccepted author manuscript, 2.56 MBLicence: Unspecified

https://ieeexplore.ieee.org/document/9041813Licence: All Rights Reserved

Rosa Filgueira
- EPCC - Chancellor's Fellow - AI & Datascience
Person: Academic: Research Active

Cite this

Filgueira Vicente, R., Jackson, M., Roubickova, A., Krause, A., Ahnert, R., Hauswedell, T., Nyhan, J., Beavan, D., Hobson, T., Coll Ardanuy, M., Colavizza, G., Hetherington, J., & Terras, M. (2020). defoe: A Spark-based Toolbox for Analysing Digital Historical Textual Data. In 2019 IEEE 15th International Conference on e-Science (e-Science) (pp. 235-242). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/eScience.2019.00033

@inproceedings{a9682a06d1594d009f92f9a27f0bb385,

title = "defoe: A Spark-based Toolbox for Analysing Digital Historical Textual Data",

abstract = "This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.",

keywords = "text mining, distributed queries, Apache Spark, High-Performance Computing, XML schemas, digital tools, digitised primary historical sources, humanities research",

author = "{Filgueira Vicente}, Rosa and Michael Jackson and Anna Roubickova and Amrey Krause and Ruth Ahnert and Tessa Hauswedell and Julianne Nyhan and David Beavan and Timothy Hobson and {Coll Ardanuy}, Mariona and Giovanni Colavizza and James Hetherington and Melissa Terras",

year = "2020",

month = mar,

day = "19",

doi = "10.1109/eScience.2019.00033",

language = "English",

isbn = "978-1-7281-2452-0",

pages = "235--242",

booktitle = "2019 IEEE 15th International Conference on e-Science (e-Science)",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

note = "2019 IEEE 15th International Conference on e-Science (e-Science), e-Science 2019 ; Conference date: 24-09-2019 Through 27-09-2019",

url = "https://escience2019.sdsc.edu/",

}

Filgueira Vicente, R , Jackson, M , Roubickova, A , Krause, A, Ahnert, R, Hauswedell, T, Nyhan, J, Beavan, D, Hobson, T, Coll Ardanuy, M, Colavizza, G, Hetherington, J & Terras, M 2020, defoe: A Spark-based Toolbox for Analysing Digital Historical Textual Data. in 2019 IEEE 15th International Conference on e-Science (e-Science). Institute of Electrical and Electronics Engineers, pp. 235-242, 2019 IEEE 15th International Conference on e-Science (e-Science), San Diego, California, United States, 24/09/19. https://doi.org/10.1109/eScience.2019.00033

TY - GEN

T1 - defoe: A Spark-based Toolbox for Analysing Digital Historical Textual Data

AU - Filgueira Vicente, Rosa

AU - Jackson, Michael

AU - Roubickova, Anna

AU - Krause, Amrey

AU - Ahnert, Ruth

AU - Hauswedell, Tessa

AU - Nyhan, Julianne

AU - Beavan, David

AU - Hobson, Timothy

AU - Coll Ardanuy, Mariona

AU - Colavizza, Giovanni

AU - Hetherington, James

AU - Terras, Melissa

PY - 2020/3/19

Y1 - 2020/3/19

N2 - This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.

AB - This work presents defoe, a new scalable and portable digital eScience toolbox that enables historical research. It allows for running text mining queries across large datasets, such as historical newspapers and books in parallel via Apache Spark. It handles queries against collections that comprise several XML schemas and physical representations. The proposed tool has been successfully evaluated using five different large-scale historical text datasets and two HPC environments, as well as on desktops. Results shows that defoe allows researchers to query multiple datasets in parallel from a single command-line interface and in a consistent way, without any HPC environment-specific requirements.

KW - text mining

KW - distributed queries

KW - Apache Spark

KW - High-Performance Computing

KW - XML schemas

KW - digital tools

KW - digitised primary historical sources

KW - humanities research

U2 - 10.1109/eScience.2019.00033

DO - 10.1109/eScience.2019.00033

M3 - Conference contribution

SN - 978-1-7281-2452-0

SP - 235

EP - 242

BT - 2019 IEEE 15th International Conference on e-Science (e-Science)

PB - Institute of Electrical and Electronics Engineers

T2 - 2019 IEEE 15th International Conference on e-Science (e-Science)

Y2 - 24 September 2019 through 27 September 2019

ER -

defoe: A Spark-based Toolbox for Analysing Digital Historical Textual Data

Abstract / Description of output

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Profiles

Rosa Filgueira

Cite this