Provenance for Seismological Processing Pipelines in a Distributed Streaming Workflow

Alessandro Spinuso, James Cheney, Malcolm Atkinson

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Harvesting provenance for streaming workflows presents challenges related to the high rate of the updates and a large distribution of the execution, which can be spread across several institutional infrastructures. Moreover, the typically large volume of data produced by each transformation step can not be always stored and preserved efficiently. This can represent an obstacle for the evaluation of the results, for instance, in real-time, suggesting the importance of customisable metadata extraction procedures. In this paper we present our approach to the aforementioned provenance challenges within a use-case driven scenario in the field of seismology, which requires the execution of processing pipelines over a large datastream. In particular, we will discuss the current implementation and the upcoming challenges for an in-worfklow programmatic approach to provenance tracing, building on composite functions, selective recording and domain specific metadata production.
Original languageEnglish
Title of host publicationProceedings of the Joint EDBT/ICDT 2013 Workshops
Place of PublicationNew York, NY, USA
PublisherACM
Pages307-312
Number of pages6
ISBN (Print)978-1-4503-1599-9
DOIs
Publication statusPublished - 2013

Publication series

NameEDBT '13
PublisherACM

Fingerprint

Dive into the research topics of 'Provenance for Seismological Processing Pipelines in a Distributed Streaming Workflow'. Together they form a unique fingerprint.

Cite this