Large-scale Incremental Data Processing with Change Propagation

Pramod Bhatotia, Alexander Wieder, İstemi Ekin Akkuş, Rodrigo Rodrigues, Umut A. Acar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Incremental processing of large-scale data is an increasingly important problem, given that many processing jobs run repeatedly with similar inputs, and that the de facto standard programming model (MapReduce) was not designed to efficiently process small updates. As a result, new systems specifically targeting this problem (e.g., Google Percolator, or Yahoo! CBP) have been proposed. Unfortunately, these approaches require the adoption of a new programming model, breaking compatibility with existing programs, and increasing the burden on the programmer, who now is required to devise an incremental update mechanism. We claim that automatic incremental processing of large-scale data is possible by leveraging previous results from the algorithms and programming languages communities. As an example, we describe how MapReduce can be improved to efficiently handle small input changes by automatically incrementalizing existing MapReduce computations, without breaking backward compatibility or demanding programmers to adopt a new programming approach.
Original languageEnglish
Title of host publicationProceedings of the 3rd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 11)
Number of pages5
Publication statusPublished - Jun 2011


Dive into the research topics of 'Large-scale Incremental Data Processing with Change Propagation'. Together they form a unique fingerprint.

Cite this