Eris: Measuring discord among multidimensional data sources

Alberto Abelló, James Cheney

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

Data integration is a classical problem in databases, typically decomposed into schema matching, entity matching and data fusion. To solve the latter, it is mostly assumed that ground truth can be determined. However, in general, the data gathering processes in the different sources are imperfect and cannot provide an accurate merging of values. Thus, in the absence of ways to determine ground truth, it is important to at least quantify how far from being internally consistent a dataset is. Hence, we propose definitions of concordant data and define a discordance metric as a way of measuring disagreement to improve decision making based on trustworthiness. We define the discord measurement problem of numerical attributes in which given a set of uncertain raw observations or aggregate results (such as case/hospitalization/death data relevant to COVID-19) and information on the alignment of different conceptualizations of the same reality (e.g., granularities or units), we wish to assess whether the different sources are concordant, or if not, use the discordance metric to quantify how discordant they are. We also define a set of algebraic operators to describe the alignments of different data sources with correctness guarantees, together with two alternative relational database implementations that reduce the problem to linear or quadratic programming. These are evaluated against both COVID-19 and synthetic data, and our experimental results show that discordance measurement can be performed efficiently in realistic situations.
Original languageEnglish
Pages (from-to)399-423
Number of pages25
JournalVLDB Journal
Volume33
Issue number2
Early online date20 Sept 2023
DOIs
Publication statusPublished - 1 Mar 2024

Keywords / Materials (for Non-textual outputs)

  • cs.DB
  • 68P15
  • H.2.4

Fingerprint

Dive into the research topics of 'Eris: Measuring discord among multidimensional data sources'. Together they form a unique fingerprint.

Cite this