Pattern Functional Dependencies for Data Cleaning

Abdulhakim Qahtan, Nan Tang, Mourad Ouzzani, Yang Cao, Michael Stonebraker

Research output: Contribution to journalArticlepeer-review


Patterns (or regex-based expressions) are widely used to constrain the format of a domain (or a column), e.g., a Year column should contain only four digits, and thus a value like “1980-” might be a typo. Moreover, integrity constraints (ICs) defined over multiple columns, such as (conditional) functional dependencies and denial constraints, e.g., a ZIP code uniquely determines a city in the UK, have been widely used in data cleaning. However, a promising, but not yet explored, direction is to combine regex- and IC-based theories to capture data dependencies involving partial attribute values. For example, in an employee ID such as “F-9-107”, “F” is sufficient to determine the finance department.
Inspired by the above observation, we propose a novel class ofICs, called pattern functional dependencies (PFDs), to model fine-grained data dependencies gleaned from partial attribute values. These dependencies cannot be modeled using traditional ICs, such as (conditional) functional dependencies, which work on entire attribute values. We also present a set of axioms for the inference of PFDs, analogous to Armstrong’s axioms for FDs, and study the complexity of consistency and implication analysis of PFDs. Moreover, we devise an effective algorithm to automatically discover PFDs even in the presence of errors in the data. Our extensive experiments on 15 real-world datasets show that our approach can effectively discover valid and useful PFDs over dirty data, which can then be used to detect data errors that are hard to capture by other types ofICs
Original languageEnglish
Pages (from-to)684-697
Number of pages14
JournalProceedings of the VLDB Endowment (PVLDB)
Issue number5
Publication statusPublished - 31 Jan 2020
Event46th International Conference on Very Large Data Bases - Tokyo, Japan
Duration: 31 Aug 20204 Sep 2020
Conference number: 46


Dive into the research topics of 'Pattern Functional Dependencies for Data Cleaning'. Together they form a unique fingerprint.

Cite this