ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies

Abdulhakim Qahtan; Nan Tang; Mourad Ouzzani; Yang Cao; Michael Stonebraker

doi:10.1145/3299869.3320209

ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies

Abdulhakim Qahtan, Nan Tang, Mourad Ouzzani, Yang Cao, Michael Stonebraker

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Knowledge discovery is critical to successful data analytics. We propose a new type of meta-knowledge, namely pattern functional dependencies (PFDs), that combine patterns (or regex-like rules) and integrity constraints (ICs) to model the dependencies (or meta-knowledge) between partial values (or patterns) across different attributes in a table. PFDs go beyond the classical functional dependencies and their ex- tensions. For instance, in an employee table, ID “F-9-107”, “F” determines the financial department, and “9” determines one’s grade. Moreover, a key application of PFDs is to use them to identify erroneous data; tuples that violate some PFDs. In this demonstration, attendees will experience the following features: PFD discovery – automatically discover PFDs from (dirty) data in different domains; and Error detection with PFDs – we will show errors that are detected by PFDs but cannot be captured by existing approaches.

Original language	English
Title of host publication	Proceedings of the 2019 International Conference on Management of Data
Place of Publication	New York
Publisher	ACM
Pages	1977-1980
Number of pages	4
ISBN (Print)	978-1-4503-5643-5
DOIs	https://doi.org/10.1145/3299869.3320209
Publication status	Published - 25 Jun 2019
Event	ACM SIGMOD/PODS International Conference on Management of Data (SIGMOD 2019) - Amsterdam, Netherlands Duration: 30 Jun 2019 → 5 Jul 2019 http://sigmod2019.org/

Conference

Conference	ACM SIGMOD/PODS International Conference on Management of Data (SIGMOD 2019)
Abbreviated title	SIGMOD 2019
Country/Territory	Netherlands
City	Amsterdam
Period	30/06/19 → 5/07/19
Internet address	http://sigmod2019.org/

Keywords / Materials (for Non-textual outputs)

data cleaning
Pattern Functional Dependencies
Constrained Patterns
Error Detection
Knowledge Discovery

Access to Document

10.1145/3299869.3320209

ANMAT Automatic Knowledge_QAHTAN_DoA090319_AFVAccepted author manuscript, 976 KB

https://dl.acm.org/citation.cfm?doid=3299869.3320209

Cite this

@inproceedings{eb066018bd2447f488b5ba6ec1b2c23e,

title = "ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies",

abstract = "Knowledge discovery is critical to successful data analytics. We propose a new type of meta-knowledge, namely pattern functional dependencies (PFDs), that combine patterns (or regex-like rules) and integrity constraints (ICs) to model the dependencies (or meta-knowledge) between partial values (or patterns) across different attributes in a table. PFDs go beyond the classical functional dependencies and their ex- tensions. For instance, in an employee table, ID “F-9-107”, “F” determines the financial department, and “9” determines one{\textquoteright}s grade. Moreover, a key application of PFDs is to use them to identify erroneous data; tuples that violate some PFDs. In this demonstration, attendees will experience the following features: PFD discovery – automatically discover PFDs from (dirty) data in different domains; and Error detection with PFDs – we will show errors that are detected by PFDs but cannot be captured by existing approaches.",

keywords = "data cleaning, Pattern Functional Dependencies, Constrained Patterns, Error Detection, Knowledge Discovery",

author = "Abdulhakim Qahtan and Nan Tang and Mourad Ouzzani and Yang Cao and Michael Stonebraker",

year = "2019",

month = jun,

day = "25",

doi = "10.1145/3299869.3320209",

language = "English",

isbn = "978-1-4503-5643-5",

pages = "1977--1980",

booktitle = "Proceedings of the 2019 International Conference on Management of Data",

publisher = "ACM",

note = "ACM SIGMOD/PODS International Conference on Management of Data (SIGMOD 2019), SIGMOD 2019 ; Conference date: 30-06-2019 Through 05-07-2019",

url = "http://sigmod2019.org/",

}

Qahtan, A, Tang, N, Ouzzani, M, Cao, Y & Stonebraker, M 2019, ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies. in Proceedings of the 2019 International Conference on Management of Data. ACM, New York, pp. 1977-1980, ACM SIGMOD/PODS International Conference on Management of Data (SIGMOD 2019), Amsterdam, Netherlands, 30/06/19. https://doi.org/10.1145/3299869.3320209

TY - GEN

T1 - ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies

AU - Qahtan, Abdulhakim

AU - Tang, Nan

AU - Ouzzani, Mourad

AU - Cao, Yang

AU - Stonebraker, Michael

PY - 2019/6/25

Y1 - 2019/6/25

N2 - Knowledge discovery is critical to successful data analytics. We propose a new type of meta-knowledge, namely pattern functional dependencies (PFDs), that combine patterns (or regex-like rules) and integrity constraints (ICs) to model the dependencies (or meta-knowledge) between partial values (or patterns) across different attributes in a table. PFDs go beyond the classical functional dependencies and their ex- tensions. For instance, in an employee table, ID “F-9-107”, “F” determines the financial department, and “9” determines one’s grade. Moreover, a key application of PFDs is to use them to identify erroneous data; tuples that violate some PFDs. In this demonstration, attendees will experience the following features: PFD discovery – automatically discover PFDs from (dirty) data in different domains; and Error detection with PFDs – we will show errors that are detected by PFDs but cannot be captured by existing approaches.

AB - Knowledge discovery is critical to successful data analytics. We propose a new type of meta-knowledge, namely pattern functional dependencies (PFDs), that combine patterns (or regex-like rules) and integrity constraints (ICs) to model the dependencies (or meta-knowledge) between partial values (or patterns) across different attributes in a table. PFDs go beyond the classical functional dependencies and their ex- tensions. For instance, in an employee table, ID “F-9-107”, “F” determines the financial department, and “9” determines one’s grade. Moreover, a key application of PFDs is to use them to identify erroneous data; tuples that violate some PFDs. In this demonstration, attendees will experience the following features: PFD discovery – automatically discover PFDs from (dirty) data in different domains; and Error detection with PFDs – we will show errors that are detected by PFDs but cannot be captured by existing approaches.

KW - data cleaning

KW - Pattern Functional Dependencies

KW - Constrained Patterns

KW - Error Detection

KW - Knowledge Discovery

U2 - 10.1145/3299869.3320209

DO - 10.1145/3299869.3320209

M3 - Conference contribution

SN - 978-1-4503-5643-5

SP - 1977

EP - 1980

BT - Proceedings of the 2019 International Conference on Management of Data

PB - ACM

CY - New York

T2 - ACM SIGMOD/PODS International Conference on Management of Data (SIGMOD 2019)

Y2 - 30 June 2019 through 5 July 2019

ER -

ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies

Abstract

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this