A Closer Look at Probability Calibration of Knowledge Graph Embedding

Ruiqi Zhu; Fangrong Wang; Alan Bundy; Xue Li; Kwabena Nuamah; Lei Xu; Stefano Mauceri; Jeff Z. Pan

doi:10.1145/3579051.3579072

A Closer Look at Probability Calibration of Knowledge Graph Embedding

Ruiqi Zhu, Fangrong Wang, Alan Bundy, Xue Li, Kwabena Nuamah, Lei Xu, Stefano Mauceri, Jeff Z. Pan^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

When the estimated probabilities do not match the relative frequencies, we say these estimated probabilities are uncalibrated [39], which may cause incorrect decision making, and is particularly undesired in high-stakes tasks [45]. Knowledge Graph embedding models are reported to produce uncalibrated probabilities [36], e.g., for all the triples predicted with probability 0.9, the percentage of them being truly correct triples is not . In this article, we take a closer look at this problem. First, we confirmed the issue that typical KG Embedding models are uncalibrated. Then, we show how off-The-shelf calibration techniques can be used to mitigate this issue, among which binning-based calibration produces more calibrated probabilities. We also investigated the possible reasons for the uncalibrated probabilities and found that the expit transform, the way used to convert embedding scores into probabilities, is ineffective in most cases.

Original language	English
Title of host publication	Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022
Editors	Alessandro Artale, Diego Calvanese, Haofen Wang, Xiaowang Zhang
Publisher	Association for Computing Machinery
Pages	104-109
Number of pages	6
ISBN (Electronic)	9781450399876
DOIs	https://doi.org/10.1145/3579051.3579072
Publication status	Published - 13 Feb 2023
Event	11th International Joint Conference on Knowledge Graphs, IJCKG 2022 - Virtual, Online, China Duration: 27 Oct 2022 → 28 Oct 2022

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	11th International Joint Conference on Knowledge Graphs, IJCKG 2022
Country/Territory	China
City	Virtual, Online
Period	27/10/22 → 28/10/22

Keywords / Materials (for Non-textual outputs)

Knowledge Graph Embedding
Probability Calibration

Access to Document

10.1145/3579051.3579072Licence: All Rights Reserved

A Closer Look_ZHU_DOA02092022_AFV_CC BYAccepted author manuscript, 637 KBLicence: Creative Commons: Attribution (CC-BY)

https://dl.acm.org/doi/10.1145/3579051.3579072Licence: All Rights Reserved

1 Finished
1 Active

TEAMER : Teaching Machines to Reason Like Humans
Lapata, M.
UK central government bodies/local authorities, health and hospital authorities
1/10/21 → 30/09/26
Project: Research
UKRI Trustworthy Autonomous Systems Node in Governance and Regulation
Ramamoorthy, R., Belle, V., Bundy, A., Jackson, P., Lascarides, A. & Rajan, A.
EPSRC
1/11/20 → 30/04/24
Project: Research

Cite this

Zhu, R., Wang, F., Bundy, A., Li, X., Nuamah, K., Xu, L., Mauceri, S., & Pan, J. Z. (2023). A Closer Look at Probability Calibration of Knowledge Graph Embedding. In A. Artale, D. Calvanese, H. Wang, & X. Zhang (Eds.), Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022 (pp. 104-109). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3579051.3579072

Zhu, Ruiqi ; Wang, Fangrong ; Bundy, Alan et al. / A Closer Look at Probability Calibration of Knowledge Graph Embedding. Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022. editor / Alessandro Artale ; Diego Calvanese ; Haofen Wang ; Xiaowang Zhang. Association for Computing Machinery, 2023. pp. 104-109 (ACM International Conference Proceeding Series).

@inproceedings{0336e962d9ec4f10a29c64ed00df5c61,

title = "A Closer Look at Probability Calibration of Knowledge Graph Embedding",

abstract = "When the estimated probabilities do not match the relative frequencies, we say these estimated probabilities are uncalibrated [39], which may cause incorrect decision making, and is particularly undesired in high-stakes tasks [45]. Knowledge Graph embedding models are reported to produce uncalibrated probabilities [36], e.g., for all the triples predicted with probability 0.9, the percentage of them being truly correct triples is not . In this article, we take a closer look at this problem. First, we confirmed the issue that typical KG Embedding models are uncalibrated. Then, we show how off-The-shelf calibration techniques can be used to mitigate this issue, among which binning-based calibration produces more calibrated probabilities. We also investigated the possible reasons for the uncalibrated probabilities and found that the expit transform, the way used to convert embedding scores into probabilities, is ineffective in most cases.",

keywords = "Knowledge Graph Embedding, Probability Calibration",

author = "Ruiqi Zhu and Fangrong Wang and Alan Bundy and Xue Li and Kwabena Nuamah and Lei Xu and Stefano Mauceri and Pan, {Jeff Z.}",

note = "Funding Information: The authors would like to thank Huawei for supporting the research on which this paper was based under grant CIENG4721/LSC. We also acknowledge the support of UKRI grant EP/V026607/1, ELIAI (The Edinburgh Laboratory for Integrated Artificial Intelligence) and EPSRC grant no EP/W002876/1. Additional appreciation to the anonymous reviewers who offered useful comments for improving the quality of the paper. ; 11th International Joint Conference on Knowledge Graphs, IJCKG 2022 ; Conference date: 27-10-2022 Through 28-10-2022",

year = "2023",

month = feb,

day = "13",

doi = "10.1145/3579051.3579072",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "104--109",

editor = "Alessandro Artale and Diego Calvanese and Haofen Wang and Xiaowang Zhang",

booktitle = "Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022",

}

Zhu, R, Wang, F, Bundy, A , Li, X , Nuamah, K, Xu, L, Mauceri, S & Pan, JZ 2023, A Closer Look at Probability Calibration of Knowledge Graph Embedding. in A Artale, D Calvanese, H Wang & X Zhang (eds), Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 104-109, 11th International Joint Conference on Knowledge Graphs, IJCKG 2022, Virtual, Online, China, 27/10/22. https://doi.org/10.1145/3579051.3579072

A Closer Look at Probability Calibration of Knowledge Graph Embedding. / Zhu, Ruiqi; Wang, Fangrong; Bundy, Alan et al.
Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022. ed. / Alessandro Artale; Diego Calvanese; Haofen Wang; Xiaowang Zhang. Association for Computing Machinery, 2023. p. 104-109 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Closer Look at Probability Calibration of Knowledge Graph Embedding

AU - Zhu, Ruiqi

AU - Wang, Fangrong

AU - Bundy, Alan

AU - Li, Xue

AU - Nuamah, Kwabena

AU - Xu, Lei

AU - Mauceri, Stefano

AU - Pan, Jeff Z.

N1 - Funding Information: The authors would like to thank Huawei for supporting the research on which this paper was based under grant CIENG4721/LSC. We also acknowledge the support of UKRI grant EP/V026607/1, ELIAI (The Edinburgh Laboratory for Integrated Artificial Intelligence) and EPSRC grant no EP/W002876/1. Additional appreciation to the anonymous reviewers who offered useful comments for improving the quality of the paper.

PY - 2023/2/13

Y1 - 2023/2/13

N2 - When the estimated probabilities do not match the relative frequencies, we say these estimated probabilities are uncalibrated [39], which may cause incorrect decision making, and is particularly undesired in high-stakes tasks [45]. Knowledge Graph embedding models are reported to produce uncalibrated probabilities [36], e.g., for all the triples predicted with probability 0.9, the percentage of them being truly correct triples is not . In this article, we take a closer look at this problem. First, we confirmed the issue that typical KG Embedding models are uncalibrated. Then, we show how off-The-shelf calibration techniques can be used to mitigate this issue, among which binning-based calibration produces more calibrated probabilities. We also investigated the possible reasons for the uncalibrated probabilities and found that the expit transform, the way used to convert embedding scores into probabilities, is ineffective in most cases.

AB - When the estimated probabilities do not match the relative frequencies, we say these estimated probabilities are uncalibrated [39], which may cause incorrect decision making, and is particularly undesired in high-stakes tasks [45]. Knowledge Graph embedding models are reported to produce uncalibrated probabilities [36], e.g., for all the triples predicted with probability 0.9, the percentage of them being truly correct triples is not . In this article, we take a closer look at this problem. First, we confirmed the issue that typical KG Embedding models are uncalibrated. Then, we show how off-The-shelf calibration techniques can be used to mitigate this issue, among which binning-based calibration produces more calibrated probabilities. We also investigated the possible reasons for the uncalibrated probabilities and found that the expit transform, the way used to convert embedding scores into probabilities, is ineffective in most cases.

KW - Knowledge Graph Embedding

KW - Probability Calibration

U2 - 10.1145/3579051.3579072

DO - 10.1145/3579051.3579072

M3 - Conference contribution

AN - SCOPUS:85148541024

T3 - ACM International Conference Proceeding Series

SP - 104

EP - 109

BT - Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022

A2 - Artale, Alessandro

A2 - Calvanese, Diego

A2 - Wang, Haofen

A2 - Zhang, Xiaowang

PB - Association for Computing Machinery

T2 - 11th International Joint Conference on Knowledge Graphs, IJCKG 2022

Y2 - 27 October 2022 through 28 October 2022

ER -

Zhu R, Wang F, Bundy A , Li X , Nuamah K, Xu L et al. A Closer Look at Probability Calibration of Knowledge Graph Embedding. In Artale A, Calvanese D, Wang H, Zhang X, editors, Proceedings of the 11th International Joint Conference on Knowledge Graphs, IJCKG 2022. Association for Computing Machinery. 2023. p. 104-109. (ACM International Conference Proceeding Series). Epub 2022 Oct 27. doi: 10.1145/3579051.3579072

A Closer Look at Probability Calibration of Knowledge Graph Embedding

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Projects

TEAMER : Teaching Machines to Reason Like Humans

UKRI Trustworthy Autonomous Systems Node in Governance and Regulation

Cite this