Projects per year
Abstract
The advent of powerful neural classifiers has increased interest in problems that require both learning and reasoning. These problems are critical for understanding important properties of models, such as trustworthiness, generalization, interpretability, and compliance to safety and structural constraints. However, recent research observed that tasks requiring both learning and reasoning on background knowledge often suffer from reasoning shortcuts (RSs): predictors can solve the downstream reasoning task without associating the correct concepts to the high-dimensional data. To address this issue, we introduce rsbench, a comprehensive benchmark suite designed to systematically evaluate the impact of RSs on models by providing easy access to highly customizable tasks affected by RSs. Furthermore, rsbench implements common metrics for evaluating concept quality and introduces novel formal verification procedures for assessing the presence of RSs in learning tasks. Using rsbench, we highlight that obtaining high quality concepts in both purely neural and neuro-symbolic models is a far-from-solved problem. rsbench is available at: https://unitn-sml.github.io/rsbench.
Original language | English |
---|---|
Title of host publication | Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems |
Publisher | ACM |
Pages | 1-44 |
Number of pages | 44 |
DOIs | |
Publication status | Accepted/In press - 26 Sept 2024 |
Event | The Thirty-Eighth Annual Conference on Neural Information Processing Systems - Vancouver Convention Center, Vancouver, Canada Duration: 10 Dec 2024 → 15 Dec 2024 Conference number: 38 https://neurips.cc/Conferences/2024 |
Conference
Conference | The Thirty-Eighth Annual Conference on Neural Information Processing Systems |
---|---|
Abbreviated title | NeurIPS 2024 |
Country/Territory | Canada |
City | Vancouver |
Period | 10/12/24 → 15/12/24 |
Internet address |
Keywords / Materials (for Non-textual outputs)
- machine learning
- artificial intelligence
Fingerprint
Dive into the research topics of 'A neuro-symbolic benchmark suite for concept quality and reasoning shortcuts'. Together they form a unique fingerprint.Projects
- 2 Active
-
-
TEAMER : Teaching Machines to Reason Like Humans
Engineering and Physical Sciences Research Council
1/10/21 → 30/09/26
Project: Research