Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation

Arushi Goel, Basura Fernando, Frank Keller, Hakan Bilen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Scene graph generation (SGG) aims to capture a wide variety of interactions between pairs of objects, which is essential for full scene understanding. Existing SGG methods trained on the entire set of relations fail to acquire complex reasoning about visual and textual correlations due to various biases in training data. Learning on trivial relations that indicate generic spatial configuration like 'on' instead of informative relations such as 'parked on' does not enforce this complex reasoning, harming generalization. To address this problem, we propose a novel framework for SGG training that exploits relation labels based on their informativeness. Our model-agnostic training procedure imputes missing informative relations for less informative samples in the training data and trains a SGG model on the imputed labels along with existing annotations. We show that this approach can successfully be used in conjunction with state-of-the-art SGG methods and improves their performance significantly in multiple metrics on the standard Visual Genome benchmark. Furthermore, we obtain considerable improvements for unseen triplets in a more challenging zero-shot setting.
Original languageEnglish
Title of host publicationProceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherInstitute of Electrical and Electronics Engineers
Pages15575-15585
Number of pages11
ISBN (Electronic)978-1-6654-6946-3
ISBN (Print)978-1-6654-6947-0
DOIs
Publication statusPublished - 27 Sept 2022
EventIEEE/CVF Conference on Computer Vision and Pattern Recognition 2022
- New Orleans, United States
Duration: 19 Jun 202224 Jun 2022
https://cvpr2022.thecvf.com/

Publication series

NameIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherIEEE
ISSN (Print)1063-6919
ISSN (Electronic)2575-7075

Conference

ConferenceIEEE/CVF Conference on Computer Vision and Pattern Recognition 2022
Abbreviated titleCVPR 2022
Country/TerritoryUnited States
CityNew Orleans
Period19/06/2224/06/22
Internet address

Keywords / Materials (for Non-textual outputs)

  • scene graph generation
  • semi-supervised learning

Fingerprint

Dive into the research topics of 'Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation'. Together they form a unique fingerprint.

Cite this