CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities

Ayush Agrawal; Raghav Arora; Ahana Datta; Snehasis Banerjee; Brojeshwar Bhowmick; Krishna Murthy Jatavallabhula; Mohan Sridharan; Madhava Krishna

doi:10.1109/RO-MAN57019.2023.10309325

CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities

Ayush Agrawal^*, Raghav Arora, Ahana Datta, Snehasis Banerjee, Brojeshwar Bhowmick, Krishna Murthy Jatavallabhula, Mohan Sridharan, Madhava Krishna

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper introduces a novel method for determining the best room to place an object in, for embodied scene rearrangement. While state-of-the-art approaches rely on large language models (LLMs) or reinforcement learned (RL) policies for this task, our approach, CLIPGraphs, efficiently combines commonsense domain knowledge, data-driven methods, and recent advances in multimodal learning. Specifically, it (a) encodes a knowledge graph of prior human preferences about the room location of different objects in home environments, (b) incorporates vision-language features to support multimodal queries based on images or text, and (c) uses a graph network to learn object-room affinities based on embeddings of the prior knowledge and the vision-language features. We demonstrate that our approach provides better estimates of the most appropriate location of objects from a benchmark set of object categories in comparison with state-of-the-art baselines.¹¹Supplementary material and code: https://clipgraphs.github.io

Original language	English
Title of host publication	2023 32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023
Publisher	IEEE Computer Society Press
Pages	2604-2609
Number of pages	6
ISBN (Electronic)	9798350336702
DOIs	https://doi.org/10.1109/RO-MAN57019.2023.10309325
Publication status	Published - 13 Nov 2023
Event	32nd IEEE International Conference on Robot and Human Interactive Communication - Busan, Korea, Republic of Duration: 28 Aug 2023 → 31 Aug 2023 Conference number: 32 https://ro-man2023.org/overview/welcomeMessage

Publication series

Name	IEEE International Workshop on Robot and Human Communication, RO-MAN
ISSN (Print)	1944-9445
ISSN (Electronic)	1944-9437

Conference

Conference	32nd IEEE International Conference on Robot and Human Interactive Communication
Abbreviated title	IEEE RO-MAN 2023
Country/Territory	Korea, Republic of
City	Busan
Period	28/08/23 → 31/08/23
Internet address	https://ro-man2023.org/overview/welcomeMessage

Keywords / Materials (for Non-textual outputs)

Commonsense knowledge
graph convolutional network
knowledge graph
large language models
scene rearrangement

Access to Document

10.1109/RO-MAN57019.2023.10309325Licence: All Rights Reserved

https://ieeexplore.ieee.org/document/10309325/Licence: All Rights Reserved

Cite this

Agrawal, A., Arora, R., Datta, A., Banerjee, S., Bhowmick, B., Jatavallabhula, K. M., Sridharan, M., & Krishna, M. (2023). CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities. In 2023 32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023 (pp. 2604-2609). (IEEE International Workshop on Robot and Human Communication, RO-MAN). IEEE Computer Society Press. https://doi.org/10.1109/RO-MAN57019.2023.10309325

@inproceedings{bd3caab6c9f849a795dc548a52d4b3f0,

title = "CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities",

abstract = "This paper introduces a novel method for determining the best room to place an object in, for embodied scene rearrangement. While state-of-the-art approaches rely on large language models (LLMs) or reinforcement learned (RL) policies for this task, our approach, CLIPGraphs, efficiently combines commonsense domain knowledge, data-driven methods, and recent advances in multimodal learning. Specifically, it (a) encodes a knowledge graph of prior human preferences about the room location of different objects in home environments, (b) incorporates vision-language features to support multimodal queries based on images or text, and (c) uses a graph network to learn object-room affinities based on embeddings of the prior knowledge and the vision-language features. We demonstrate that our approach provides better estimates of the most appropriate location of objects from a benchmark set of object categories in comparison with state-of-the-art baselines.11Supplementary material and code: https://clipgraphs.github.io",

keywords = "Commonsense knowledge, graph convolutional network, knowledge graph, large language models, scene rearrangement",

author = "Ayush Agrawal and Raghav Arora and Ahana Datta and Snehasis Banerjee and Brojeshwar Bhowmick and Jatavallabhula, {Krishna Murthy} and Mohan Sridharan and Madhava Krishna",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 32nd IEEE International Conference on Robot and Human Interactive Communication, IEEE RO-MAN 2023 ; Conference date: 28-08-2023 Through 31-08-2023",

year = "2023",

month = nov,

day = "13",

doi = "10.1109/RO-MAN57019.2023.10309325",

language = "English",

series = "IEEE International Workshop on Robot and Human Communication, RO-MAN",

publisher = "IEEE Computer Society Press",

pages = "2604--2609",

booktitle = "2023 32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023",

url = "https://ro-man2023.org/overview/welcomeMessage",

}

Agrawal, A, Arora, R, Datta, A, Banerjee, S, Bhowmick, B, Jatavallabhula, KM, Sridharan, M & Krishna, M 2023, CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities. in 2023 32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023. IEEE International Workshop on Robot and Human Communication, RO-MAN, IEEE Computer Society Press, pp. 2604-2609, 32nd IEEE International Conference on Robot and Human Interactive Communication, Busan, Korea, Republic of, 28/08/23. https://doi.org/10.1109/RO-MAN57019.2023.10309325

CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities. / Agrawal, Ayush; Arora, Raghav; Datta, Ahana et al.
2023 32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023. IEEE Computer Society Press, 2023. p. 2604-2609 (IEEE International Workshop on Robot and Human Communication, RO-MAN).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - CLIPGraphs

T2 - 32nd IEEE International Conference on Robot and Human Interactive Communication

AU - Agrawal, Ayush

AU - Arora, Raghav

AU - Datta, Ahana

AU - Banerjee, Snehasis

AU - Bhowmick, Brojeshwar

AU - Jatavallabhula, Krishna Murthy

AU - Sridharan, Mohan

AU - Krishna, Madhava

N1 - Conference code: 32

PY - 2023/11/13

Y1 - 2023/11/13

N2 - This paper introduces a novel method for determining the best room to place an object in, for embodied scene rearrangement. While state-of-the-art approaches rely on large language models (LLMs) or reinforcement learned (RL) policies for this task, our approach, CLIPGraphs, efficiently combines commonsense domain knowledge, data-driven methods, and recent advances in multimodal learning. Specifically, it (a) encodes a knowledge graph of prior human preferences about the room location of different objects in home environments, (b) incorporates vision-language features to support multimodal queries based on images or text, and (c) uses a graph network to learn object-room affinities based on embeddings of the prior knowledge and the vision-language features. We demonstrate that our approach provides better estimates of the most appropriate location of objects from a benchmark set of object categories in comparison with state-of-the-art baselines.11Supplementary material and code: https://clipgraphs.github.io

AB - This paper introduces a novel method for determining the best room to place an object in, for embodied scene rearrangement. While state-of-the-art approaches rely on large language models (LLMs) or reinforcement learned (RL) policies for this task, our approach, CLIPGraphs, efficiently combines commonsense domain knowledge, data-driven methods, and recent advances in multimodal learning. Specifically, it (a) encodes a knowledge graph of prior human preferences about the room location of different objects in home environments, (b) incorporates vision-language features to support multimodal queries based on images or text, and (c) uses a graph network to learn object-room affinities based on embeddings of the prior knowledge and the vision-language features. We demonstrate that our approach provides better estimates of the most appropriate location of objects from a benchmark set of object categories in comparison with state-of-the-art baselines.11Supplementary material and code: https://clipgraphs.github.io

KW - Commonsense knowledge

KW - graph convolutional network

KW - knowledge graph

KW - large language models

KW - scene rearrangement

U2 - 10.1109/RO-MAN57019.2023.10309325

DO - 10.1109/RO-MAN57019.2023.10309325

M3 - Conference contribution

AN - SCOPUS:85179525078

T3 - IEEE International Workshop on Robot and Human Communication, RO-MAN

SP - 2604

EP - 2609

BT - 2023 32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023

PB - IEEE Computer Society Press

Y2 - 28 August 2023 through 31 August 2023

ER -

Agrawal A, Arora R, Datta A, Banerjee S, Bhowmick B, Jatavallabhula KM et al. CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities. In 2023 32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023. IEEE Computer Society Press. 2023. p. 2604-2609. (IEEE International Workshop on Robot and Human Communication, RO-MAN). doi: 10.1109/RO-MAN57019.2023.10309325

CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities

Abstract

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this