Open X-Embodiment: Robotic learning datasets and RT-X models

Open X-Embodiment Collaboration; Abby O’Neill; Abdul Rehman; Abhiram Maddukuri; Abhishek Gupta; Abhishek Padalkar; Abraham Lee; Acorn Pooley; Agrim Gupta; Ajay Mandlekar; Ajinkya Jain; Albert Tung; Alex Bewley; Alex Herzog; Alex Irpan; Alexander Khazatsky; Anant Rai; Anchit Gupta; Andrew Wang; Anikait Singh; Animesh Garg; Aniruddha Kembhavi; Annie Xie; Anthony Brohan; Antonin Raffin; Archit Sharma; Arefeh Yavary; Arhan Jain; Ashwin Balakrishna; Ayzaan Wahid; Ben Burgess-Limerick; Beomjoon Kim; Bernhard Schölkopf; Blake Wulfe; Brian Ichter; Cewu Lu; Charles Xu; Charlotte Le; Chelsea Finn; Chen Wang; Chenfeng Xu; Cheng Chi; Chenguang Huang; Christine Chan; Christopher Agia; Chuer Pan; Chuyuan Fu; Coline Devin; Danfei Xu; Daniel Morton; Danny Driess; Daphne Chen; Deepak Pathak; Dhruv Shah; Dieter Büchler; Dinesh Jayaraman; Dmitry Kalashnikov; Dorsa Sadigh; Edward Johns; Ethan Foster; Fangchen Liu; Federico Ceola; Fei Xia; Feiyu Zhao; Freek Stulp; Gaoyue Zhou; Gaurav S. Sukhatme; Gautam Salhotra; Ge Yan; Gilbert Feng; Giulio Schiavi; Glen Berseth; Gregory Kahn; Guanzhi Wang; Hao Su; Hao-Shu Fang; Haochen Shi; Henghui Bao; Heni Ben Amor; Henrik I. Christensen; Hiroki Furuta; Homer Walke; Hongjie Fang; Huy Ha; Igor Mordatch; Ilija Radosavovic; Isabel Leal; Jacky Liang; Jad Abou-Chakra; Jaehyung Kim; Jaimyn Drake; Jan Peters; Jan Schneider; Jasmine Hsu; Jeannette Bohg; Jeffrey Bingham; Jeffrey Wu; Jensen Gao; Jiaheng Hu; Jiajun Wu; Jialin Wu; Jiankai Sun; Jianlan Luo; Jiayuan Gu; Jie Tan; Jihoon Oh; Jimmy Wu; Jingpei Lu; Jingyun Yang; Jitendra Malik; João Silvério; Joey Hejna; Jonathan Booher; Jonathan Tompson; Jonathan Yang; Jordi Salvador; Joseph J. Lim; Junhyek Han; Kaiyuan Wang; Kanishka Rao; Karl Pertsch; Karol Hausman; Keegan Go; Keerthana Gopalakrishnan; Ken Goldberg; Kendra Byrne; Kenneth Oslund; Kento Kawaharazuka; Kevin Black; Kevin Lin; Kevin Zhang; Kiana Ehsani; Kiran Lekkala; Kirsty Ellis; Krishan Rana; Krishnan Srinivasan; Kuan Fang; Kunal Pratap Singh; Kuo-Hao Zeng; Kyle Hatch; Kyle Hsu; Laurent Itti; Lawrence Yunliang Chen; Lerrel Pinto; Li Fei-Fei; Liam Tan; Linxi Jim Fan; Lionel Ott; Lisa Lee; Luca Weihs; Magnum Chen; Marion Lepert; Marius Memmel; Masayoshi Tomizuka; Masha Itkina; Mateo Guaman Castro; Max Spero; Maximilian Du; Michael Ahn; Michael C. Yip; Mingtong Zhang; Mingyu Ding; Minho Heo; Mohan Kumar Srirama; Mohit Sharma; Moo Jin Kim; Naoaki Kanazawa; Nicklas Hansen; Nicolas Heess; Nikhil J. Joshi; Niko Suenderhauf; Ning Liu; Norman Di Palo; Nur Muhammad Mahi Shafiullah; Oier Mees; Oliver Kroemer; Osbert Bastani; Pannag R. Sanketi; Patrick Tree Miller; Patrick Yin; Paul Wohlhart; Peng Xu; Peter David Fagan; Peter Mitrano; Pierre Sermanet; Pieter Abbeel; Priya Sundaresan; Qiuyu Chen; Quan Vuong; Rafael Rafailov; Ran Tian; Ria Doshi; Roberto Martín-Martín; Rohan Baijal; Rosario Scalise; Rose Hendrix; Roy Lin; Runjia Qian; Ruohan Zhang; Russell Mendonca; Rutav Shah; Ryan Hoque; Ryan Julian; Samuel Bustamante; Sean Kirmani; Sergey Levine; Shan Lin; Sherry Moore; Shikhar Bahl; Shivin Dass; Shubham Sonawani; Shuran Song; Sichun Xu; Siddhant Haldar; Siddharth Karamcheti; Simeon Adebola; Simon Guist; Soroush Nasiriany; Stefan Schaal; Stefan Welker; Stephen Tian; Subramanian Ramamoorthy; Sudeep Dasari; Suneel Belkhale; Sungjae Park; Suraj Nair; Suvir Mirchandani; Takayuki Osa; Tanmay Gupta; Tatsuya Harada; Tatsuya Matsushima; Ted Xiao; Thomas Kollar; Tianhe Yu; Tianli Ding; Todor Davchev; Tony Z. Zhao; Travis Armstrong; Trevor Darrell; Trinity Chung; Vidhi Jain; Vincent Vanhoucke; Wei Zhan; Wenxuan Zhou; Wolfram Burgard; Xi Chen; Xiaolong Wang; Xinghao Zhu; Xinyang Geng; Xiyuan Liu; Xu Liangwei; Xuanlin Li; Yao Lu; Yecheng Jason Ma; Yejin Kim; Yevgen Chebotar; Yifan Zhou; Yifeng Zhu; Yilin Wu; Ying Xu; Yixuan Wang; Yonatan Bisk; Yoonyoung Cho; Youngwoon Lee; Yuchen Cui; Yue Cao; Yueh-Hua Wu; Yujin Tang; Yuke Zhu; Yunchu Zhang; Yunfan Jiang; Yunshuang Li; Yunzhu Li; Yusuke Iwasawa; Yutaka Matsuo; Zehan Ma; Zhuo Xu; Zichen Jeff Cui; Zichen Zhang; Zipeng Lin

doi:10.48550/arXiv.2310.08864

Open X-Embodiment: Robotic learning datasets and RT-X models

Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait SinghAnimesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi Jim Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J. Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Lin

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io.

Original language	English
Title of host publication	2024 IEEE International Conference on Robotics and Automation (ICRA)
Publisher	Institute of Electrical and Electronics Engineers
Pages	6892-6903
Number of pages	12
ISBN (Electronic)	9798350384574
ISBN (Print)	9798350384581
DOIs	https://doi.org/10.48550/arXiv.2310.08864 https://doi.org/10.1109/ICRA57147.2024.10611477
Publication status	Published - 8 Aug 2024
Event	2024 IEEE International Conference on Robotics and Automation (ICRA) - Yokohama, Japan Duration: 13 May 2024 → 17 May 2024

Publication series

Name	IEEE International Conference on Robotics and Automation
Publisher	IEEE
ISSN (Print)	1050-4729

Conference

Conference	2024 IEEE International Conference on Robotics and Automation (ICRA)
Period	13/05/24 → 17/05/24

Keywords / Materials (for Non-textual outputs)

learning systems
adaptation models
computer vision
computational modeling
collaboration
data models
task analysis

Access to Document

ONeillEtalICRA2024OpenX-EmbodimentAccepted author manuscript, 3.83 MBLicence: Creative Commons: Attribution (CC-BY)

Cite this

Open X-Embodiment Collaboration, O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., Tung, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Gupta, A., Wang, A., ... Lin, Z. (2024). Open X-Embodiment: Robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 6892-6903). Article 10611477 (IEEE International Conference on Robotics and Automation). Institute of Electrical and Electronics Engineers. https://doi.org/10.48550/arXiv.2310.08864, https://doi.org/10.1109/ICRA57147.2024.10611477

@inproceedings{f5c31f8b3f1e4cdebd8eb2c12dde8a75,

title = "Open X-Embodiment: Robotic learning datasets and RT-X models",

abstract = "Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train {"}generalist{"} X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io.",

keywords = "learning systems, adaptation models, computer vision, computational modeling, collaboration, data models, task analysis",

author = "{Open X-Embodiment Collaboration} and Abby O{\textquoteright}Neill and Abdul Rehman and Abhiram Maddukuri and Abhishek Gupta and Abhishek Padalkar and Abraham Lee and Acorn Pooley and Agrim Gupta and Ajay Mandlekar and Ajinkya Jain and Albert Tung and Alex Bewley and Alex Herzog and Alex Irpan and Alexander Khazatsky and Anant Rai and Anchit Gupta and Andrew Wang and Anikait Singh and Animesh Garg and Aniruddha Kembhavi and Annie Xie and Anthony Brohan and Antonin Raffin and Archit Sharma and Arefeh Yavary and Arhan Jain and Ashwin Balakrishna and Ayzaan Wahid and Ben Burgess-Limerick and Beomjoon Kim and Bernhard Sch{\"o}lkopf and Blake Wulfe and Brian Ichter and Cewu Lu and Charles Xu and Charlotte Le and Chelsea Finn and Chen Wang and Chenfeng Xu and Cheng Chi and Chenguang Huang and Christine Chan and Christopher Agia and Chuer Pan and Chuyuan Fu and Coline Devin and Danfei Xu and Daniel Morton and Danny Driess and Daphne Chen and Deepak Pathak and Dhruv Shah and Dieter B{\"u}chler and Dinesh Jayaraman and Dmitry Kalashnikov and Dorsa Sadigh and Edward Johns and Ethan Foster and Fangchen Liu and Federico Ceola and Fei Xia and Feiyu Zhao and Freek Stulp and Gaoyue Zhou and Sukhatme, {Gaurav S.} and Gautam Salhotra and Ge Yan and Gilbert Feng and Giulio Schiavi and Glen Berseth and Gregory Kahn and Guanzhi Wang and Hao Su and Hao-Shu Fang and Haochen Shi and Henghui Bao and Amor, {Heni Ben} and Christensen, {Henrik I.} and Hiroki Furuta and Homer Walke and Hongjie Fang and Huy Ha and Igor Mordatch and Ilija Radosavovic and Isabel Leal and Jacky Liang and Jad Abou-Chakra and Jaehyung Kim and Jaimyn Drake and Jan Peters and Jan Schneider and Jasmine Hsu and Jeannette Bohg and Jeffrey Bingham and Jeffrey Wu and Jensen Gao and Jiaheng Hu and Jiajun Wu and Jialin Wu and Jiankai Sun and Jianlan Luo and Jiayuan Gu and Jie Tan and Jihoon Oh and Jimmy Wu and Jingpei Lu and Jingyun Yang and Jitendra Malik and Jo{\~a}o Silv{\'e}rio and Joey Hejna and Jonathan Booher and Jonathan Tompson and Jonathan Yang and Jordi Salvador and Lim, {Joseph J.} and Junhyek Han and Kaiyuan Wang and Kanishka Rao and Karl Pertsch and Karol Hausman and Keegan Go and Keerthana Gopalakrishnan and Ken Goldberg and Kendra Byrne and Kenneth Oslund and Kento Kawaharazuka and Kevin Black and Kevin Lin and Kevin Zhang and Kiana Ehsani and Kiran Lekkala and Kirsty Ellis and Krishan Rana and Krishnan Srinivasan and Kuan Fang and Singh, {Kunal Pratap} and Kuo-Hao Zeng and Kyle Hatch and Kyle Hsu and Laurent Itti and Chen, {Lawrence Yunliang} and Lerrel Pinto and Li Fei-Fei and Liam Tan and Fan, {Linxi Jim} and Lionel Ott and Lisa Lee and Luca Weihs and Magnum Chen and Marion Lepert and Marius Memmel and Masayoshi Tomizuka and Masha Itkina and Castro, {Mateo Guaman} and Max Spero and Maximilian Du and Michael Ahn and Yip, {Michael C.} and Mingtong Zhang and Mingyu Ding and Minho Heo and Srirama, {Mohan Kumar} and Mohit Sharma and Kim, {Moo Jin} and Naoaki Kanazawa and Nicklas Hansen and Nicolas Heess and Joshi, {Nikhil J.} and Niko Suenderhauf and Ning Liu and Palo, {Norman Di} and Shafiullah, {Nur Muhammad Mahi} and Oier Mees and Oliver Kroemer and Osbert Bastani and Sanketi, {Pannag R.} and Miller, {Patrick Tree} and Patrick Yin and Paul Wohlhart and Peng Xu and Fagan, {Peter David} and Peter Mitrano and Pierre Sermanet and Pieter Abbeel and Priya Sundaresan and Qiuyu Chen and Quan Vuong and Rafael Rafailov and Ran Tian and Ria Doshi and Roberto Mart{\'i}n-Mart{\'i}n and Rohan Baijal and Rosario Scalise and Rose Hendrix and Roy Lin and Runjia Qian and Ruohan Zhang and Russell Mendonca and Rutav Shah and Ryan Hoque and Ryan Julian and Samuel Bustamante and Sean Kirmani and Sergey Levine and Shan Lin and Sherry Moore and Shikhar Bahl and Shivin Dass and Shubham Sonawani and Shuran Song and Sichun Xu and Siddhant Haldar and Siddharth Karamcheti and Simeon Adebola and Simon Guist and Soroush Nasiriany and Stefan Schaal and Stefan Welker and Stephen Tian and Subramanian Ramamoorthy and Sudeep Dasari and Suneel Belkhale and Sungjae Park and Suraj Nair and Suvir Mirchandani and Takayuki Osa and Tanmay Gupta and Tatsuya Harada and Tatsuya Matsushima and Ted Xiao and Thomas Kollar and Tianhe Yu and Tianli Ding and Todor Davchev and Zhao, {Tony Z.} and Travis Armstrong and Trevor Darrell and Trinity Chung and Vidhi Jain and Vincent Vanhoucke and Wei Zhan and Wenxuan Zhou and Wolfram Burgard and Xi Chen and Xiaolong Wang and Xinghao Zhu and Xinyang Geng and Xiyuan Liu and Xu Liangwei and Xuanlin Li and Yao Lu and Ma, {Yecheng Jason} and Yejin Kim and Yevgen Chebotar and Yifan Zhou and Yifeng Zhu and Yilin Wu and Ying Xu and Yixuan Wang and Yonatan Bisk and Yoonyoung Cho and Youngwoon Lee and Yuchen Cui and Yue Cao and Yueh-Hua Wu and Yujin Tang and Yuke Zhu and Yunchu Zhang and Yunfan Jiang and Yunshuang Li and Yunzhu Li and Yusuke Iwasawa and Yutaka Matsuo and Zehan Ma and Zhuo Xu and Cui, {Zichen Jeff} and Zichen Zhang and Zipeng Lin",

year = "2024",

month = aug,

day = "8",

doi = "10.48550/arXiv.2310.08864",

language = "English",

isbn = "9798350384581",

series = "IEEE International Conference on Robotics and Automation",

publisher = "Institute of Electrical and Electronics Engineers",

pages = "6892--6903",

booktitle = "2024 IEEE International Conference on Robotics and Automation (ICRA)",

address = "United States",

note = "2024 IEEE International Conference on Robotics and Automation (ICRA) ; Conference date: 13-05-2024 Through 17-05-2024",

}

Open X-Embodiment Collaboration, O’Neill, A, Rehman, A, Maddukuri, A, Gupta, A, Padalkar, A, Lee, A, Pooley, A, Gupta, A, Mandlekar, A, Jain, A, Tung, A, Bewley, A, Herzog, A, Irpan, A, Khazatsky, A, Rai, A, Gupta, A, Wang, A, Singh, A, Garg, A, Kembhavi, A, Xie, A, Brohan, A, Raffin, A, Sharma, A, Yavary, A, Jain, A, Balakrishna, A, Wahid, A, Burgess-Limerick, B, Kim, B, Schölkopf, B, Wulfe, B, Ichter, B, Lu, C, Xu, C, Le, C, Finn, C, Wang, C, Xu, C, Chi, C, Huang, C, Chan, C, Agia, C, Pan, C, Fu, C, Devin, C, Xu, D, Morton, D, Driess, D, Chen, D, Pathak, D, Shah, D, Büchler, D, Jayaraman, D, Kalashnikov, D, Sadigh, D, Johns, E, Foster, E, Liu, F, Ceola, F, Xia, F, Zhao, F, Stulp, F, Zhou, G, Sukhatme, GS, Salhotra, G, Yan, G, Feng, G, Schiavi, G, Berseth, G, Kahn, G, Wang, G, Su, H, Fang, H-S, Shi, H, Bao, H, Amor, HB, Christensen, HI, Furuta, H, Walke, H, Fang, H, Ha, H, Mordatch, I, Radosavovic, I, Leal, I, Liang, J, Abou-Chakra, J, Kim, J, Drake, J, Peters, J, Schneider, J, Hsu, J, Bohg, J, Bingham, J, Wu, J, Gao, J, Hu, J, Wu, J, Wu, J, Sun, J, Luo, J, Gu, J, Tan, J, Oh, J, Wu, J, Lu, J, Yang, J, Malik, J, Silvério, J, Hejna, J, Booher, J, Tompson, J, Yang, J, Salvador, J, Lim, JJ, Han, J, Wang, K, Rao, K, Pertsch, K, Hausman, K, Go, K, Gopalakrishnan, K, Goldberg, K, Byrne, K, Oslund, K, Kawaharazuka, K, Black, K, Lin, K, Zhang, K, Ehsani, K, Lekkala, K, Ellis, K, Rana, K, Srinivasan, K, Fang, K, Singh, KP, Zeng, K-H, Hatch, K, Hsu, K, Itti, L, Chen, LY, Pinto, L, Fei-Fei, L, Tan, L, Fan, LJ, Ott, L, Lee, L, Weihs, L, Chen, M, Lepert, M, Memmel, M, Tomizuka, M, Itkina, M, Castro, MG, Spero, M, Du, M, Ahn, M, Yip, MC, Zhang, M, Ding, M, Heo, M, Srirama, MK, Sharma, M, Kim, MJ, Kanazawa, N, Hansen, N, Heess, N, Joshi, NJ, Suenderhauf, N, Liu, N, Palo, ND, Shafiullah, NMM, Mees, O, Kroemer, O, Bastani, O, Sanketi, PR, Miller, PT, Yin, P, Wohlhart, P, Xu, P, Fagan, PD, Mitrano, P, Sermanet, P, Abbeel, P, Sundaresan, P, Chen, Q, Vuong, Q, Rafailov, R, Tian, R, Doshi, R, Martín-Martín, R, Baijal, R, Scalise, R, Hendrix, R, Lin, R, Qian, R, Zhang, R, Mendonca, R, Shah, R, Hoque, R, Julian, R, Bustamante, S, Kirmani, S, Levine, S, Lin, S, Moore, S, Bahl, S, Dass, S, Sonawani, S, Song, S, Xu, S, Haldar, S, Karamcheti, S, Adebola, S, Guist, S, Nasiriany, S, Schaal, S, Welker, S, Tian, S, Ramamoorthy, S, Dasari, S, Belkhale, S, Park, S, Nair, S, Mirchandani, S, Osa, T, Gupta, T, Harada, T, Matsushima, T, Xiao, T, Kollar, T, Yu, T, Ding, T, Davchev, T, Zhao, TZ, Armstrong, T, Darrell, T, Chung, T, Jain, V, Vanhoucke, V, Zhan, W, Zhou, W, Burgard, W, Chen, X, Wang, X, Zhu, X, Geng, X, Liu, X, Liangwei, X, Li, X, Lu, Y, Ma, YJ, Kim, Y, Chebotar, Y, Zhou, Y, Zhu, Y, Wu, Y, Xu, Y, Wang, Y, Bisk, Y, Cho, Y, Lee, Y, Cui, Y, Cao, Y, Wu, Y-H, Tang, Y, Zhu, Y, Zhang, Y, Jiang, Y, Li, Y, Li, Y, Iwasawa, Y, Matsuo, Y, Ma, Z, Xu, Z, Cui, ZJ, Zhang, Z & Lin, Z 2024, Open X-Embodiment: Robotic learning datasets and RT-X models. in 2024 IEEE International Conference on Robotics and Automation (ICRA)., 10611477, IEEE International Conference on Robotics and Automation, Institute of Electrical and Electronics Engineers, pp. 6892-6903, 2024 IEEE International Conference on Robotics and Automation (ICRA), 13/05/24. https://doi.org/10.48550/arXiv.2310.08864, https://doi.org/10.1109/ICRA57147.2024.10611477

Open X-Embodiment: Robotic learning datasets and RT-X models. / Open X-Embodiment Collaboration; O’Neill, Abby; Rehman, Abdul et al.
2024 IEEE International Conference on Robotics and Automation (ICRA). Institute of Electrical and Electronics Engineers, 2024. p. 6892-6903 10611477 (IEEE International Conference on Robotics and Automation).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Open X-Embodiment

T2 - 2024 IEEE International Conference on Robotics and Automation (ICRA)

AU - Open X-Embodiment Collaboration

AU - O’Neill, Abby

AU - Rehman, Abdul

AU - Maddukuri, Abhiram

AU - Gupta, Abhishek

AU - Padalkar, Abhishek

AU - Lee, Abraham

AU - Pooley, Acorn

AU - Gupta, Agrim

AU - Mandlekar, Ajay

AU - Jain, Ajinkya

AU - Tung, Albert

AU - Bewley, Alex

AU - Herzog, Alex

AU - Irpan, Alex

AU - Khazatsky, Alexander

AU - Rai, Anant

AU - Gupta, Anchit

AU - Wang, Andrew

AU - Singh, Anikait

AU - Garg, Animesh

AU - Kembhavi, Aniruddha

AU - Xie, Annie

AU - Brohan, Anthony

AU - Raffin, Antonin

AU - Sharma, Archit

AU - Yavary, Arefeh

AU - Jain, Arhan

AU - Balakrishna, Ashwin

AU - Wahid, Ayzaan

AU - Burgess-Limerick, Ben

AU - Kim, Beomjoon

AU - Schölkopf, Bernhard

AU - Wulfe, Blake

AU - Ichter, Brian

AU - Lu, Cewu

AU - Xu, Charles

AU - Le, Charlotte

AU - Finn, Chelsea

AU - Wang, Chen

AU - Xu, Chenfeng

AU - Chi, Cheng

AU - Huang, Chenguang

AU - Chan, Christine

AU - Agia, Christopher

AU - Pan, Chuer

AU - Fu, Chuyuan

AU - Devin, Coline

AU - Xu, Danfei

AU - Morton, Daniel

AU - Driess, Danny

AU - Chen, Daphne

AU - Pathak, Deepak

AU - Shah, Dhruv

AU - Büchler, Dieter

AU - Jayaraman, Dinesh

AU - Kalashnikov, Dmitry

AU - Sadigh, Dorsa

AU - Johns, Edward

AU - Foster, Ethan

AU - Liu, Fangchen

AU - Ceola, Federico

AU - Xia, Fei

AU - Zhao, Feiyu

AU - Stulp, Freek

AU - Zhou, Gaoyue

AU - Sukhatme, Gaurav S.

AU - Salhotra, Gautam

AU - Yan, Ge

AU - Feng, Gilbert

AU - Schiavi, Giulio

AU - Berseth, Glen

AU - Kahn, Gregory

AU - Wang, Guanzhi

AU - Su, Hao

AU - Fang, Hao-Shu

AU - Shi, Haochen

AU - Bao, Henghui

AU - Amor, Heni Ben

AU - Christensen, Henrik I.

AU - Furuta, Hiroki

AU - Walke, Homer

AU - Fang, Hongjie

AU - Ha, Huy

AU - Mordatch, Igor

AU - Radosavovic, Ilija

AU - Leal, Isabel

AU - Liang, Jacky

AU - Abou-Chakra, Jad

AU - Kim, Jaehyung

AU - Drake, Jaimyn

AU - Peters, Jan

AU - Schneider, Jan

AU - Hsu, Jasmine

AU - Bohg, Jeannette

AU - Bingham, Jeffrey

AU - Wu, Jeffrey

AU - Gao, Jensen

AU - Hu, Jiaheng

AU - Wu, Jiajun

AU - Wu, Jialin

AU - Sun, Jiankai

AU - Luo, Jianlan

AU - Gu, Jiayuan

AU - Tan, Jie

AU - Oh, Jihoon

AU - Wu, Jimmy

AU - Lu, Jingpei

AU - Yang, Jingyun

AU - Malik, Jitendra

AU - Silvério, João

AU - Hejna, Joey

AU - Booher, Jonathan

AU - Tompson, Jonathan

AU - Yang, Jonathan

AU - Salvador, Jordi

AU - Lim, Joseph J.

AU - Han, Junhyek

AU - Wang, Kaiyuan

AU - Rao, Kanishka

AU - Pertsch, Karl

AU - Hausman, Karol

AU - Go, Keegan

AU - Gopalakrishnan, Keerthana

AU - Goldberg, Ken

AU - Byrne, Kendra

AU - Oslund, Kenneth

AU - Kawaharazuka, Kento

AU - Black, Kevin

AU - Lin, Kevin

AU - Zhang, Kevin

AU - Ehsani, Kiana

AU - Lekkala, Kiran

AU - Ellis, Kirsty

AU - Rana, Krishan

AU - Srinivasan, Krishnan

AU - Fang, Kuan

AU - Singh, Kunal Pratap

AU - Zeng, Kuo-Hao

AU - Hatch, Kyle

AU - Hsu, Kyle

AU - Itti, Laurent

AU - Chen, Lawrence Yunliang

AU - Pinto, Lerrel

AU - Fei-Fei, Li

AU - Tan, Liam

AU - Fan, Linxi Jim

AU - Ott, Lionel

AU - Lee, Lisa

AU - Weihs, Luca

AU - Chen, Magnum

AU - Lepert, Marion

AU - Memmel, Marius

AU - Tomizuka, Masayoshi

AU - Itkina, Masha

AU - Castro, Mateo Guaman

AU - Spero, Max

AU - Du, Maximilian

AU - Ahn, Michael

AU - Yip, Michael C.

AU - Zhang, Mingtong

AU - Ding, Mingyu

AU - Heo, Minho

AU - Srirama, Mohan Kumar

AU - Sharma, Mohit

AU - Kim, Moo Jin

AU - Kanazawa, Naoaki

AU - Hansen, Nicklas

AU - Heess, Nicolas

AU - Joshi, Nikhil J.

AU - Suenderhauf, Niko

AU - Liu, Ning

AU - Palo, Norman Di

AU - Shafiullah, Nur Muhammad Mahi

AU - Mees, Oier

AU - Kroemer, Oliver

AU - Bastani, Osbert

AU - Sanketi, Pannag R.

AU - Miller, Patrick Tree

AU - Yin, Patrick

AU - Wohlhart, Paul

AU - Xu, Peng

AU - Fagan, Peter David

AU - Mitrano, Peter

AU - Sermanet, Pierre

AU - Abbeel, Pieter

AU - Sundaresan, Priya

AU - Chen, Qiuyu

AU - Vuong, Quan

AU - Rafailov, Rafael

AU - Tian, Ran

AU - Doshi, Ria

AU - Martín-Martín, Roberto

AU - Baijal, Rohan

AU - Scalise, Rosario

AU - Hendrix, Rose

AU - Lin, Roy

AU - Qian, Runjia

AU - Zhang, Ruohan

AU - Mendonca, Russell

AU - Shah, Rutav

AU - Hoque, Ryan

AU - Julian, Ryan

AU - Bustamante, Samuel

AU - Kirmani, Sean

AU - Levine, Sergey

AU - Lin, Shan

AU - Moore, Sherry

AU - Bahl, Shikhar

AU - Dass, Shivin

AU - Sonawani, Shubham

AU - Song, Shuran

AU - Xu, Sichun

AU - Haldar, Siddhant

AU - Karamcheti, Siddharth

AU - Adebola, Simeon

AU - Guist, Simon

AU - Nasiriany, Soroush

AU - Schaal, Stefan

AU - Welker, Stefan

AU - Tian, Stephen

AU - Ramamoorthy, Subramanian

AU - Dasari, Sudeep

AU - Belkhale, Suneel

AU - Park, Sungjae

AU - Nair, Suraj

AU - Mirchandani, Suvir

AU - Osa, Takayuki

AU - Gupta, Tanmay

AU - Harada, Tatsuya

AU - Matsushima, Tatsuya

AU - Xiao, Ted

AU - Kollar, Thomas

AU - Yu, Tianhe

AU - Ding, Tianli

AU - Davchev, Todor

AU - Zhao, Tony Z.

AU - Armstrong, Travis

AU - Darrell, Trevor

AU - Chung, Trinity

AU - Jain, Vidhi

AU - Vanhoucke, Vincent

AU - Zhan, Wei

AU - Zhou, Wenxuan

AU - Burgard, Wolfram

AU - Chen, Xi

AU - Wang, Xiaolong

AU - Zhu, Xinghao

AU - Geng, Xinyang

AU - Liu, Xiyuan

AU - Liangwei, Xu

AU - Li, Xuanlin

AU - Lu, Yao

AU - Ma, Yecheng Jason

AU - Kim, Yejin

AU - Chebotar, Yevgen

AU - Zhou, Yifan

AU - Zhu, Yifeng

AU - Wu, Yilin

AU - Xu, Ying

AU - Wang, Yixuan

AU - Bisk, Yonatan

AU - Cho, Yoonyoung

AU - Lee, Youngwoon

AU - Cui, Yuchen

AU - Cao, Yue

AU - Wu, Yueh-Hua

AU - Tang, Yujin

AU - Zhu, Yuke

AU - Zhang, Yunchu

AU - Jiang, Yunfan

AU - Li, Yunshuang

AU - Li, Yunzhu

AU - Iwasawa, Yusuke

AU - Matsuo, Yutaka

AU - Ma, Zehan

AU - Xu, Zhuo

AU - Cui, Zichen Jeff

AU - Zhang, Zichen

AU - Lin, Zipeng

PY - 2024/8/8

Y1 - 2024/8/8

N2 - Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io.

AB - Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io.

KW - learning systems

KW - adaptation models

KW - computer vision

KW - computational modeling

KW - collaboration

KW - data models

KW - task analysis

U2 - 10.48550/arXiv.2310.08864

DO - 10.48550/arXiv.2310.08864

M3 - Conference contribution

SN - 9798350384581

T3 - IEEE International Conference on Robotics and Automation

SP - 6892

EP - 6903

BT - 2024 IEEE International Conference on Robotics and Automation (ICRA)

PB - Institute of Electrical and Electronics Engineers

Y2 - 13 May 2024 through 17 May 2024

ER -

Open X-Embodiment Collaboration, O’Neill A, Rehman A, Maddukuri A, Gupta A, Padalkar A et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA). Institute of Electrical and Electronics Engineers. 2024. p. 6892-6903. 10611477. (IEEE International Conference on Robotics and Automation). doi: 10.48550/arXiv.2310.08864, 10.1109/ICRA57147.2024.10611477

Open X-Embodiment: Robotic learning datasets and RT-X models

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this