TiV-ODE: A Neural ODE-based Approach for Controllable Video Generation From Text-Image Pairs

Yucheng Xu, Li Nanbo, Arushi Goel, Zonghai Yao, Zijian Guo, Hamidreza Kasaei, Mohammadreza Kasaei, Alex Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Videos capture the evolution of continuous dynamical systems over time in the form of discrete image sequences. Recently, video generation models have been widely used in robotic research. However, generating controllable videos from image-text pairs is an important yet underexplored research topic in both robotic and computer vision communities. This paper introduces an innovative and elegant framework named \mbox{TiV-ODE}, formulating this task as modeling the dynamical system in a continuous space. Specifically, our framework leverages the ability of Neural Ordinary Differential Equations~(Neural ODEs) to model the complex dynamical system depicted by videos as a nonlinear ordinary differential equation. The resulting framework offers control over the generated videos' dynamics, content, and frame rate, a feature not provided by previous methods. Experiments demonstrate the ability of the proposed method to generate highly controllable and visually consistent videos and its capability of modeling dynamical systems. Overall, this work is a significant step towards developing advanced controllable video generation models that can handle complex and dynamic scenes.
Original languageEnglish
Title of host publication2024 IEEE International Conference on Robotics and Automation (ICRA)
PublisherIEEE
Number of pages8
Publication statusAccepted/In press - 29 Jan 2024
Event2024 IEEE International Conference on Robotics and Automation - Yokohama, Japan
Duration: 13 May 202417 May 2024
https://2024.ieee-icra.org/

Conference

Conference2024 IEEE International Conference on Robotics and Automation
Abbreviated titleICRA 2024
Country/TerritoryJapan
CityYokohama
Period13/05/2417/05/24
Internet address

Fingerprint

Dive into the research topics of 'TiV-ODE: A Neural ODE-based Approach for Controllable Video Generation From Text-Image Pairs'. Together they form a unique fingerprint.

Cite this