References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Compositional Video Synthesis with Action Graphs
(Submitted on 27 Jun 2020 (v1), last revised 10 Jun 2021 (this version, v4))
Abstract: Videos of actions are complex signals containing rich compositional structure in space and time. Current video generation methods lack the ability to condition the generation on multiple coordinated and potentially simultaneous timed actions. To address this challenge, we propose to represent the actions in a graph structure called Action Graph and present the new ``Action Graph To Video'' synthesis task. Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation. We train and evaluate AG2Vid on the CATER and Something-Something V2 datasets, and show that the resulting videos have better visual quality and semantic consistency compared to baselines. Finally, our model demonstrates zero-shot abilities by synthesizing novel compositions of the learned actions. For code and pretrained models, see the project page this https URL
Submission history
From: Amir Bar [view email][v1] Sat, 27 Jun 2020 09:39:04 GMT (9491kb,D)
[v2] Mon, 13 Jul 2020 11:31:18 GMT (9491kb,D)
[v3] Wed, 14 Oct 2020 11:01:21 GMT (18432kb,D)
[v4] Thu, 10 Jun 2021 21:07:15 GMT (15308kb,D)
Link back to: arXiv, form interface, contact.