References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Spatiotemporal Deformable Scene Graphs for Complex Activity Detection
(Submitted on 16 Apr 2021 (v1), last revised 30 Oct 2022 (this version, v2))
Abstract: Long-term complex activity recognition and localisation can be crucial for decision making in autonomous systems such as smart cars and surgical robots. Here we address the problem via a novel deformable, spatiotemporal scene graph approach, consisting of three main building blocks: (i) action tube detection, (ii) the modelling of the deformable geometry of parts, and (iii) a graph convolutional network. Firstly, action tubes are detected in a series of snippets. Next, a new 3D deformable RoI pooling layer is designed for learning the flexible, deformable geometry of the constituent action tubes. Finally, a scene graph is constructed by considering all parts as nodes and connecting them based on different semantics such as order of appearance, sharing the same action label and feature similarity. We also contribute fresh temporal complex activity annotation for the recently released ROAD autonomous driving and SARAS-ESAD surgical action datasets and show the adaptability of our framework to different domains. Our method is shown to significantly outperform graph-based competitors on both augmented datasets.
Submission history
From: Salman Khan [view email][v1] Fri, 16 Apr 2021 16:05:34 GMT (4469kb,D)
[v2] Sun, 30 Oct 2022 08:26:22 GMT (10620kb,D)
Link back to: arXiv, form interface, contact.