References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation
(Submitted on 20 Jun 2022 (v1), last revised 15 Dec 2022 (this version, v2))
Abstract: Real-time video segmentation is a crucial task for many real-world applications such as autonomous driving and robot control. Since state-of-the-art semantic segmentation models are often too heavy for real-time applications despite their impressive performance, researchers have proposed lightweight architectures with speed-accuracy trade-offs, achieving real-time speed at the expense of reduced accuracy. In this paper, we propose a novel framework to speed up any architecture with skip-connections for real-time vision tasks by exploiting the temporal locality in videos. Specifically, at the arrival of each frame, we transform the features from the previous frame to reuse them at specific spatial bins. We then perform partial computation of the backbone network on the regions of the current frame that captures temporal differences between the current and previous frame. This is done by dynamically dropping out residual blocks using a gating mechanism which decides which blocks to drop based on inter-frame distortion. We validate our Spatial-Temporal Mask Generator (STMG) on video semantic segmentation benchmarks with multiple backbone networks, and show that our method largely speeds up inference with minimal loss of accuracy.
Submission history
From: Hyunsu Rhee [view email][v1] Mon, 20 Jun 2022 07:20:02 GMT (9340kb,D)
[v2] Thu, 15 Dec 2022 07:21:01 GMT (9023kb,D)
Link back to: arXiv, form interface, contact.