Current browse context:
stat
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Sideways: Depth-Parallel Training of Video Models
(Submitted on 17 Jan 2020 (v1), last revised 30 Mar 2020 (this version, v3))
Abstract: We propose Sideways, an approximate backpropagation scheme for training video models. In standard backpropagation, the gradients and activations at every computation step through the model are temporally synchronized. The forward activations need to be stored until the backward pass is executed, preventing inter-layer (depth) parallelization. However, can we leverage smooth, redundant input streams such as videos to develop a more efficient training scheme? Here, we explore an alternative to backpropagation; we overwrite network activations whenever new ones, i.e., from new frames, become available. Such a more gradual accumulation of information from both passes breaks the precise correspondence between gradients and activations, leading to theoretically more noisy weight updates. Counter-intuitively, we show that Sideways training of deep convolutional video networks not only still converges, but can also potentially exhibit better generalization compared to standard synchronized backpropagation.
Submission history
From: Mateusz Malinowski [view email][v1] Fri, 17 Jan 2020 10:49:55 GMT (7386kb,D)
[v2] Mon, 9 Mar 2020 18:16:44 GMT (7394kb,D)
[v3] Mon, 30 Mar 2020 22:48:10 GMT (7394kb,D)
Link back to: arXiv, form interface, contact.