Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding
(Submitted on 1 Nov 2019 (v1), last revised 27 Sep 2021 (this version, v4))
Abstract: Videos capture events that typically contain multiple sequential, and simultaneous, actions even in the span of only a few seconds. However, most large-scale datasets built to train models for action recognition in video only provide a single label per video. Consequently, models can be incorrectly penalized for classifying actions that exist in the videos but are not explicitly labeled and do not learn the full spectrum of information present in each video in training. Towards this goal, we present the Multi-Moments in Time dataset (M-MiT) which includes over two million action labels for over one million three second videos. This multi-label dataset introduces novel challenges on how to train and analyze models for multi-action detection. Here, we present baseline results for multi-action recognition using loss functions adapted for long tail multi-label learning, provide improved methods for visualizing and interpreting models trained for multi-label action detection and show the strength of transferring models trained on M-MiT to smaller datasets.
Submission history
From: Mathew Monfort [view email][v1] Fri, 1 Nov 2019 07:09:36 GMT (5851kb,D)
[v2] Mon, 4 Nov 2019 20:24:15 GMT (5857kb,D)
[v3] Fri, 10 Jan 2020 00:24:30 GMT (8900kb,D)
[v4] Mon, 27 Sep 2021 22:59:12 GMT (7295kb,D)
Link back to: arXiv, form interface, contact.