We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computer Vision and Pattern Recognition

Title: Learning To Recognize Procedural Activities with Distant Supervision

Abstract: In this paper we consider the problem of classifying fine-grained, multi-step activities (e.g., cooking different recipes, making disparate home improvements, creating various forms of arts and crafts) from long videos spanning up to several minutes. Accurately categorizing these activities requires not only recognizing the individual steps that compose the task but also capturing their temporal dependencies. This problem is dramatically different from traditional action classification, where models are typically optimized on videos that span only a few seconds and that are manually trimmed to contain simple atomic actions. While step annotations could enable the training of models to recognize the individual steps of procedural activities, existing large-scale datasets in this area do not include such segment labels due to the prohibitive cost of manually annotating temporal boundaries in long videos. To address this issue, we propose to automatically identify steps in instructional videos by leveraging the distant supervision of a textual knowledge base (wikiHow) that includes detailed descriptions of the steps needed for the execution of a wide variety of complex activities. Our method uses a language model to match noisy, automatically-transcribed speech from the video to step descriptions in the knowledge base. We demonstrate that video models trained to recognize these automatically-labeled steps (without manual supervision) yield a representation that achieves superior generalization performance on four downstream tasks: recognition of procedural activities, step classification, step forecasting and egocentric video classification.
Comments: CVPR 2022. Code will be released here this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2201.10990 [cs.CV]
  (or arXiv:2201.10990v3 [cs.CV] for this version)

Submission history

From: Xudong Lin [view email]
[v1] Wed, 26 Jan 2022 15:06:28 GMT (13552kb,D)
[v2] Tue, 12 Apr 2022 13:07:11 GMT (9445kb,D)
[v3] Thu, 16 Jun 2022 21:44:50 GMT (13579kb,D)

Link back to: arXiv, form interface, contact.