Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Understanding Self-supervised Learning with Dual Deep Networks
(Submitted on 1 Oct 2020 (v1), revised 12 Oct 2020 (this version, v2), latest version 15 Feb 2021 (v6))
Abstract: We propose a novel theoretical framework to understand self-supervised learning methods that employ dual pairs of deep ReLU networks (e.g., SimCLR, BYOL). First, we prove that in each SGD update of SimCLR, the weights at each layer are updated by a \emph{covariance operator} that specifically amplifies initial random selectivities that vary across data samples but survive averages over data augmentations, which we show leads to the emergence of hierarchical features, if the input data are generated from a hierarchical latent tree model. With the same framework, we also show analytically that BYOL works due to an implicit contrastive term, acting as an approximate covariance operator. The term is formed by the inter-play between the zero-mean operation of BatchNorm and the extra predictor in the online network. Extensive ablation studies justify our theoretical findings.
Submission history
From: Yuandong Tian [view email][v1] Thu, 1 Oct 2020 17:51:49 GMT (735kb,D)
[v2] Mon, 12 Oct 2020 17:42:46 GMT (739kb,D)
[v3] Thu, 22 Oct 2020 17:13:52 GMT (742kb,D)
[v4] Tue, 10 Nov 2020 18:52:22 GMT (743kb,D)
[v5] Wed, 2 Dec 2020 18:34:58 GMT (1314kb,D)
[v6] Mon, 15 Feb 2021 04:51:42 GMT (3210kb,D)
Link back to: arXiv, form interface, contact.