We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models

Abstract: For protein sequence datasets, unlabeled data has greatly outpaced labeled data due to the high cost of wet-lab characterization. Recent deep-learning approaches to protein prediction have shown that pre-training on unlabeled data can yield useful representations for downstream tasks. However, the optimal pre-training strategy remains an open question. Instead of strictly borrowing from natural language processing (NLP) in the form of masked or autoregressive language modeling, we introduce a new pre-training task: directly predicting protein profiles derived from multiple sequence alignments. Using a set of five, standardized downstream tasks for protein models, we demonstrate that our pre-training task along with a multi-task objective outperforms masked language modeling alone on all five tasks. Our results suggest that protein sequence models may benefit from leveraging biologically-inspired inductive biases that go beyond existing language modeling techniques in NLP.
Subjects: Machine Learning (cs.LG); Biomolecules (q-bio.BM)
Cite as: arXiv:2012.00195 [cs.LG]
  (or arXiv:2012.00195v1 [cs.LG] for this version)

Submission history

From: Jesse Vig [view email]
[v1] Tue, 1 Dec 2020 01:01:34 GMT (155kb,D)

Link back to: arXiv, form interface, contact.