We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: BERTops: Studying BERT Representations under a Topological Lens

Abstract: Proposing scoring functions to effectively understand, analyze and learn various properties of high dimensional hidden representations of large-scale transformer models like BERT can be a challenging task. In this work, we explore a new direction by studying the topological features of BERT hidden representations using persistent homology (PH). We propose a novel scoring function named "persistence scoring function (PSF)" which: (i) accurately captures the homology of the high-dimensional hidden representations and correlates well with the test set accuracy of a wide range of datasets and outperforms existing scoring metrics, (ii) captures interesting post fine-tuning "per-class" level properties from both qualitative and quantitative viewpoints, (iii) is more stable to perturbations as compared to the baseline functions, which makes it a very robust proxy, and (iv) finally, also serves as a predictor of the attack success rates for a wide category of black-box and white-box adversarial attack methods. Our extensive correlation experiments demonstrate the practical utility of PSF on various NLP tasks relevant to BERT.
Subjects: Machine Learning (cs.LG)
Journal reference: IJCNN 2022
Cite as: arXiv:2205.00953 [cs.LG]
  (or arXiv:2205.00953v2 [cs.LG] for this version)

Submission history

From: Jatin Chauhan [view email]
[v1] Mon, 2 May 2022 14:56:17 GMT (1232kb,D)
[v2] Sat, 29 Oct 2022 23:06:56 GMT (3238kb,D)

Link back to: arXiv, form interface, contact.