We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision

Abstract: Recent work has shown success in incorporating pre-trained models like BERT to improve NLP systems. However, existing pre-trained models lack of causal knowledge which prevents today's NLP systems from thinking like humans. In this paper, we investigate the problem of injecting causal knowledge into pre-trained models. There are two fundamental problems: 1) how to collect various granularities of causal pairs from unstructured texts; 2) how to effectively inject causal knowledge into pre-trained models. To address these issues, we extend the idea of CausalBERT from previous studies, and conduct experiments on various datasets to evaluate its effectiveness. In addition, we adopt a regularization-based method to preserve the already learned knowledge with an extra regularization term while injecting causal knowledge. Extensive experiments on 7 datasets, including four causal pair classification tasks, two causal QA tasks and a causal inference task, demonstrate that CausalBERT captures rich causal knowledge and outperforms all pre-trained models-based state-of-the-art methods, achieving a new causal inference benchmark.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2107.09852 [cs.CL]
  (or arXiv:2107.09852v2 [cs.CL] for this version)

Submission history

From: Zhongyang Li [view email]
[v1] Wed, 21 Jul 2021 02:49:46 GMT (4646kb,D)
[v2] Sun, 8 Aug 2021 03:49:26 GMT (4475kb,D)

Link back to: arXiv, form interface, contact.