Title: SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation

Abstract: Acronym disambiguation means finding the correct meaning of an ambiguous acronym in a given sentence from the dictionary, which is one of the key points for scientific document understanding (SDU@AAAI-22). Recently, many attempts have tried to solve this problem via fine-tuning the pre-trained masked language models (MLMs) in order to obtain a better acronym representation. However, the acronym meaning is varied under different contexts, whose corresponded sentence representation is the anisotropic distribution occupied with a narrow subset of the entire representation space. Such representations from pre-trained MLMs are not ideal for the acronym disambiguation from the given dictionary. In this paper, we propose a Simple framework for Contrastive Learning of Acronym Disambiguation (SimCLAD) method to better understand the acronym meanings. Specifically, we design a novel continual contrastive pre-training method that enhances the pre-trained model's generalization ability by learning the isotropic and discriminative distribution of the acronym sentence representations. The results on the acronym disambiguation of the scientific domain in English show that the proposed method outperforms all other competitive state-of-the-art (SOTA) methods.