We gratefully acknowledge support from
the Simons Foundation and member institutions.

Computation and Language

New submissions

[ total of 47 entries: 1-47 ]
[ showing up to 2000 entries per page: fewer | more ]

New submissions for Mon, 27 Sep 21

[1]  arXiv:2109.11541 [pdf, other]
Title: CSAGN: Conversational Structure Aware Graph Network for Conversational Semantic Role Labeling
Authors: Han Wu, Kun Xu, Linqi Song
Comments: To appear in EMNLP 2021
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

Conversational semantic role labeling (CSRL) is believed to be a crucial step towards dialogue understanding. However, it remains a major challenge for existing CSRL parser to handle conversational structural information. In this paper, we present a simple and effective architecture for CSRL which aims to address this problem. Our model is based on a conversational structure-aware graph network which explicitly encodes the speaker dependent information. We also propose a multi-task learning method to further improve the model. Experimental results on benchmark datasets show that our model with our proposed training objectives significantly outperforms previous baselines.

[2]  arXiv:2109.11603 [pdf]
Title: Document Automation Architectures and Technologies: A Survey
Comments: 34 pages, 11 figures, 5 tables
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)

This paper surveys the current state of the art in document automation (DA). The objective of DA is to reduce the manual effort during the generation of documents by automatically integrating input from different sources and assembling documents conforming to defined templates. There have been reviews of commercial solutions of DA, particularly in the legal domain, but to date there has been no comprehensive review of the academic research on DA architectures and technologies. The current survey of DA reviews the academic literature and provides a clearer definition and characterization of DA and its features, identifies state-of-the-art DA architectures and technologies in academic research, and provides ideas that can lead to new research opportunities within the DA field in light of recent advances in artificial intelligence and deep neural networks.

[3]  arXiv:2109.11621 [pdf, other]
Title: iFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document Exploration
Comments: Proceedings of EMNLP 2021, System Demonstrations. 7 pages and an appendix
Subjects: Computation and Language (cs.CL)

We introduce iFacetSum, a web application for exploring topical document sets. iFacetSum integrates interactive summarization together with faceted search, by providing a novel faceted navigation scheme that yields abstractive summaries for the user's selections. This approach offers both a comprehensive overview as well as concise details regarding subtopics of choice. Fine-grained facets are automatically produced based on cross-document coreference pipelines, rendering generic concepts, entities and statements surfacing in the source texts. We analyze the effectiveness of our application through small-scale user studies, which suggest the usefulness of our approach.

[4]  arXiv:2109.11635 [pdf, other]
Title: Revisiting the Uniform Information Density Hypothesis
Journal-ref: Proceedings of EMNLP 2021
Subjects: Computation and Language (cs.CL)

The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. While its implications on language production have been well explored, the hypothesis potentially makes predictions about language comprehension and linguistic acceptability as well. Further, it is unclear how uniformity in a linguistic signal -- or lack thereof -- should be measured, and over which linguistic unit, e.g., the sentence or language level, this uniformity should hold. Here we investigate these facets of the UID hypothesis using reading time and acceptability data. While our reading time results are generally consistent with previous work, they are also consistent with a weakly super-linear effect of surprisal, which would be compatible with UID's predictions. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability. We then explore multiple operationalizations of UID, motivated by different interpretations of the original hypothesis, and analyze the scope over which the pressure towards uniformity is exerted. The explanatory power of a subset of the proposed operationalizations suggests that the strongest trend may be a regression towards a mean surprisal across the language, rather than the phrase, sentence, or document -- a finding that supports a typical interpretation of UID, namely that it is the byproduct of language users maximizing the use of a (hypothetical) communication channel.

[5]  arXiv:2109.11680 [pdf, other]
Title: Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD)

Recent progress in self-training, self-supervised pretraining and unsupervised learning enabled well performing speech recognition systems without any labeled data. However, in many cases there is labeled data available for related languages which is not utilized by these methods. This paper extends previous work on zero-shot cross-lingual transfer learning by fine-tuning a multilingually pretrained wav2vec 2.0 model to transcribe unseen languages. This is done by mapping phonemes of the training languages to the target language using articulatory features. Experiments show that this simple method significantly outperforms prior work which introduced task-specific architectures and used only part of a monolingually pretrained model.

[6]  arXiv:2109.11708 [pdf, other]
Title: Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding
Comments: To appear at EMNLP-2021 as Findings
Subjects: Computation and Language (cs.CL)

Written language carries explicit and implicit biases that can distract from meaningful signals. For example, letters of reference may describe male and female candidates differently, or their writing style may indirectly reveal demographic characteristics. At best, such biases distract from the meaningful content of the text; at worst they can lead to unfair outcomes. We investigate the challenge of re-generating input sentences to 'neutralize' sensitive attributes while maintaining the semantic meaning of the original text (e.g. is the candidate qualified?). We propose a gradient-based rewriting framework, Detect and Perturb to Neutralize (DEPEN), that first detects sensitive components and masks them for regeneration, then perturbs the generation model at decoding time under a neutralizing constraint that pushes the (predicted) distribution of sensitive attributes towards a uniform distribution. Our experiments in two different scenarios show that DEPEN can regenerate fluent alternatives that are neutral in the sensitive attribute while maintaining the semantics of other attributes.

[7]  arXiv:2109.11728 [pdf, other]
Title: AES Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses
Comments: arXiv admin note: text overlap with arXiv:2012.13872
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)

Deep-learning based Automatic Essay Scoring (AES) systems are being actively used by states and language testing agencies alike to evaluate millions of candidates for life-changing decisions ranging from college applications to visa approvals. However, little research has been put to understand and interpret the black-box nature of deep-learning based scoring algorithms. Previous studies indicate that scoring models can be easily fooled. In this paper, we explore the reason behind their surprising adversarial brittleness. We utilize recent advances in interpretability to find the extent to which features such as coherence, content, vocabulary, and relevance are important for automated scoring mechanisms. We use this to investigate the oversensitivity i.e., large change in output score with a little change in input essay content) and overstability i.e., little change in output scores with large changes in input essay content) of AES. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models with rich contextual embeddings such as BERT, behave like bag-of-words models. A few words determine the essay score without the requirement of any context making the model largely overstable. This is in stark contrast to recent probing studies on pre-trained representation learning models, which show that rich linguistic features such as parts-of-speech and morphology are encoded by them. Further, we also find that the models have learnt dataset biases, making them oversensitive. To deal with these issues, we propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies. We find that our proposed models are able to detect unusual attribution patterns and flag adversarial samples successfully.

[8]  arXiv:2109.11745 [pdf, other]
Title: DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Large-scale pre-trained language models have shown remarkable results in diverse NLP applications. Unfortunately, these performance gains have been accompanied by a significant increase in computation time and model size, stressing the need to develop new or complementary strategies to increase the efficiency of these models. In this paper we propose DACT-BERT, a differentiable adaptive computation time strategy for BERT-like models. DACT-BERT adds an adaptive computational mechanism to BERT's regular processing pipeline, which controls the number of Transformer blocks that need to be executed at inference time. By doing this, the model learns to combine the most appropriate intermediate representations for the task at hand. Our experiments demonstrate that our approach, when compared to the baselines, excels on a reduced computational regime and is competitive in other less restrictive ones.

[9]  arXiv:2109.11763 [pdf, other]
Title: Lacking the embedding of a word? Look it up into a traditional dictionary
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Word embeddings are powerful dictionaries, which may easily capture language variations. However, these dictionaries fail to give sense to rare words, which are surprisingly often covered by traditional dictionaries. In this paper, we propose to use definitions retrieved in traditional dictionaries to produce word embeddings for rare words. For this purpose, we introduce two methods: Definition Neural Network (DefiNNet) and Define BERT (DefBERT). In our experiments, DefiNNet and DefBERT significantly outperform state-of-the-art as well as baseline methods devised for producing embeddings of unknown words. In fact, DefiNNet significantly outperforms FastText, which implements a method for the same task-based on n-grams, and DefBERT significantly outperforms the BERT method for OOV words. Then, definitions in traditional dictionaries are useful to build word embeddings for rare words.

[10]  arXiv:2109.11800 [pdf, other]
Title: How Does Knowledge Graph Embedding Extrapolate to Unseen Data: a Semantic Evidence View
Comments: Main paper: 7 pages, References: 2 pages, Appendix: 2 pages
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Social and Information Networks (cs.SI)

Knowledge Graph Embedding (KGE) aims to learn representations for entities and relations. Most KGE models have gained great success, especially on extrapolation scenarios. Specifically, given an unseen triple (h, r, t), a trained model can still correctly predict t from (h, r, ?), or h from (?, r, t), such extrapolation ability is impressive. However, most existing KGE works focus on the design of delicate triple modeling function, which mainly tell us how to measure the plausibility of observed triples, but we have limited understanding of why the methods can extrapolate to unseen data, and what are the important factors to help KGE extrapolate. Therefore in this work, we attempt to, from a data relevant view, study KGE extrapolation of two problems: 1. How does KGE extrapolate to unseen data? 2. How to design the KGE model with better extrapolation ability? For the problem 1, we first discuss the impact factors for extrapolation and from relation, entity and triple level respectively, propose three Semantic Evidences (SEs), which can be observed from training set and provide important semantic information for extrapolation to unseen data. Then we verify the effectiveness of SEs through extensive experiments on several typical KGE methods, and demonstrate that SEs serve as an important role for understanding the extrapolation ability of KGE. For the problem 2, to make better use of the SE information for more extrapolative knowledge representation, we propose a novel GNN-based KGE model, called Semantic Evidence aware Graph Neural Network (SE-GNN). Finally, through extensive experiments on FB15k-237 and WN18RR datasets, we show that SE-GNN achieves state-of-the-art performance on Knowledge Graph Completion task and perform a better extrapolation ability.

[11]  arXiv:2109.11834 [pdf, other]
Title: A Diversity-Enhanced and Constraints-Relaxed Augmentation for Low-Resource Classification
Subjects: Computation and Language (cs.CL)

Data augmentation (DA) aims to generate constrained and diversified data to improve classifiers in Low-Resource Classification (LRC). Previous studies mostly use a fine-tuned Language Model (LM) to strengthen the constraints but ignore the fact that the potential of diversity could improve the effectiveness of generated data. In LRC, strong constraints but weak diversity in DA result in the poor generalization ability of classifiers. To address this dilemma, we propose a {D}iversity-{E}nhanced and {C}onstraints-\{R}elaxed {A}ugmentation (DECRA). Our DECRA has two essential components on top of a transformer-based backbone model. 1) A k-beta augmentation, an essential component of DECRA, is proposed to enhance the diversity in generating constrained data. It expands the changing scope and improves the degree of complexity of the generated data. 2) A masked language model loss, instead of fine-tuning, is used as a regularization. It relaxes constraints so that the classifier can be trained with more scattered generated data. The combination of these two components generates data that can reach or approach category boundaries and hence help the classifier generalize better. We evaluate our DECRA on three public benchmark datasets under low-resource settings. Extensive experiments demonstrate that our DECRA outperforms state-of-the-art approaches by 3.8% in the overall score.

[12]  arXiv:2109.11888 [pdf, other]
Title: Robustness and Sensitivity of BERT Models Predicting Alzheimer's Disease from Text
Comments: Accepted to W-NUT @ EMNLP 2021
Subjects: Computation and Language (cs.CL)

Understanding robustness and sensitivity of BERT models predicting Alzheimer's disease from text is important for both developing better classification models and for understanding their capabilities and limitations. In this paper, we analyze how a controlled amount of desired and undesired text alterations impacts performance of BERT. We show that BERT is robust to natural linguistic variations in text. On the other hand, we show that BERT is not sensitive to removing clinically important information from text.

[13]  arXiv:2109.11969 [pdf, other]
Title: Rethinking Crowd Sourcing for Semantic Similarity
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)

Estimation of semantic similarity is crucial for a variety of natural language processing (NLP) tasks. In the absence of a general theory of semantic information, many papers rely on human annotators as the source of ground truth for semantic similarity estimation. This paper investigates the ambiguities inherent in crowd-sourced semantic labeling. It shows that annotators that treat semantic similarity as a binary category (two sentences are either similar or not similar and there is no middle ground) play the most important role in the labeling. The paper offers heuristics to filter out unreliable annotators and stimulates further discussions on human perception of semantic similarity.

[14]  arXiv:2109.12008 [pdf, other]
Title: Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction
Comments: Accepted at EMNLP 2021
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)

State-of-the-art NLP models can adopt shallow heuristics that limit their generalization capability (McCoy et al., 2019). Such heuristics include lexical overlap with the training set in Named-Entity Recognition (Taill\'e et al., 2020) and Event or Type heuristics in Relation Extraction (Rosenman et al., 2020). In the more realistic end-to-end RE setting, we can expect yet another heuristic: the mere retention of training relation triples. In this paper, we propose several experiments confirming that retention of known facts is a key factor of performance on standard benchmarks. Furthermore, one experiment suggests that a pipeline model able to use intermediate type representations is less prone to over-rely on retention.

[15]  arXiv:2109.12012 [pdf, other]
Title: Unsupervised Translation of German--Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language
Subjects: Computation and Language (cs.CL)

This paper describes the methods behind the systems submitted by the University of Groningen for the WMT 2021 Unsupervised Machine Translation task for German--Lower Sorbian (DE--DSB): a high-resource language to a low-resource one. Our system uses a transformer encoder-decoder architecture in which we make three changes to the standard training procedure. First, our training focuses on two languages at a time, contrasting with a wealth of research on multilingual systems. Second, we introduce a novel method for initializing the vocabulary of an unseen language, achieving improvements of 3.2 BLEU for DE$\rightarrow$DSB and 4.0 BLEU for DSB$\rightarrow$DE. Lastly, we experiment with the order in which offline and online back-translation are used to train an unsupervised system, finding that using online back-translation first works better for DE$\rightarrow$DSB by 2.76 BLEU. Our submissions ranked first (tied with another team) for DSB$\rightarrow$DE and third for DE$\rightarrow$DSB.

[16]  arXiv:2109.12023 [pdf, ps, other]
Title: Indirectly Supervised English Sentence Break Prediction Using Paragraph Break Probability Estimates
Authors: Robert C. Moore
Subjects: Computation and Language (cs.CL)

This report explores the use of paragraph break probability estimates to help predict the location of sentence breaks in English natural language text. We show that a sentence break predictor based almost solely on paragraph break probability estimates can achieve high accuracy on this task. This sentence break predictor is trained almost entirely on a large amount of naturally occurring text without sentence break annotations, with only a small amount of annotated data needed to tune two hyperparameters. We also show that even better results can be achieved across in-domain and out-of-domain test data, if paragraph break probability signals are combined with a support vector machine classifier trained on a somewhat larger amount of sentence-break-annotated data. Numerous related issues are addressed along the way.

[17]  arXiv:2109.12028 [pdf]
Title: Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering
Comments: Accepted at MRQA Workshop 2021
Subjects: Computation and Language (cs.CL)

Human knowledge is collectively encoded in the roughly 6500 languages spoken around the world, but it is not distributed equally across languages. Hence, for information-seeking question answering (QA) systems to adequately serve speakers of all languages, they need to operate cross-lingually. In this work we investigate the capabilities of multilingually pre-trained language models on cross-lingual QA. We find that explicitly aligning the representations across languages with a post-hoc fine-tuning step generally leads to improved performance. We additionally investigate the effect of data size as well as the language choice in this fine-tuning step, also releasing a dataset for evaluating cross-lingual QA systems. Code and dataset are publicly available here: https://github.com/ffaisal93/aligned_qa

[18]  arXiv:2109.12036 [pdf, other]
Title: Transformers Generalize Linearly
Subjects: Computation and Language (cs.CL)

Natural language exhibits patterns of hierarchically governed dependencies, in which relations between words are sensitive to syntactic structure rather than linear ordering. While re-current network models often fail to generalize in a hierarchically sensitive way (McCoy et al.,2020) when trained on ambiguous data, the improvement in performance of newer Trans-former language models (Vaswani et al., 2017)on a range of syntactic benchmarks trained on large data sets (Goldberg, 2019; Warstadtet al., 2019) opens the question of whether these models might exhibit hierarchical generalization in the face of impoverished data.In this paper we examine patterns of structural generalization for Transformer sequence-to-sequence models and find that not only do Transformers fail to generalize hierarchically across a wide variety of grammatical mapping tasks, but they exhibit an even stronger preference for linear generalization than comparable recurrent networks

[19]  arXiv:2109.12053 [pdf, other]
Title: Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus
Comments: Findings of EMNLP 2021. Dataset available at this https URL
Subjects: Computation and Language (cs.CL)

The development of automated approaches to linguistic acceptability has been greatly fostered by the availability of the English CoLA corpus, which has also been included in the widely used GLUE benchmark. However, this kind of research for languages other than English, as well as the analysis of cross-lingual approaches, has been hindered by the lack of resources with a comparable size in other languages. We have therefore developed the ItaCoLA corpus, containing almost 10,000 sentences with acceptability judgments, which has been created following the same approach and the same steps as the English one. In this paper we describe the corpus creation, we detail its content, and we present the first experiments on this new resource. We compare in-domain and out-of-domain classification, and perform a specific evaluation of nine linguistic phenomena. We also present the first cross-lingual experiments, aimed at assessing whether multilingual transformerbased approaches can benefit from using sentences in two languages during fine-tuning.

[20]  arXiv:2109.12068 [pdf, other]
Title: AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation
Comments: All authors contributed equally
Subjects: Computation and Language (cs.CL)

Transfer learning with a unified Transformer framework (T5) that converts all language problems into a text-to-text format has recently been proposed as a simple, yet effective, transfer learning approach. Although a multilingual version of the T5 model (mT5) has been introduced, it is not clear how well it can fare on non-English tasks involving diverse data. To investigate this question, we apply mT5 on a language with a wide variety of dialects--Arabic. For evaluation, we use an existing benchmark for Arabic language understanding and introduce a new benchmark for Arabic language generation (ARGEN). We also pre-train three powerful Arabic-specific text-to-text Transformer based models and evaluate them on the two benchmarks. Our new models perform significantly better than mT5 and exceed MARBERT, the current state-of-the-art Arabic BERT-based model, on Arabic language understanding. The models also set new SOTA on the generation benchmark. Our new models and are publicly released at https://github.com/UBC-NLP/araT5 and ARLGE will be released through the same repository.

[21]  arXiv:2109.12072 [pdf]
Title: SD-QA: Spoken Dialectal Question Answering for the Real World
Comments: EMNLP 2021 Findings
Subjects: Computation and Language (cs.CL)

Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not account for the errors that speech recognition models might introduce, nor do they consider the language variations (dialects) of the users. To address this gap, we augment an existing QA dataset to construct a multi-dialect, spoken QA benchmark on five languages (Arabic, Bengali, English, Kiswahili, Korean) with more than 68k audio prompts in 24 dialects from 255 speakers. We provide baseline results showcasing the real-world performance of QA systems and analyze the effect of language variety and other sensitive speaker attributes on downstream performance. Last, we study the fairness of the ASR and QA models with respect to the underlying user populations. The dataset, model outputs, and code for reproducing all our experiments are available: https://github.com/ffaisal93/SD-QA.

[22]  arXiv:2109.12082 [pdf, other]
Title: Progressive Adversarial Learning for Bootstrapping: A Case Study on Entity Set Expansion
Comments: Accepted to the main conference of EMNLP2021
Subjects: Computation and Language (cs.CL)

Bootstrapping has become the mainstream method for entity set expansion. Conventional bootstrapping methods mostly define the expansion boundary using seed-based distance metrics, which heavily depend on the quality of selected seeds and are hard to be adjusted due to the extremely sparse supervision. In this paper, we propose BootstrapGAN, a new learning method for bootstrapping which jointly models the bootstrapping process and the boundary learning process in a GAN framework. Specifically, the expansion boundaries of different bootstrapping iterations are learned via different discriminator networks; the bootstrapping network is the generator to generate new positive entities, and the discriminator networks identify the expansion boundaries by trying to distinguish the generated entities from known positive entities. By iteratively performing the above adversarial learning, the generator and the discriminators can reinforce each other and be progressively refined along the whole bootstrapping process. Experiments show that BootstrapGAN achieves the new state-of-the-art entity set expansion performance.

[23]  arXiv:2109.12085 [pdf, other]
Title: Text-based NP Enrichment
Subjects: Computation and Language (cs.CL)

Understanding the relations between entities denoted by NPs in text is a critical part of human-like natural language understanding. However, only a fraction of such relations is covered by NLP tasks and models nowadays. In this work, we establish the task of text-based NP enrichment (TNE), that is, enriching each NP with all the preposition-mediated relations that hold between this and the other NPs in the text. The relations are represented as triplets, each denoting two NPs linked via a preposition. Humans recover such relations seamlessly, while current state-of-the-art models struggle with them due to the implicit nature of the problem. We build the first large-scale dataset for the problem, provide the formal framing and scope of annotation, analyze the data, and report the result of fine-tuned neural language models on the task, demonstrating the challenge it poses to current technology. We created a webpage with the data, data-exploration UI, code, models, and demo to foster further research into this challenging text understanding problem at yanaiela.github.io/TNE/.

[24]  arXiv:2109.12093 [pdf, other]
Title: SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Stepping from sentence-level to document-level relation extraction, the research community confronts increasing text length and more complicated entity interactions. Consequently, it is more challenging to encode the key sources of information--relevant contexts and entity types. However, existing methods only implicitly learn to model these critical information sources while being trained for relation extraction. As a result, they suffer the problems of ineffective supervision and uninterpretable model predictions. In contrast, we propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately so as to enhance interpretability. By assessing model uncertainty, SAIS further boosts the performance via evidence-based data augmentation and ensemble inference while reducing the computational cost. Eventually, SAIS delivers state-of-the-art relation extraction results on three benchmarks (DocRED, CDR, and GDA) and achieves 5.04% relative gains in F1 score compared to the runner-up in evidence retrieval on DocRED.

[25]  arXiv:2109.12104 [pdf, other]
Title: GERNERMED -- An Open German Medical NER Model
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Machine Learning (cs.LG)

The current state of adoption of well-structured electronic health records and integration of digital methods for storing medical patient data in structured formats can often considered as inferior compared to the use of traditional, unstructured text based patient data documentation. Data mining in the field of medical data analysis often needs to rely solely on processing of unstructured data to retrieve relevant data. In natural language processing (NLP), statistical models have been shown successful in various tasks like part-of-speech tagging, relation extraction (RE) and named entity recognition (NER). In this work, we present GERNERMED, the first open, neural NLP model for NER tasks dedicated to detect medical entity types in German text data. Here, we avoid the conflicting goals of protection of sensitive patient data from training data extraction and the publication of the statistical model weights by training our model on a custom dataset that was translated from publicly available datasets in foreign language by a pretrained neural machine translation model. The sample code and the statistical model is available at: https://github.com/frankkramer-lab/GERNERMED

[26]  arXiv:2109.12105 [pdf, other]
Title: Faithful Target Attribute Prediction in Neural Machine Translation
Comments: Withdrawn from Findings of ACL 2021
Subjects: Computation and Language (cs.CL)

The training data used in NMT is rarely controlled with respect to specific attributes, such as word casing or gender, which can cause errors in translations. We argue that predicting the target word and attributes simultaneously is an effective way to ensure that translations are more faithful to the training data distribution with respect to these attributes. Experimental results on two tasks, uppercased input translation and gender prediction, show that this strategy helps mirror the training data distribution in testing. It also facilitates data augmentation on the task of uppercased input translation.

Cross-lists for Mon, 27 Sep 21

[27]  arXiv:2109.11778 (cross-list from cs.CV) [pdf, other]
Title: Dense Contrastive Visual-Linguistic Pretraining
Comments: Accepted by ACM Multimedia 2021. arXiv admin note: text overlap with arXiv:2007.13135
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)

Inspired by the success of BERT, several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. In particular, LXMERT and UNITER adopt visual region feature regression and label classification as pretext tasks. However, they tend to suffer from the problems of noisy labels and sparse semantic annotations, based on the visual features having been pretrained on a crowdsourced dataset with limited and inconsistent semantic labeling. To overcome these issues, we propose unbiased Dense Contrastive Visual-Linguistic Pretraining (DCVLP), which replaces the region regression and classification with cross-modality region contrastive learning that requires no annotations. Two data augmentation strategies (Mask Perturbation and Intra-/Inter-Adversarial Perturbation) are developed to improve the quality of negative samples used in contrastive learning. Overall, DCVLP allows cross-modality dense region contrastive learning in a self-supervised setting independent of any object annotations. We compare our method against prior visual-linguistic pretraining frameworks to validate the superiority of dense contrastive learning on multimodal representation learning.

[28]  arXiv:2109.11797 (cross-list from cs.CV) [pdf, other]
Title: CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
Comments: Work in progress
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)

Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists a significant gap between the objective forms of model pre-training and fine-tuning, resulting in a need for quantities of labeled data to stimulate the visual grounding capability of VL-PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, our prompt tuning approach enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that prompt tuned VL-PTMs outperform their fine-tuned counterparts by a large margin (e.g., 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). All the data and code will be available to facilitate future research.

[29]  arXiv:2109.12098 (cross-list from cs.RO) [pdf, other]
Title: CLIPort: What and Where Pathways for Robotic Manipulation
Comments: CoRL 2021. Project Website: this https URL
Subjects: Robotics (cs.RO); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

Replacements for Mon, 27 Sep 21

[30]  arXiv:2004.14454 (replaced) [pdf, other]
Title: SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification
Comments: offensive language, hate speech, cyberbullying, cyber-aggression, taxonomy for offensive language identification
Journal-ref: ACL-2021 (Findings)
Subjects: Computation and Language (cs.CL)
[31]  arXiv:2007.06796 (replaced) [pdf, other]
Title: Calling Out Bluff: Evaluation Toolkit For Robustness Testing Of Automatic Essay Scoring Systems
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
[32]  arXiv:2009.09191 (replaced) [pdf, other]
Title: OpenAttack: An Open-source Textual Adversarial Attack Toolkit
Comments: ACL-IJCNLP 2021 Demo. 9 pages, 3 figures
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR)
[33]  arXiv:2010.12730 (replaced) [pdf, other]
Title: Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality
Comments: Findings of EMNLP 2020
Subjects: Computation and Language (cs.CL)
[34]  arXiv:2101.00416 (replaced) [pdf, other]
Title: Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Subjects: Computation and Language (cs.CL)
[35]  arXiv:2104.03776 (replaced) [pdf, other]
Title: Statistically significant detection of semantic shifts using contextual word embeddings
Subjects: Computation and Language (cs.CL)
[36]  arXiv:2106.01732 (replaced) [pdf, other]
Title: Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
Comments: 5 pages; accepted to MRQA 2021 @ EMNLP 2021
Subjects: Computation and Language (cs.CL)
[37]  arXiv:2107.00910 (replaced) [pdf, other]
Title: Learned Token Pruning for Transformers
Subjects: Computation and Language (cs.CL)
[38]  arXiv:2107.05541 (replaced) [pdf, other]
Title: End-to-End Natural Language Understanding Pipeline for Bangla Conversational Agent
Comments: Accepted in IEEE International Conference on Machine Learning and Applications 2021
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
[39]  arXiv:2108.01879 (replaced) [pdf, other]
Title: Summary Explorer: Visualizing the State of the Art in Text Summarization
Comments: Accepted as system demonstration at EMNLP 2021
Subjects: Computation and Language (cs.CL)
[40]  arXiv:2109.03481 (replaced) [pdf, other]
Title: Sequence Level Contrastive Learning for Text Summarization
Comments: 2 figures, 12 tables
Subjects: Computation and Language (cs.CL)
[41]  arXiv:2109.03777 (replaced) [pdf, other]
Title: Forget me not: A Gentle Reminder to Mind the Simple Multi-Layer Perceptron Baseline for Text Classification
Comments: 5 pages, added link to code
Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)
[42]  arXiv:2109.10080 (replaced) [pdf, other]
Title: NADE: A Benchmark for Robust Adverse Drug Events Extraction in Face of Negations
Comments: W-NUT Workshop, EMLNP 2021
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
[43]  arXiv:2109.11010 (replaced) [pdf, ps, other]
Title: Alzheimers Dementia Detection using Acoustic & Linguistic features and Pre-Trained BERT
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD)
[44]  arXiv:2109.11247 (replaced) [pdf, other]
Title: The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21
Comments: 10 pages, 5 figures, WMT2021
Subjects: Computation and Language (cs.CL)
[45]  arXiv:1908.06738 (replaced) [src]
Title: Semantic Source Code Search: A Study of the Past and a Glimpse at the Future
Authors: Muhammad Khalifa
Comments: The paper is outdated as there have been new methods and I have little time to work on it
Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL)
[46]  arXiv:2006.03950 (replaced) [pdf, other]
Title: ValNorm Quantifies Semantics to Reveal Consistent Valence Biases Across Languages and Over Centuries
Comments: 16 pages, 3 figures, 12 tables
Journal-ref: Empirical Methods in Natural Language Processing 2021
Subjects: Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
[47]  arXiv:2109.10598 (replaced) [pdf, other]
Title: Diarisation using location tracking with agglomerative clustering
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
[ total of 47 entries: 1-47 ]
[ showing up to 2000 entries per page: fewer | more ]

Disable MathJax (What is MathJax?)

Links to: arXiv, form interface, find, cs, recent, 2109, contact, help  (Access key information)