We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Evaluating Input Representation for Language Identification in Hindi-English Code Mixed Text

Abstract: Natural language processing (NLP) techniques have become mainstream in the recent decade. Most of these advances are attributed to the processing of a single language. More recently, with the extensive growth of social media platforms focus has shifted to code-mixed text. The code-mixed text comprises text written in more than one language. People naturally tend to combine local language with global languages like English. To process such texts, current NLP techniques are not sufficient. As a first step, the text is processed to identify the language of the words in the text. In this work, we focus on language identification in code-mixed sentences for Hindi-English mixed text. The task of language identification is formulated as a token classification task. In the supervised setting, each word in the sentence has an associated language label. We evaluate different deep learning models and input representation combinations for this task. Mainly, character, sub-word, and word embeddings are considered in combination with CNN and LSTM based models. We show that sub-word representation along with the LSTM model gives the best results. In general sub-word representations perform significantly better than other input representations. We report the best accuracy of 94.52% using a single layer LSTM model on the standard SAIL ICON 2017 test set.
Comments: Accepted at ICDSMLA 2020
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2011.11263 [cs.CL]
  (or arXiv:2011.11263v2 [cs.CL] for this version)

Submission history

From: Raviraj Joshi [view email]
[v1] Mon, 23 Nov 2020 08:08:09 GMT (1891kb,D)
[v2] Wed, 25 Nov 2020 13:22:04 GMT (1891kb,D)

Link back to: arXiv, form interface, contact.