We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages

Abstract: While there has been significant progress towards developing NLU resources for Indic languages, syntactic evaluation has been relatively less explored. Unlike English, Indic languages have rich morphosyntax, grammatical genders, free linear word-order, and highly inflectional morphology. In this paper, we introduce Vy\=akarana: a benchmark of Colorless Green sentences in Indic languages for syntactic evaluation of multilingual language models. The benchmark comprises four syntax-related tasks: PoS Tagging, Syntax Tree-depth Prediction, Grammatical Case Marking, and Subject-Verb Agreement. We use the datasets from the evaluation tasks to probe five multilingual language models of varying architectures for syntax in Indic languages. Due to its prevalence, we also include a code-switching setting in our experiments. Our results show that the token-level and sentence-level representations from the Indic language models (IndicBERT and MuRIL) do not capture the syntax in Indic languages as efficiently as the other highly multilingual language models. Further, our layer-wise probing experiments reveal that while mBERT, DistilmBERT, and XLM-R localize the syntax in middle layers, the Indic language models do not show such syntactic localization.
Comments: Accepted at the Multilingual Representation Learning workshop (EMNLP 2021)
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2103.00854 [cs.CL]
  (or arXiv:2103.00854v3 [cs.CL] for this version)

Submission history

From: Rajaswa Patil [view email]
[v1] Mon, 1 Mar 2021 09:07:58 GMT (5388kb,D)
[v2] Sun, 16 May 2021 15:34:26 GMT (5390kb,D)
[v3] Sat, 2 Oct 2021 07:01:45 GMT (5558kb,D)

Link back to: arXiv, form interface, contact.