We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: RuMedBench: A Russian Medical Language Understanding Benchmark

Abstract: The paper describes the open Russian medical language understanding benchmark covering several task types (classification, question answering, natural language inference, named entity recognition) on a number of novel text sets. Given the sensitive nature of the data in healthcare, such a benchmark partially closes the problem of Russian medical dataset absence. We prepare the unified format labeling, data split, and evaluation metrics for new tasks. The remaining tasks are from existing datasets with a few modifications. A single-number metric expresses a model's ability to cope with the benchmark. Moreover, we implement several baseline models, from simple ones to neural networks with transformer architecture, and release the code. Expectedly, the more advanced models yield better performance, but even a simple model is enough for a decent result in some tasks. Furthermore, for all tasks, we provide a human evaluation. Interestingly the models outperform humans in the large-scale classification tasks. However, the advantage of natural intelligence remains in the tasks requiring more knowledge and reasoning.
Comments: 11 pages, code available at this https URL; Published in the proceedings of 20th International Conference on Artificial Intelligence in Medicine, Halifax, Canada; code available at this https URL
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2201.06499 [cs.CL]
  (or arXiv:2201.06499v2 [cs.CL] for this version)

Submission history

From: Vladimir Kokh [view email]
[v1] Mon, 17 Jan 2022 16:23:33 GMT (218kb)
[v2] Tue, 24 May 2022 12:39:23 GMT (31kb)

Link back to: arXiv, form interface, contact.