We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: An improved Bayesian TRIE based model for SMS text normalization

Abstract: Normalization of SMS text, commonly known as texting language, is being pursued for more than a decade. A probabilistic approach based on the Trie data structure was proposed in literature which was found to be better performing than HMM based approaches proposed earlier in predicting the correct alternative for an out-of-lexicon word. However, success of the Trie based approach depends largely on how correctly the underlying probabilities of word occurrences are estimated. In this work we propose a structural modification to the existing Trie-based model along with a novel training algorithm and probability generation scheme. We prove two theorems on statistical properties of the proposed Trie and use them to claim that is an unbiased and consistent estimator of the occurrence probabilities of the words. We further fuse our model into the paradigm of noisy channel based error correction and provide a heuristic to go beyond a Damerau Levenshtein distance of one. We also run simulations to support our claims and show superiority of the proposed scheme over previous works.
Comments: 7 pages, 8 figures, under review at Pattern Recognition Letters
Subjects: Computation and Language (cs.CL); Data Structures and Algorithms (cs.DS)
Cite as: arXiv:2008.01297 [cs.CL]
  (or arXiv:2008.01297v2 [cs.CL] for this version)

Submission history

From: Abhinava Sikdar [view email]
[v1] Tue, 4 Aug 2020 03:01:23 GMT (2429kb,D)
[v2] Wed, 18 Nov 2020 17:19:31 GMT (2429kb,D)

Link back to: arXiv, form interface, contact.