References & Citations
Computer Science > Information Retrieval
Title: SS4MCT: A Statistical Stemmer for Morphologically Complex Texts
(Submitted on 25 May 2016 (v1), last revised 20 Jun 2016 (this version, v2))
Abstract: There have been multiple attempts to resolve various inflection matching problems in information retrieval. Stemming is a common approach to this end. Among many techniques for stemming, statistical stemming has been shown to be effective in a number of languages, particularly highly inflected languages. In this paper we propose a method for finding affixes in different positions of a word. Common statistical techniques heavily rely on string similarity in terms of prefix and suffix matching. Since infixes are common in irregular/informal inflections in morphologically complex texts, it is required to find infixes for stemming. In this paper we propose a method whose aim is to find statistical inflectional rules based on minimum edit distance table of word pairs and the likelihoods of the rules in a language. These rules are used to statistically stem words and can be used in different text mining tasks. Experimental results on CLEF 2008 and CLEF 2009 English-Persian CLIR tasks indicate that the proposed method significantly outperforms all the baselines in terms of MAP.
Submission history
From: Javid Dadashkarimi [view email][v1] Wed, 25 May 2016 12:25:26 GMT (1026kb,D)
[v2] Mon, 20 Jun 2016 21:37:19 GMT (1019kb,D)
Link back to: arXiv, form interface, contact.