We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Approximation Vector Machines for Large-scale Online Learning

Abstract: One of the most challenging problems in kernel online learning is to bound the model size and to promote the model sparsity. Sparse models not only improve computation and memory usage, but also enhance the generalization capacity, a principle that concurs with the law of parsimony. However, inappropriate sparsity modeling may also significantly degrade the performance. In this paper, we propose Approximation Vector Machine (AVM), a model that can simultaneously encourage the sparsity and safeguard its risk in compromising the performance. When an incoming instance arrives, we approximate this instance by one of its neighbors whose distance to it is less than a predefined threshold. Our key intuition is that since the newly seen instance is expressed by its nearby neighbor the optimal performance can be analytically formulated and maintained. We develop theoretical foundations to support this intuition and further establish an analysis to characterize the gap between the approximation and optimal solutions. This gap crucially depends on the frequency of approximation and the predefined threshold. We perform the convergence analysis for a wide spectrum of loss functions including Hinge, smooth Hinge, and Logistic for classification task, and $l_1$, $l_2$, and $\epsilon$-insensitive for regression task. We conducted extensive experiments for classification task in batch and online modes, and regression task in online mode over several benchmark datasets. The results show that our proposed AVM achieved a comparable predictive performance with current state-of-the-art methods while simultaneously achieving significant computational speed-up due to the ability of the proposed AVM in maintaining the model size.
Comments: 54 pages
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:1604.06518 [cs.LG]
  (or arXiv:1604.06518v4 [cs.LG] for this version)

Submission history

From: Vu Nguyen [view email]
[v1] Fri, 22 Apr 2016 01:57:01 GMT (7797kb)
[v2] Mon, 25 Apr 2016 01:16:21 GMT (7797kb)
[v3] Wed, 5 Apr 2017 01:43:29 GMT (4070kb,D)
[v4] Sun, 28 May 2017 01:26:48 GMT (5450kb)

Link back to: arXiv, form interface, contact.