Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Approximation Vector Machines for Large-scale Online Learning
(Submitted on 22 Apr 2016 (this version), latest version 28 May 2017 (v4))
Abstract: One of the most challenging problems in kernel online learning is to bound the model size and to promote the model sparsity. Sparse models not only improve computation and memory usage, but also enhance the generalization capacity, a principle that concurs with the law of parsimony. However, inappropriate sparsity modeling may also signi?cantly degrade the performance. In this paper, we propose Approximation Vector Machine (AVM), a model that can simultaneously encourage the sparsity and safeguard its risk in compromising the performance. When an incoming instance arrives, we approximate this instance by one of its neighbors whose distance to it is less than a prede?ned threshold. Our key intuition is that since the newly seen instance is expressed by its nearby neighbor the optimal performance can be analytically formulated and maintained. We develop theoretical foundations to support this intuition and further establish an analysis to characterize the gap between the approximation and optimal solutions. This gap crucially depends on the frequency of approximation and the prede?ned threshold. We perform the convergence analysis for a wide spectrum of loss functions including Hinge, smooth Hinge, and Logistic for classi?cation task, and `1, `2, and "-insensitive for regression task. We conducted extensive experiments for classi?cation task in batch and online modes, and regression task in online mode over several benchmark datasets. The results show that our proposed AVM achieved a comparable predictive performance with current state-of-the-art methods while simultaneously achieving signi?cant computational speed-up due to the ability of the proposed AVM in maintaining the model size.
Submission history
From: Vu Nguyen [view email][v1] Fri, 22 Apr 2016 01:57:01 GMT (7797kb)
[v2] Mon, 25 Apr 2016 01:16:21 GMT (7797kb)
[v3] Wed, 5 Apr 2017 01:43:29 GMT (4070kb,D)
[v4] Sun, 28 May 2017 01:26:48 GMT (5450kb)
Link back to: arXiv, form interface, contact.