Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Direct loss minimization algorithms for sparse Gaussian processes
(Submitted on 7 Apr 2020 (v1), last revised 27 Oct 2020 (this version, v3))
Abstract: The paper provides a thorough investigation of Direct loss minimization (DLM), which optimizes the posterior to minimize predictive loss, in sparse Gaussian processes. For the conjugate case, we consider DLM for log-loss and DLM for square loss showing a significant performance improvement in both cases. The application of DLM in non-conjugate cases is more complex because the logarithm of expectation in the log-loss DLM objective is often intractable and simple sampling leads to biased estimates of gradients. The paper makes two technical contributions to address this. First, a new method using product sampling is proposed, which gives unbiased estimates of gradients (uPS) for the objective function. Second, a theoretical analysis of biased Monte Carlo estimates (bMC) shows that stochastic gradient descent converges despite the biased gradients. Experiments demonstrate empirical success of DLM. A comparison of the sampling methods shows that, while uPS is potentially more sample-efficient, bMC provides a better tradeoff in terms of convergence time and computational efficiency.
Submission history
From: Yadi Wei [view email][v1] Tue, 7 Apr 2020 02:31:00 GMT (1627kb,D)
[v2] Wed, 10 Jun 2020 15:30:10 GMT (386kb,D)
[v3] Tue, 27 Oct 2020 18:36:12 GMT (648kb,D)
Link back to: arXiv, form interface, contact.