Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Stochastic Variance Reduction Methods for Policy Evaluation
(Submitted on 25 Feb 2017 (v1), last revised 9 Jun 2017 (this version, v2))
Abstract: Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evaluation problem into a (quadratic) convex-concave saddle point problem, and then present a primal-dual batch gradient method, as well as two stochastic variance reduction methods for solving the problem. These algorithms scale linearly in both sample size and feature dimension. Moreover, they achieve linear convergence even when the saddle-point problem has only strong concavity in the dual variables but no strong convexity in the primal variables. Numerical experiments on benchmark problems demonstrate the effectiveness of our methods.
Submission history
From: Simon Du [view email][v1] Sat, 25 Feb 2017 20:15:55 GMT (636kb,D)
[v2] Fri, 9 Jun 2017 06:02:47 GMT (641kb,D)
Link back to: arXiv, form interface, contact.