Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Variational Actor-Critic Algorithms
(Submitted on 3 Aug 2021 (v1), last revised 14 Jan 2023 (this version, v4))
Abstract: We introduce a class of variational actor-critic algorithms based on a variational formulation over both the value function and the policy. The objective function of the variational formulation consists of two parts: one for maximizing the value function and the other for minimizing the Bellman residual. Besides the vanilla gradient descent with both the value function and the policy updates, we propose two variants, the clipping method and the flipping method, in order to speed up the convergence. We also prove that, when the prefactor of the Bellman residual is sufficiently large, the fixed point of the algorithm is close to the optimal policy.
Submission history
From: Yuhua Zhu [view email][v1] Tue, 3 Aug 2021 00:24:36 GMT (1679kb,D)
[v2] Wed, 4 Aug 2021 20:21:25 GMT (1679kb,D)
[v3] Sun, 15 Aug 2021 23:08:54 GMT (1680kb,D)
[v4] Sat, 14 Jan 2023 01:29:11 GMT (1681kb,D)
Link back to: arXiv, form interface, contact.