Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: A nearly Blackwell-optimal policy gradient method
(Submitted on 28 May 2021 (v1), last revised 3 Jul 2022 (this version, v3))
Abstract: For continuing environments, reinforcement learning (RL) methods commonly maximize the discounted reward criterion with discount factor close to 1 in order to approximate the average reward (the gain). However, such a criterion only considers the long-run steady-state performance, ignoring the transient behaviour in transient states. In this work, we develop a policy gradient method that optimizes the gain, then the bias (which indicates the transient performance and is important to capably select from policies with equal gain). We derive expressions that enable sampling for the gradient of the bias and its preconditioning Fisher matrix. We further devise an algorithm that solves the gain-then-bias (bi-level) optimization. Its key ingredient is an RL-specific logarithmic barrier function. Experimental results provide insights into the fundamental mechanisms of our proposal.
Submission history
From: Vektor Dewanto [view email][v1] Fri, 28 May 2021 06:37:02 GMT (18717kb,D)
[v2] Fri, 4 Jun 2021 00:44:49 GMT (18717kb,D)
[v3] Sun, 3 Jul 2022 06:34:57 GMT (4697kb,D)
Link back to: arXiv, form interface, contact.