Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Reinforcement Learning via Fenchel-Rockafellar Duality
(Submitted on 7 Jan 2020 (v1), last revised 9 Jan 2020 (this version, v2))
Abstract: We review basic concepts of convex duality, focusing on the very general and supremely useful Fenchel-Rockafellar duality. We summarize how this duality may be applied to a variety of reinforcement learning (RL) settings, including policy evaluation or optimization, online or offline learning, and discounted or undiscounted rewards. The derivations yield a number of intriguing results, including the ability to perform policy evaluation and on-policy policy gradient with behavior-agnostic offline data and methods to learn a policy via max-likelihood optimization. Although many of these results have appeared previously in various forms, we provide a unified treatment and perspective on these results, which we hope will enable researchers to better use and apply the tools of convex duality to make further progress in RL.
Submission history
From: Ofir Nachum [view email][v1] Tue, 7 Jan 2020 02:59:59 GMT (35kb,D)
[v2] Thu, 9 Jan 2020 19:08:09 GMT (36kb,D)
Link back to: arXiv, form interface, contact.