Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
(Submitted on 23 Jun 2020 (v1), last revised 30 Oct 2020 (this version, v2))
Abstract: Offline Reinforcement Learning (RL) is a promising approach for learning optimal policies in environments where direct exploration is expensive or unfeasible. However, the adoption of such policies in practice is often challenging, as they are hard to interpret within the application context, and lack measures of uncertainty for the learned policy value and its decisions. To overcome these issues, we propose an Expert-Supervised RL (ESRL) framework which uses uncertainty quantification for offline policy learning. In particular, we have three contributions: 1) the method can learn safe and optimal policies through hypothesis testing, 2) ESRL allows for different levels of risk averse implementations tailored to the application context, and finally, 3) we propose a way to interpret ESRL's policy at every state through posterior distributions, and use this framework to compute off-policy value function posteriors. We provide theoretical guarantees for our estimators and regret bounds consistent with Posterior Sampling for RL (PSRL). Sample efficiency of ESRL is independent of the chosen risk aversion threshold and quality of the behavior policy.
Submission history
From: Aaron Sonabend [view email][v1] Tue, 23 Jun 2020 17:43:44 GMT (1133kb,D)
[v2] Fri, 30 Oct 2020 19:14:14 GMT (1895kb,D)
Link back to: arXiv, form interface, contact.