Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Optimizing Percentile Criterion Using Robust MDPs
(Submitted on 23 Oct 2019 (v1), last revised 25 Feb 2021 (this version, v3))
Abstract: We address the problem of computing reliable policies in reinforcement learning problems with limited data. In particular, we compute policies that achieve good returns with high confidence when deployed. This objective, known as the \emph{percentile criterion}, can be optimized using Robust MDPs~(RMDPs). RMDPs generalize MDPs to allow for uncertain transition probabilities chosen adversarially from given ambiguity sets. We show that the RMDP solution's sub-optimality depends on the spans of the ambiguity sets along the value function. We then propose new algorithms that minimize the span of ambiguity sets defined by weighted $L_1$ and $L_\infty$ norms. Our primary focus is on Bayesian guarantees, but we also describe how our methods apply to frequentist guarantees and derive new concentration inequalities for weighted $L_1$ and $L_\infty$ norms. Experimental results indicate that our optimized ambiguity sets improve significantly on prior construction methods.
Submission history
From: Reazul Hasan Russel [view email][v1] Wed, 23 Oct 2019 20:00:11 GMT (59kb,D)
[v2] Fri, 25 Oct 2019 13:08:55 GMT (59kb,D)
[v3] Thu, 25 Feb 2021 22:34:25 GMT (364kb,D)
Link back to: arXiv, form interface, contact.