Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model
(Submitted on 26 May 2020 (v1), last revised 17 Apr 2023 (this version, v8))
Abstract: This paper is concerned with the sample efficiency of reinforcement learning, assuming access to a generative model (or simulator). We first consider $\gamma$-discounted infinite-horizon Markov decision processes (MDPs) with state space $\mathcal{S}$ and action space $\mathcal{A}$. Despite a number of prior works tackling this problem, a complete picture of the trade-offs between sample complexity and statistical accuracy is yet to be determined. In particular, all prior results suffer from a severe sample size barrier, in the sense that their claimed statistical guarantees hold only when the sample size exceeds at least $\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^2}$. The current paper overcomes this barrier by certifying the minimax optimality of two algorithms -- a perturbed model-based algorithm and a conservative model-based algorithm -- as soon as the sample size exceeds the order of $\frac{|\mathcal{S}||\mathcal{A}|}{1-\gamma}$ (modulo some log factor). Moving beyond infinite-horizon MDPs, we further study time-inhomogeneous finite-horizon MDPs, and prove that a plain model-based planning algorithm suffices to achieve minimax-optimal sample complexity given any target accuracy level. To the best of our knowledge, this work delivers the first minimax-optimal guarantees that accommodate the entire range of sample sizes (beyond which finding a meaningful policy is information theoretically infeasible).
Submission history
From: Yuting Wei [view email][v1] Tue, 26 May 2020 17:53:18 GMT (58kb)
[v2] Wed, 17 Jun 2020 17:50:39 GMT (59kb)
[v3] Thu, 3 Sep 2020 02:02:42 GMT (154kb,D)
[v4] Sat, 25 Dec 2021 21:39:28 GMT (829kb,D)
[v5] Thu, 15 Sep 2022 15:16:01 GMT (252kb)
[v6] Fri, 16 Sep 2022 00:48:47 GMT (63kb)
[v7] Fri, 17 Mar 2023 16:08:00 GMT (836kb)
[v8] Mon, 17 Apr 2023 15:37:59 GMT (767kb)
Link back to: arXiv, form interface, contact.