Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses
(Submitted on 16 May 2022 (v1), last revised 22 Jun 2022 (this version, v2))
Abstract: We propose the Bayes-UCBVI algorithm for reinforcement learning in tabular, stage-dependent, episodic Markov decision process: a natural extension of the Bayes-UCB algorithm by Kaufmann et al. (2012) for multi-armed bandits. Our method uses the quantile of a Q-value function posterior as upper confidence bound on the optimal Q-value function. For Bayes-UCBVI, we prove a regret bound of order $\widetilde{O}(\sqrt{H^3SAT})$ where $H$ is the length of one episode, $S$ is the number of states, $A$ the number of actions, $T$ the number of episodes, that matches the lower-bound of $\Omega(\sqrt{H^3SAT})$ up to poly-$\log$ terms in $H,S,A,T$ for a large enough $T$. To the best of our knowledge, this is the first algorithm that obtains an optimal dependence on the horizon $H$ (and $S$) without the need for an involved Bernstein-like bonus or noise. Crucial to our analysis is a new fine-grained anti-concentration bound for a weighted Dirichlet sum that can be of independent interest. We then explain how Bayes-UCBVI can be easily extended beyond the tabular setting, exhibiting a strong link between our algorithm and Bayesian bootstrap (Rubin, 1981).
Submission history
From: Daniil Tiapkin [view email][v1] Mon, 16 May 2022 14:13:06 GMT (448kb,D)
[v2] Wed, 22 Jun 2022 06:58:54 GMT (482kb,D)
Link back to: arXiv, form interface, contact.