Current browse context:
cs.AI
Change to browse by:
References & Citations
Computer Science > Artificial Intelligence
Title: The Uncertainty Bellman Equation and Exploration
(Submitted on 15 Sep 2017 (v1), last revised 22 Oct 2018 (this version, v4))
Abstract: We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar \textit{uncertainty} Bellman equation (UBE), which connects the uncertainty at any time-step to the expected uncertainties at subsequent time-steps, thereby extending the potential exploratory benefit of a policy beyond individual time-steps. We prove that the unique fixed point of the UBE yields an upper bound on the variance of the posterior distribution of the Q-values induced by any policy. This bound can be much tighter than traditional count-based bonuses that compound standard deviation rather than variance. Importantly, and unlike several existing approaches to optimism, this method scales naturally to large systems with complex generalization. Substituting our UBE-exploration strategy for $\epsilon$-greedy improves DQN performance on 51 out of 57 games in the Atari suite.
Submission history
From: Brendan O'Donoghue [view email][v1] Fri, 15 Sep 2017 19:55:58 GMT (3471kb,D)
[v2] Fri, 8 Jun 2018 11:47:43 GMT (340kb,D)
[v3] Tue, 19 Jun 2018 15:42:36 GMT (340kb,D)
[v4] Mon, 22 Oct 2018 15:25:04 GMT (372kb,D)
Link back to: arXiv, form interface, contact.