Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: A Discrete-Time Switching System Analysis of Q-learning
(Submitted on 17 Feb 2021 (v1), last revised 15 Sep 2022 (this version, v9))
Abstract: This paper develops a novel control-theoretic framework to analyze the non-asymptotic convergence of Q-learning. We show that the dynamics of asynchronous Q-learning with a constant step-size can be naturally formulated as a discrete-time stochastic affine switching system. Moreover, the evolution of the Q-learning estimation error is over- and underestimated by trajectories of two simpler dynamical systems. Based on these two systems, we derive a new finite-time error bound of asynchronous Q-learning when a constant stepsize is used. Our analysis also sheds light on the overestimation phenomenon of Q-learning. We further illustrate and validate the analysis through numerical simulations.
Submission history
From: Donghwan Lee [view email][v1] Wed, 17 Feb 2021 05:32:07 GMT (405kb)
[v2] Fri, 19 Feb 2021 15:52:19 GMT (405kb)
[v3] Mon, 22 Feb 2021 05:11:23 GMT (406kb)
[v4] Sat, 15 May 2021 15:49:52 GMT (405kb)
[v5] Tue, 18 May 2021 14:26:32 GMT (405kb)
[v6] Wed, 26 May 2021 03:35:50 GMT (128kb)
[v7] Thu, 13 Jan 2022 07:51:13 GMT (128kb)
[v8] Tue, 10 May 2022 08:36:58 GMT (361kb)
[v9] Thu, 15 Sep 2022 12:26:04 GMT (361kb)
Link back to: arXiv, form interface, contact.