References & Citations
Computer Science > Machine Learning
Title: The best of both worlds: stochastic and adversarial episodic MDPs with unknown transition
(Submitted on 8 Jun 2021 (v1), last revised 17 Oct 2021 (this version, v2))
Abstract: We consider the best-of-both-worlds problem for learning an episodic Markov Decision Process through $T$ episodes, with the goal of achieving $\widetilde{\mathcal{O}}(\sqrt{T})$ regret when the losses are adversarial and simultaneously $\mathcal{O}(\text{polylog}(T))$ regret when the losses are (almost) stochastic. Recent work by [Jin and Luo, 2020] achieves this goal when the fixed transition is known, and leaves the case of unknown transition as a major open question. In this work, we resolve this open problem by using the same Follow-the-Regularized-Leader ($\text{FTRL}$) framework together with a set of new techniques. Specifically, we first propose a loss-shifting trick in the $\text{FTRL}$ analysis, which greatly simplifies the approach of [Jin and Luo, 2020] and already improves their results for the known transition case. Then, we extend this idea to the unknown transition case and develop a novel analysis which upper bounds the transition estimation error by (a fraction of) the regret itself in the stochastic setting, a key property to ensure $\mathcal{O}(\text{polylog}(T))$ regret.
Submission history
From: Tiancheng Jin [view email][v1] Tue, 8 Jun 2021 05:46:35 GMT (988kb)
[v2] Sun, 17 Oct 2021 07:02:46 GMT (989kb)
Link back to: arXiv, form interface, contact.