Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Improving reinforcement learning algorithms: towards optimal learning rate policies
(Submitted on 6 Nov 2019 (v1), last revised 20 Oct 2021 (this version, v6))
Abstract: This paper investigates to what extent one can improve reinforcement learning algorithms. Our study is split in three parts. First, our analysis shows that the classical asymptotic convergence rate $O(1/\sqrt{N})$ is pessimistic and can be replaced by $O((\log(N)/N)^{\beta})$ with $\frac{1}{2}\leq \beta \leq 1$ and $N$ the number of iterations. Second, we propose a dynamic optimal policy for the choice of the learning rate $(\gamma_k)_{k\geq 0}$ used in stochastic approximation (SA). We decompose our policy into two interacting levels: the inner and the outer level. In the inner level, we present the \nameref{Alg:v_4_s} algorithm (for "PAst Sign Search") which, based on a predefined sequence $(\gamma^o_k)_{k\geq 0}$, constructs a new sequence $(\gamma^i_k)_{k\geq 0}$ whose error decreases faster. In the outer level, we propose an optimal methodology for the selection of the predefined sequence $(\gamma^o_k)_{k\geq 0}$. Third, we show empirically that our selection methodology of the learning rate outperforms significantly standard algorithms used in reinforcement learning (RL) in the three following applications: the estimation of a drift, the optimal placement of limit orders and the optimal execution of large number of shares.
Submission history
From: Othmane Mounjid [view email][v1] Wed, 6 Nov 2019 11:17:53 GMT (749kb,D)
[v2] Tue, 10 Mar 2020 19:05:39 GMT (748kb,D)
[v3] Thu, 12 Mar 2020 21:12:12 GMT (747kb,D)
[v4] Tue, 21 Apr 2020 18:04:00 GMT (746kb,D)
[v5] Sun, 17 Oct 2021 00:44:56 GMT (744kb,D)
[v6] Wed, 20 Oct 2021 06:46:22 GMT (750kb,D)
Link back to: arXiv, form interface, contact.