We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Optimization and Control

Title: Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon

Abstract: We study finite-time horizon continuous-time linear-quadratic reinforcement learning problems in an episodic setting, where both the state and control coefficients are unknown to the controller. We first propose a least-squares algorithm based on continuous-time observations and controls, and establish a logarithmic regret bound of order $O((\ln M)(\ln\ln M))$, with $M$ being the number of learning episodes. The analysis consists of two parts: perturbation analysis, which exploits the regularity and robustness of the associated Riccati differential equation; and parameter estimation error, which relies on sub-exponential properties of continuous-time least-squares estimators. We further propose a practically implementable least-squares algorithm based on discrete-time observations and piecewise constant controls, which achieves similar logarithmic regret with an additional term depending explicitly on the time stepsizes used in the algorithm.
Comments: In this version, we added some comments and references
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2006.15316 [math.OC]
  (or arXiv:2006.15316v4 [math.OC] for this version)

Submission history

From: Anran Hu [view email]
[v1] Sat, 27 Jun 2020 08:14:59 GMT (25kb)
[v2] Tue, 10 Nov 2020 07:14:07 GMT (29kb)
[v3] Mon, 17 May 2021 20:00:45 GMT (30kb)
[v4] Fri, 17 Jun 2022 18:48:26 GMT (45kb)

Link back to: arXiv, form interface, contact.