We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: On the Statistical Benefits of Temporal Difference Learning

Abstract: Given a dataset on actions and resulting long-term rewards, a direct estimation approach fits value functions that minimize prediction error on the training data. Temporal difference learning (TD) methods instead fit value functions by minimizing the degree of temporal inconsistency between estimates made at successive time-steps. Focusing on finite state Markov chains, we provide a crisp asymptotic theory of the statistical advantages of this approach. First, we show that an intuitive inverse trajectory pooling coefficient completely characterizes the percent reduction in mean-squared error of value estimates. Depending on problem structure, the reduction could be enormous or nonexistent. Next, we prove that there can be dramatic improvements in estimates of the difference in value-to-go for two states: TD's errors are bounded in terms of a novel measure - the problem's trajectory crossing time - which can be much smaller than the problem's time horizon.
Comments: 26 pages, 7 figures, submitted to ICML 2023
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2301.13289 [cs.LG]
  (or arXiv:2301.13289v2 [cs.LG] for this version)

Submission history

From: David Cheikhi [view email]
[v1] Mon, 30 Jan 2023 21:02:25 GMT (736kb,D)
[v2] Thu, 9 Feb 2023 21:11:01 GMT (81kb,D)

Link back to: arXiv, form interface, contact.