We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Beyond Target Networks: Improving Deep $Q$-learning with Functional Regularization

Abstract: Target networks are at the core of recent success in Reinforcement Learning. They stabilize the training by using old parameters to estimate the $Q$-values, but this also limits the propagation of newly-encountered rewards which could ultimately slow down the training. In this work, we propose an alternative training method based on functional regularization which does not have this deficiency. Unlike target networks, our method uses up-to-date parameters to estimate the target $Q$-values, thereby speeding up training while maintaining stability. Surprisingly, in some cases, we can show that target networks are a special, restricted type of functional regularizers. Using this approach, we show empirical improvements in sample efficiency and performance across a range of Atari and simulated robotics environments.
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Cite as: arXiv:2106.02613 [stat.ML]
  (or arXiv:2106.02613v2 [stat.ML] for this version)

Submission history

From: Alexandre Piché [view email]
[v1] Fri, 4 Jun 2021 17:21:07 GMT (5678kb,D)
[v2] Mon, 7 Jun 2021 20:23:18 GMT (5679kb,D)

Link back to: arXiv, form interface, contact.