We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: The Impact of Non-stationarity on Generalisation in Deep Reinforcement Learning

Abstract: Non-stationarity arises in Reinforcement Learning (RL) even in stationary environments. Most RL algorithms collect new data throughout training, using a non-stationary behaviour policy. Furthermore, training targets in RL can change even with a fixed state distribution when the policy, critic, or bootstrap values are updated. We study these types of non-stationarity in supervised learning settings as well as in RL, finding that they can lead to worse generalisation performance when using deep neural network function approximators. Consequently, to improve generalisation of deep RL agents, we propose Iterated Relearning (ITER). ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training. Experimentally, we show that ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
Cite as: arXiv:2006.05826 [cs.LG]
  (or arXiv:2006.05826v2 [cs.LG] for this version)

Submission history

From: Maximilian Igl [view email]
[v1] Wed, 10 Jun 2020 13:26:31 GMT (6977kb,D)
[v2] Tue, 16 Jun 2020 11:25:37 GMT (6687kb,D)
[v3] Sat, 17 Apr 2021 08:32:53 GMT (17758kb,D)
[v4] Wed, 22 Sep 2021 08:03:34 GMT (8878kb,D)

Link back to: arXiv, form interface, contact.