We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Identifying Critical States by the Action-Based Variance of Expected Return

Abstract: The balance of exploration and exploitation plays a crucial role in accelerating reinforcement learning (RL). To deploy an RL agent in human society, its explainability is also essential. However, basic RL approaches have difficulties in deciding when to choose exploitation as well as in extracting useful points for a brief explanation of its operation. One reason for the difficulties is that these approaches treat all states the same way. Here, we show that identifying critical states and treating them specially is commonly beneficial to both problems. These critical states are the states at which the action selection changes the potential of success and failure substantially. We propose to identify the critical states using the variance in the Q-function for the actions and to perform exploitation with high probability on the identified states. These simple methods accelerate RL in a grid world with cliffs and two baseline tasks of deep RL. Our results also demonstrate that the identified critical states are intuitively interpretable regarding the crucial nature of the action selection. Furthermore, our analysis of the relationship between the timing of the identification of especially critical states and the rapid progress of learning suggests there are a few especially critical states that have important information for accelerating RL rapidly.
Comments: 12 pages, 6 figures
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
DOI: 10.1007/978-3-030-61609-0_29
Cite as: arXiv:2008.11332 [stat.ML]
  (or arXiv:2008.11332v2 [stat.ML] for this version)

Submission history

From: Izumi Karino [view email]
[v1] Wed, 26 Aug 2020 01:38:58 GMT (786kb,D)
[v2] Sun, 8 Nov 2020 10:54:34 GMT (792kb,D)

Link back to: arXiv, form interface, contact.