References & Citations
Computer Science > Artificial Intelligence
Title: Disentangling causal effects for hierarchical reinforcement learning
(Submitted on 3 Oct 2020 (v1), last revised 21 Feb 2022 (this version, v2))
Abstract: Exploration and credit assignment under sparse rewards are still challenging problems. We argue that these challenges arise in part due to the intrinsic rigidity of operating at the level of actions. Actions can precisely define how to perform an activity but are ill-suited to describe what activity to perform. Instead, causal effects are inherently composable and temporally abstract, making them ideal for descriptive tasks. By leveraging a hierarchy of causal effects, this study aims to expedite the learning of task-specific behavior and aid exploration. Borrowing counterfactual and normality measures from causal literature, we disentangle controllable effects from effects caused by other dynamics of the environment. We propose CEHRL, a hierarchical method that models the distribution of controllable effects using a Variational Autoencoder. This distribution is used by a high-level policy to 1) explore the environment via random effect exploration so that novel effects are continuously discovered and learned, and to 2) learn task-specific behavior by prioritizing the effects that maximize a given reward function. In comparison to exploring with random actions, experimental results show that random effect exploration is a more efficient mechanism and that by assigning credit to few effects rather than many actions, CEHRL learns tasks more rapidly.
Submission history
From: Oriol Corcoll [view email][v1] Sat, 3 Oct 2020 13:19:16 GMT (3110kb,D)
[v2] Mon, 21 Feb 2022 19:18:09 GMT (3177kb,D)
Link back to: arXiv, form interface, contact.