Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Quantifying Differences in Reward Functions
(Submitted on 24 Jun 2020 (v1), last revised 17 Mar 2021 (this version, v3))
Abstract: For many tasks, the reward function is inaccessible to introspection or too complex to be specified procedurally, and must instead be learned from user data. Prior work has evaluated learned reward functions by evaluating policies optimized for the learned reward. However, this method cannot distinguish between the learned reward function failing to reflect user preferences and the policy optimization process failing to optimize the learned reward. Moreover, this method can only tell us about behavior in the evaluation environment, but the reward may incentivize very different behavior in even a slightly different deployment environment. To address these problems, we introduce the Equivalent-Policy Invariant Comparison (EPIC) distance to quantify the difference between two reward functions directly, without a policy optimization step. We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy. Furthermore, we find EPIC can be efficiently approximated and is more robust than baselines to the choice of coverage distribution. Finally, we show that EPIC distance bounds the regret of optimal policies even under different transition dynamics, and we confirm empirically that it predicts policy training success. Our source code is available at this https URL
Submission history
From: Adam Gleave [view email][v1] Wed, 24 Jun 2020 17:35:15 GMT (1478kb,D)
[v2] Thu, 8 Oct 2020 14:35:42 GMT (1483kb,D)
[v3] Wed, 17 Mar 2021 21:54:55 GMT (1914kb,D)
Link back to: arXiv, form interface, contact.