Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
(Submitted on 11 Feb 2020 (v1), last revised 13 Jul 2020 (this version, v2))
Abstract: Off-policy evaluation of sequential decision policies from observational data is necessary in applications of batch reinforcement learning such as education and healthcare. In such settings, however, unobserved variables confound observed actions, rendering exact evaluation of new policies impossible, i.e., unidentifiable. We develop a robust approach that estimates sharp bounds on the (unidentifiable) value of a given policy in an infinite-horizon problem given data from another policy with unobserved confounding, subject to a sensitivity model. We consider stationary or baseline unobserved confounding and compute bounds by optimizing over the set of all stationary state-occupancy ratios that agree with a new partially identified estimating equation and the sensitivity model. We prove convergence to the sharp bounds as we collect more confounded data. Although checking set membership is a linear program, the support function is given by a difficult nonconvex optimization problem. We develop approximations based on nonconvex projected gradient descent and demonstrate the resulting bounds empirically.
Submission history
From: Angela Zhou [view email][v1] Tue, 11 Feb 2020 16:18:14 GMT (354kb,D)
[v2] Mon, 13 Jul 2020 04:03:17 GMT (79kb,D)
Link back to: arXiv, form interface, contact.