We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Bootstrapping Fitted Q-Evaluation for Off-Policy Inference

Abstract: Bootstrapping provides a flexible and effective approach for assessing the quality of batch reinforcement learning, yet its theoretical property is less understood. In this paper, we study the use of bootstrapping in off-policy evaluation (OPE), and in particular, we focus on the fitted Q-evaluation (FQE) that is known to be minimax-optimal in the tabular and linear-model cases. We propose a bootstrapping FQE method for inferring the distribution of the policy evaluation error and show that this method is asymptotically efficient and distributionally consistent for off-policy statistical inference. To overcome the computation limit of bootstrapping, we further adapt a subsampling procedure that improves the runtime by an order of magnitude. We numerically evaluate the bootrapping method in classical RL environments for confidence interval estimation, estimating the variance of off-policy evaluator, and estimating the correlation between multiple off-policy evaluators.
Comments: Accepted at ICML 2021
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST)
Cite as: arXiv:2102.03607 [stat.ML]
  (or arXiv:2102.03607v3 [stat.ML] for this version)

Submission history

From: Botao Hao [view email]
[v1] Sat, 6 Feb 2021 16:45:33 GMT (920kb,D)
[v2] Tue, 9 Feb 2021 11:19:15 GMT (920kb,D)
[v3] Sun, 22 May 2022 18:35:34 GMT (934kb,D)

Link back to: arXiv, form interface, contact.