Current browse context:
math.PR
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: A Low-rank Approximation for MDPs via Moment Coupling
(Submitted on 18 Sep 2020 (v1), last revised 9 Apr 2021 (this version, v2))
Abstract: We introduce a framework to approximate a Markov Decision Process that stands on two pillars: state aggregation -- as the algorithmic infrastructure; and central-limit-theorem-type approximations -- as the mathematical underpinning of optimality guarantees. The theory is grounded in recent work Braverman et al (2020} that relates the solution of the Bellman equation to that of a PDE where, in the spirit of the central limit theorem, the transition matrix is reduced to its local first and second moments. Solving the PDE is $\textit{not}$ required by our method. Instead, we construct a "sister" (controlled) Markov chain whose two local transition moments are approximately identical with those of the focal chain. Because of this $\textit{moment matching}$, the original chain and its "sister" are coupled through the PDE, a coupling that facilitates optimality guarantees. Embedded into standard soft aggregation algorithms, moment matching provided a disciplined mechanism to tune the aggregation and disaggregation probabilities. The computational gains arise from the reduction of the effective state space from $N$ to $N^{\frac{1}{2}+\epsilon}$ is as one might intuitively expect from approximations grounded in the central limit theorem.
Submission history
From: Amy B.Z. Zhang [view email][v1] Fri, 18 Sep 2020 17:53:17 GMT (18188kb,D)
[v2] Fri, 9 Apr 2021 19:53:36 GMT (36630kb,D)
Link back to: arXiv, form interface, contact.