We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.SY

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Systems and Control

Title: Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints

Authors: Jie Fu, Ufuk Topcu
Abstract: We consider synthesis of control policies that maximize the probability of satisfying given temporal logic specifications in unknown, stochastic environments. We model the interaction between the system and its environment as a Markov decision process (MDP) with initially unknown transition probabilities. The solution we develop builds on the so-called model-based probably approximately correct Markov decision process (PAC-MDP) methodology. The algorithm attains an $\varepsilon$-approximately optimal policy with probability $1-\delta$ using samples (i.e. observations), time and space that grow polynomially with the size of the MDP, the size of the automaton expressing the temporal logic specification, $\frac{1}{\varepsilon}$, $\frac{1}{\delta}$ and a finite time horizon. In this approach, the system maintains a model of the initially unknown MDP, and constructs a product MDP based on its learned model and the specification automaton that expresses the temporal logic constraints. During execution, the policy is iteratively updated using observation of the transitions taken by the system. The iteration terminates in finitely many steps. With high probability, the resulting policy is such that, for any state, the difference between the probability of satisfying the specification under this policy and the optimal one is within a predefined bound.
Comments: 9 pages, 5 figures, Accepted by 2014 Robotics: Science and Systems (RSS)
Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG); Logic in Computer Science (cs.LO); Robotics (cs.RO)
MSC classes: 93E35
ACM classes: I.2.8; G.3
Cite as: arXiv:1404.7073 [cs.SY]
  (or arXiv:1404.7073v2 [cs.SY] for this version)

Submission history

From: Jie Fu [view email]
[v1] Mon, 28 Apr 2014 17:57:48 GMT (156kb,D)
[v2] Wed, 30 Apr 2014 17:20:57 GMT (156kb,D)

Link back to: arXiv, form interface, contact.