1. Introduction / Eugene A. Feinberg and Adam Shwartz
Pt. 1. Finite State and Action Models
2. Finite State and Action MDPs / Lodewijk Kallenberg
3. Bias Optimality / Mark E. Lewis and Martin L. Puterman
4. Singular Perturbations of Markov Chains and Decision Processes / Konstantin E. Avrachenkov, Jerzy Filar and Moshe Haviv
Pt. II. Infinite State Models
5. Average Reward Optimization Theory for Denumerable State Spaces / Linn I. Sennott
6. Total Reward Criteria / Eugene A. Feinberg
7. Mixed Criteria / Eugene A. Feinberg and Adam Shwartz
8. Blackwell Optimality / Arie Hordijk and Alexander A. Yushkevich
9. Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations / Armand M. Makowski and Adam Shwartz
10. Stability, Performance Evaluation, and Optimization / Sean P. Meyn
11. Convex Analytic Methods in Markov Decision Processes / Vivek S. Borkar
12. Linear Programming Approach / Onesimo Hernandez-Lerma and Jean B. Lasserre
13. Invariant Gambling Problems and Markov Decision Processes / Lester E. Dubins, Ashok P. Maitra and William D. Sudderth
14. Neuro-Dynamic Programming: Overview and Recent Trends / Benjamin Van Roy
15. Markov Decision Processes in Finance and Dynamic Options / Manfred Schal
16. Applications of Markov Decision Processes in Communication Networks / Eitan Altman
17. Water Reservoir Applications of Markov Decision Processes / Bernard F. Lamond and Abdeslem Boukhtouta.
