Stochastic Learning and Optimization: A Sensitivity-Based Approach

Xi-Ren Cao9780387367873, 038736787X, 0387690824

Stochastic learning and optimization is a multidisciplinary subject that has wide applications in modern engineering, social, and financial problems, including those in Internet and wireless communications, manufacturing, robotics, logistics, biomedical systems, and investment science. This book is unique in the following aspects. 1. (Four areas in one book) This book covers various disciplines in learning and optimization, including perturbation analysis (PA) of discrete-event dynamic systems, Markov decision processes (MDP)s), reinforcement learning (RL), and adaptive control, within a unified framework. 2. (A simple approach to MDPs) This book introduces MDP theory through a simple approach based on performance difference formulas. This approach leads to results for the n-bias optimality with long-run average-cost criteria and Blackwell’s optimality without discounting. 3. (Event-based optimization) This book introduces the recently developed event-based optimization approach, which opens up a research direction in overcoming or alleviating the difficulties due to the curse of dimensionality issue by utilizing the system’s special features. 4. (Sample-path construction) This book emphasizes physical interpretations based on the sample-path construction.

---

Table of contents :
Optimal Policies……Page 5
Canonical Form……Page 19
Part III Appendices: Mathematical Background……Page 1
Markov Chains on the Same State Space……Page 2
Probability……Page 3
Convergence of Potential Estimates……Page 7
The Event and Event Space……Page 14
Gradients with Discounted Reward Criteria……Page 18
A More General Case……Page 24
On-Line Optimization and Adaptive Control……Page 33
Problems……Page 40
Performance Sensitivity Formulas*……Page 45
Service Rate Control……Page 52
Constructing a Perturbed Sample Path……Page 55
Perturbation Realization……Page 65
Performance Derivatives……Page 71
Remarks on Theoretical Issues*……Page 75
Other Methods*……Page 82
Problems……Page 87
Part I Four Disciplines in Learning and Optimization……Page 115
Markov Processes……Page 10
Problems……Page 15
One Is a Subspace of the Other*……Page 16
Markov Decision Processes (MDPs)……Page 26
Problems……Page 28
Reinforcement Learning (RL)……Page 31
Part I Four Disciplines in Learning and Optimization……Page 151
An Overview of the Event-Based Approach……Page 4
Perturbation Analysis (PA)……Page 21
The Limiting Matrix……Page 23
Identification and Adaptive Control (I&AC)……Page 34
MDPs with Discounted Rewards……Page 44
Aggregated Potentials in the Event-Based Optimization……Page 46
Some Useful Techniques……Page 47
Optimality Equations*……Page 50
General Applications……Page 58
Problems……Page 62
Part I Four Disciplines in Learning and Optimization……Page 221
Convergence Properties……Page 6
Sample Paths with a Fixed Number of Regenerative Periods……Page 8
TD Methods for Performance Derivatives……Page 25
Single-Server Queues……Page 30
Part I Four Disciplines in Learning and Optimization……Page 257
Estimating Mean Values……Page 9
Eigenvalues……Page 20
Event-Based Optimization and Potential Aggregation……Page 37
Optimization……Page 39
Problems……Page 42
Part I Four Disciplines in Learning and Optimization……Page 309
Parameterized Systems: An Example……Page 13
Part II The Event-Based Optimization – A New Approach……Page 353
Fundamental Limitations of Learning and Optimization……Page 12
A Sensitivity-Based View of Learning and Optimization……Page 17
Aggregated Potentials……Page 43
Manufacturing……Page 48
Problems……Page 59
Part II The Event-Based Optimization – A New Approach……Page 421
A Map of the Learning and Optimization World……Page 41
978-0-387-69082-7_Part_3_OnlinePDF.pdf……Page 501
Queueing Networks……Page 35
Problems……Page 49
References……Page 57
Index……Page 72