Technical Note—Dynamic Programming and Probabilistic Constraints

This note deals with the manner in which dynamic problems, involving probabilistic constraints, may be tackled using the ideas of Lagrange multipliers and efficient solutions. Both the infinite and finite time horizon are considered. Under very general conditions, Lagrange-multiplier and efficient-solution methods will readily produce, via the dynamic-programming formulations, classes of optimal solutions. However there may be gaps in the constraint levels thus generated. It is shown that, providing we admit mixed policies, these gaps can be filled in and that, furthermore, the dynamic programming calculations may, in some general circumstances, be carried out initially in terms of pure policies, and optimal mixed policies can be generated from these. The probabilistic constraints are treated in two ways, viz., by considering situations in which constraints are placed on the probabilities with which systems enter into specific states, and by considering situations in which minimum variances of performance are required subject to constraints on mean performance. Finally the mean/variance problem is viewed from the point of view of efficient solution theory. It is seen that some of the main variance-minimization theorems may be related to this more general theory, and that efficient solutions may also be obtained using dynamic-programming methods.