Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics

In this paper we explore some of the benefits of using the finite-state Markov chain approximation (MCA) method of Kushner and Dupuis (2001) to solve continuous-time optimal control problems. We first show that the implicit finite-difference scheme of Achdou et al. (2017) amounts to a limiting form of the MCA method for a certain choice of approximating chains and policy function iteration for the resulting system of equations. We then illustrate the benefits of departing from policy function iteration by showing that using variations of modified policy function iteration to solve income fluctuation problems in two and three dimensions can lead to an increase in the speed of convergence of more than an order of magnitude. We then show that the MCA method is also well-suited to solving portfolio problems with highly correlated state variables, a setting that commonly occurs within general equilibrium models with financial frictions and for which it is difficult to construct monotone (and hence convergent) finite-difference schemes.