Stochastic traffic modelling and decentralised signal control based on a state transition probability model

In this study, the authors develop a state transition probability model for an isolated intersection on the basis of link‐based state transition probability matrix. The state of an intersection is defined as the combination of its input links’ states which represent their respective congestion level by occupancy rate. This way of modelling is further extended to a compact region by likening the region to a virtual node with several virtual input links to avoid dimension expansion. Based on the state transition probability model, the stochastic signal control problem for both intersections and compact regions is formulated as a Markov decision process with the specified definition of state, action, probability and reward. A sensitivity‐based policy iteration algorithm is employed to solve the Markov decision process in real‐time, which has a great advantage in computational efficiency. The results of the numerical study on a calibrated network of Caohejing District in Shanghai indicate that the authors’ proposed method outperforms the fixed‐time and actuated signal control at high loads in terms of many indices and greatly decreases the variability in the traffic performance. In addition, the compact region control can improve the optimisation efficiency while providing similar performance to the intersection control at different loads.