Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion

Several earlier studies have shown impressive control performance in complexrobotic systems by designing the controller using a neural network and trainingit with model-free reinforcement learning. However, these outstandingcontrollers with natural motion style and high task performance are developedthrough extensive reward engineering, which is a highly laborious andtime-consuming process of designing numerous reward terms and determiningsuitable reward coefficients. In this work, we propose a novel reinforcementlearning framework for training neural network controllers for complex roboticsystems consisting of both rewards and constraints. To let the engineersappropriately reflect their intent to constraints and handle them with minimalcomputation overhead, two constraint types and an efficient policy optimizationalgorithm are suggested. The learning framework is applied to train locomotioncontrollers for several legged robots with different morphology and physicalattributes to traverse challenging terrains. Extensive simulation andreal-world experiments demonstrate that performant controllers can be trainedwith significantly less reward engineering, by tuning only a single rewardcoefficient. Furthermore, a more straightforward and intuitive engineeringprocess can be utilized, thanks to the interpretability and generalizability ofconstraints. The summary video is available at https://youtu.be/KAlm3yskhvM.