Hierarchical Multi-agent Reinforcement Learning Method Using Energy Field in Sports Games

This paper proposes an energy-field-based hierarchical multi-agent reinforcement learning method (HES-COMA) for evaluating individual agent contributions and learning efficient policies in dynamic and complex multi-agent environments such as sports games. The proposed method addresses the limitations of the conventional single-layer approach by using energy fields in a global layer to learn strategic positioning, and in a local layer to determine tactical actions (e.g., shooting, stealing, and blocking) from those positions. Specifically, the method assigns an energy value to represent the relative importance of key elements in the game space (ball, opponents, teammates, basket, and shooting probability spots), and builds a dynamically changing energy field depending on the state of play (offense, defense, free scenario, etc.). Experimental results in a commercialized 3vs3 basketball game environment show that HES-COMA achieves approximately 1.5 times faster learning speed than Counterfactual Multi-Agent Policy Gradients (COMA). It also improved the success rates of steals, rebounds, and blocks by factors of 1.38, 1.87, and 2.71, respectively. Moreover, by combining global strategic positioning information with local tactical decision-making, HES-COMA’s movement patterns more closely resemble those of users and FSM-based agents in terms of spatial utilization. Consequently, HES-COMA effectively addresses contribution evaluation and data diversity issue in dynamic multi-agent sports games, thereby boosting both learning efficiency and overall performance.