Open AccessEvaluating Strategic Structures in Multi-Agent Inverse Reinforcement Learning
Author(s) -
Justin Fu,
Andrea Tacchetti,
Julien Pérolat,
Yoram Bachrach
Publication year - 2021
Publication title -
journal of artificial intelligence research/the journal of artificial intelligence research
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.79
H-Index - 123
eISSN - 1943-5037
pISSN - 1076-9757
DOI - 10.1613/jair.1.12594
Subject(s) - rationality , reinforcement learning , computer science , core (optical fiber) , markov decision process , common value auction , artificial intelligence , reinforcement , inverse , mathematical optimization , machine learning , markov process , mathematics , microeconomics , psychology , economics , social psychology , telecommunications , statistics , political science , law , geometry
A core question in multi-agent systems is understanding the motivations for an agent's actions based on their behavior. Inverse reinforcement learning provides a framework for extracting utility functions from observed agent behavior, casting the problem as finding domain parameters which induce such a behavior from rational decision makers. We show how to efficiently and scalably extend inverse reinforcement learning to multi-agent settings, by reducing the multi-agent problem to N single-agent problems while still satisfying rationality conditions such as strong rationality. However, we observe that rewards learned naively tend to lack insightful structure, which causes them to produce undesirable behavior when optimized in games with different players from those encountered during training. We further investigate conditions under which rewards or utility functions can be precisely identified, on problem domains such as normal-form and Markov games, as well as auctions, where we show we can learn reward functions that properly generalize to new settings.