Premium
Discounted Reinforcement Learning Does Not Scale
Author(s) -
McDonald Matthew A. F.,
Hingston Philip
Publication year - 1997
Publication title -
computational intelligence
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.353
H-Index - 52
eISSN - 1467-8640
pISSN - 0824-7935
DOI - 10.1111/0824-7935.00035
Subject(s) - reinforcement learning , computer science , reinforcement , bellman equation , value (mathematics) , function (biology) , artificial intelligence , scaling , machine learning , scale (ratio) , state (computer science) , function approximation , mathematical optimization , mathematics , algorithm , psychology , artificial neural network , social psychology , physics , geometry , quantum mechanics , evolutionary biology , biology
Currently popular reinforcement learning methods are based on estimating value functions that indicate the long–term value of each problem state. In many domains, such as those traditionally studied in AI planning research, the size of state spaces precludes the individual storage of state value estimates. Consequently, most practical implementations of reinforcement learning methods have stored value functions using generalizing function approximators, with mixed results. We analyze the effects of approximation error on the performance of goal–based tasks, revealing potentially severe scaling difficulties. Empirical evidence is presented that suggests when difficulties are likely to occur and explains some of the widely differing results reported in the literature.