English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Off-policy dynamic programming (DP) techniques such as $Q$-learning haveproven to be important in sequential decision-making problems. In the presenceof function approximation, however, these techniques often diverge due to theabsence of Bellman completeness in the function classes considered, a crucialcondition for the success of DP-based methods. In this paper, we show howoff-policy learning techniques based on return-conditioned supervised learning(RCSL) are able to circumvent these challenges of Bellman completeness,converging under significantly more relaxed assumptions inherited fromsupervised learning. We prove there exists a natural environment in which ifone uses two-layer multilayer perceptron as the function approximator, thelayer width needs to grow linearly with the state space size to satisfy Bellmancompleteness while a constant layer width is enough for RCSL. These findingstake a step towards explaining the superior empirical performance of RCSLmethods compared to DP-based methods in environments with near-optimaldatasets. Furthermore, in order to learn from sub-optimal datasets, we proposea simple framework called MBRCSL, granting RCSL methods the ability of dynamicprogramming to stitch together segments from distinct trajectories. MBRCSLleverages learned dynamics models and forward sampling to accomplish trajectorystitching while avoiding the need for Bellman completeness that plagues alldynamic programming algorithms. We propose both theoretical analysis andexperimental evaluation to back these claims, outperforming state-of-the-artmodel-free and model-based offline RL algorithms across several simulatedrobotics problems.

Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning