English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Off-Policy Evaluation (OPE) serves as one of the cornerstones inReinforcement Learning (RL). Fitted Q Evaluation (FQE) with various functionapproximators, especially deep neural networks, has gained practical success.While statistical analysis has proved FQE to be minimax-optimal with tabular,linear and several nonparametric function families, its practical performancewith more general function approximator is less theoretically understood. Wefocus on FQE with general differentiable function approximators, making ourtheory applicable to neural function approximations. We approach this problemusing the Z-estimation theory and establish the following results: The FQEestimation error is asymptotically normal with explicit variance determinedjointly by the tangent space of the function class at the ground truth, thereward structure, and the distribution shift due to off-policy learning; Thefinite-sample FQE error bound is dominated by the same variance term, and itcan also be bounded by function class-dependent divergence, which measures howthe off-policy distribution shift intertwines with the function approximator.In addition, we study bootstrapping FQE estimators for error distributioninference and estimating confidence intervals, accompanied by a Cramer-Raolower bound that matches our upper bounds. The Z-estimation analysis provides ageneralizable theoretical framework for studying off-policy estimation in RLand provides sharp statistical theory for FQE with differentiable functionapproximators.

Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theory