English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this analytical study we derive the optimal unbiased value estimator (MVU)and compare its statistical risk to three well known value estimators: TemporalDifference learning (TD), Monte Carlo estimation (MC) and Least-SquaresTemporal Difference Learning (LSTD). We demonstrate that LSTD is equivalent tothe MVU if the Markov Reward Process (MRP) is acyclic and show that both differfor most cyclic MRPs as LSTD is then typically biased. More generally, we showthat estimators that fulfill the Bellman equation can only be unbiased forspecial cyclic MRPs. The main reason being the probability measures with whichthe expectations are taken. These measure vary from state to state and due tothe strong coupling by the Bellman equation it is typically not possible for aset of value estimators to be unbiased with respect to each of these measures.Furthermore, we derive relations of the MVU to MC and TD. The most importantone being the equivalence of MC to the MVU and to LSTD for undiscounted MRPs inwhich MC has the same amount of information. In the discounted case thisequivalence does not hold anymore. For TD we show that it is essentiallyunbiased for acyclic MRPs and biased for cyclic MRPs. We also order estimatorsaccording to their risk and present counter-examples to show that no generalordering exists between the MVU and LSTD, between MC and LSTD and between TDand MC. Theoretical results are supported by examples and an empiricalevaluation.

The Optimal Unbiased Value Estimator and its Relation to LSTD, TD and MC