English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider model-free reinforcement learning (RL) in non-stationary Markovdecision processes. Both the reward functions and the state transitionfunctions are allowed to vary arbitrarily over time as long as their cumulativevariations do not exceed certain variation budgets. We propose RestartedQ-Learning with Upper Confidence Bounds (RestartQ-UCB), the first model-freealgorithm for non-stationary RL, and show that it outperforms existingsolutions in terms of dynamic regret. Specifically, RestartQ-UCB withFreedman-type bonus terms achieves a dynamic regret bound of$\widetilde{O}(S^{\frac{1}{3}} A^{\frac{1}{3}} \Delta^{\frac{1}{3}} HT^{\frac{2}{3}})$, where $S$ and $A$ are the numbers of states and actions,respectively, $\Delta>0$ is the variation budget, $H$ is the number of timesteps per episode, and $T$ is the total number of time steps. We furtherpresent a parameter-free algorithm named Double-Restart Q-UCB that does notrequire prior knowledge of the variation budget. We show that our algorithmsare \emph{nearly optimal} by establishing an information-theoretical lowerbound of $\Omega(S^{\frac{1}{3}} A^{\frac{1}{3}} \Delta^{\frac{1}{3}}H^{\frac{2}{3}} T^{\frac{2}{3}})$, the first lower bound in non-stationary RL.Numerical experiments validate the advantages of RestartQ-UCB in terms of bothcumulative rewards and computational efficiency. We demonstrate the power ofour results in examples of multi-agent RL and inventory control across relatedproducts.

Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Control