English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic networks and queueing systems often lead to Markov decisionprocesses (MDPs) with large state and action spaces as well as nonconvexobjective functions, which hinders the convergence of many reinforcementlearning (RL) algorithms. Policy-gradient methods perform well on MDPs withlarge state and action spaces, but they sometimes experience slow convergencedue to the high variance of the gradient estimator. In this paper, we show thatsome of these difficulties can be circumvented by exploiting the structure ofthe underlying MDP. We first introduce a new family of gradient estimatorscalled score-aware gradient estimators (SAGEs). When the stationarydistribution of the MDP belongs to an exponential family parametrized by thepolicy parameters, SAGEs allow us to estimate the policy gradient withoutrelying on value-function estimation, contrary to classical policy-gradientmethods like actor-critic. To demonstrate their applicability, we examine twocommon control problems arising in stochastic networks and queueing systemswhose stationary distributions have a product-form, a special case ofexponential families. As a second contribution, we show that, under appropriateassumptions, the policy under a SAGE-based policy-gradient method has a largeprobability of converging to an optimal policy, provided that it startssufficiently close to it, even with a nonconvex objective function and multiplemaximizers. Our key assumptions are that, locally around a maximizer, anondegeneracy property of the Hessian of the objective function holds and aLyapunov function exists. Finally, we conduct a numerical comparison between aSAGE-based policy-gradient method and an actor-critic algorithm. The resultsdemonstrate that the SAGE-based method finds close-to-optimal policies morerapidly, highlighting its superior performance over the traditionalactor-critic method.

Score-Aware Policy-Gradient Methods and Performance Guarantees using  Local Lyapunov Conditions: Applications to Product-Form Stochastic Networks  and Queueing Systems