English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Analysing and computing with Gaussian processes arising from infinitely wideneural networks has recently seen a resurgence in popularity. Despite this,many explicit covariance functions of networks with activation functions usedin modern networks remain unknown. Furthermore, while the kernels of deepnetworks can be computed iteratively, theoretical understanding of deep kernelsis lacking, particularly with respect to fixed-point dynamics. Firstly, wederive the covariance functions of multi-layer perceptrons (MLPs) withexponential linear units (ELU) and Gaussian error linear units (GELU) andevaluate the performance of the limiting Gaussian processes on some benchmarks.Secondly, and more generally, we analyse the fixed-point dynamics of iteratedkernels corresponding to a broad range of activation functions. We find thatunlike some previously studied neural network kernels, these new kernelsexhibit non-trivial fixed-point dynamics which are mirrored in finite-widthneural networks. The fixed point behaviour present in some networks explains amechanism for implicit regularisation in overparameterised deep models. Ourresults relate to both the static iid parameter conjugate kernel and thedynamic neural tangent kernel constructions. Software atgithub.com/RussellTsuchida/ELU_GELU_kernels.

Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks