English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

It is fundamental to measure model complexity of deep neural networks. Theexisting literature on model complexity mainly focuses on neural networks withpiecewise linear activation functions. Model complexity of neural networks withgeneral curve activation functions remains an open problem. To tackle thechallenge, in this paper, we first propose the linear approximation neuralnetwork (LANN for short), a piecewise linear framework to approximate a givendeep model with curve activation function. LANN constructs individual piecewiselinear approximation for the activation function of each neuron, and minimizesthe number of linear regions to satisfy a required approximation degree. Then,we analyze the upper bound of the number of linear regions formed by LANNs, andderive the complexity measure based on the upper bound. To examine theusefulness of the complexity measure, we experimentally explore the trainingprocess of neural networks and detect overfitting. Our results demonstrate thatthe occurrence of overfitting is positively correlated with the increase ofmodel complexity during training. We find that the $L^1$ and $L^2$regularizations suppress the increase of model complexity. Finally, we proposetwo approaches to prevent overfitting by directly constraining modelcomplexity, namely neuron pruning and customized $L^1$ regularization.

Measuring Model Complexity of Neural Networks with Curve Activation Functions