Stochastic Dynamics and Learning Rules in Layered Neural Networks

The perceptron is a typical feedforward neural network model which has capability of pattern recognition. The learning rule for the perceptron and its convergence was shown by Rosenblatt!) and Block) A typical perceptron has three layers and the connections between the second and third layers are modified by an error-correcting learning rule. Minsky and Papert analyzed the perceptron and showed its limitations.) Recently the PDP group proposed a learning rule to change the connections between the first layer and the second layer besides those between second and third layers.) Thereafter complicated problems have been sucessfully solved by the networks.) In their model, the output of each cell is expressed by a semilinear activation function of the weighed sum of the inputs through the connections. The connections are changed to decrease an error function. The learning rule is called 'back propagation of error' algorithm. Both the input-output relation and the learning rule are deterministic in their model. We propose two kinds of stochastic models in this paper. In the first model, the input-output relation is probabilistic. This model is similar to the Boltzmann machine),6) but the connections are not symmetric like the Boltzmann machine. So this model is an intermediate model between the perceptron and the Boltzmann machine. The other model uses a stochastic dynamics for the learning rule. In the deterministic model the system may encounter some local minima of an error function and it means that the problem cannot be solved. In the stochastic model the local minima can be got over with thermal noises, and it becomes possible to approach the correct solution by the simulated annealing technique.)