Training Restricted Boltzmann Machines Using Modified Objective Function Based on Limiting the Free Energy Value

A restricted Boltzmann machine (RBM) is a generative model that can capture the probability distribution of data relevant to the problem domain and is usually deployed as the fundamental building block to form more complex deep architectures, such as deep belief network, deep Boltzmann machine, and deep stacked auto-encoder. In addition, the RBM itself can be used as a feature extractor to learn features from raw data. In addition, an RBM is a special type of energy-based model. This paper proposes a modified loss function as an example of an energy-based model defined by restricting the free energy value of the training data. This restriction punishes very low free energy value to reduce the model complexity, which is helpful to the training procedure of the RBM. We validate our method using the MNIST and MNIST-ROTATION datasets. Experiments reveal that the modified loss function behaves better in learning discriminative features as well as in providing better parameters when used to initialize deep feed-forward neural networks (DNN). The convergence speed of the DNN can improve by 44% on both datasets.