PPolyNets: Achieving High Prediction Accuracy and Efficiency With Parametric Polynomial Activations

Recently, to implement cloud-based machine learning approaches while maintaining data privacy, a scheme named CryptoNets is proposed to perform prediction on encrypted data using neural networks. By applying the leveled homomorphic encryption scheme, CryptoNets enables the cloud server to securely run the computation process without participations of other parties. Since the encryption scheme only supports polynomial operations, the authors simply use square activation as a substitution of the conventional activations, obtaining a relatively low prediction accuracy on the MNIST dataset (98.95%). Later work try to improve the accuracy by using polynomial approximations of the ReLU activation with large neural networks, which introduce heavy computation cost, making the prediction process impractical. In this paper, to achieve better prediction performance, we propose new parametric polynomial (PPoly) activations, which can adaptively learn the parameters during the training phase. Using our PPoly activations, we achieve higher accuracy (99.33%, 99.64%, and 99.70%) with shallow and narrow networks, guaranteeing the efficiency of prediction process. We conduct extensive experiments to show the expressiveness of our PPoly activations and discuss the tradeoff between accuracy and efficiency for the prediction on encrypted data.