Open AccessMachine Learning Algorithm Based on Big Data
Author(s) -
Hao Huang
Publication year - 2020
Publication title -
journal of physics. conference series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.21
H-Index - 85
eISSN - 1742-6596
pISSN - 1742-6588
DOI - 10.1088/1742-6596/1533/4/042015
Subject(s) - computer science , big data , machine learning , online machine learning , artificial intelligence , schedule , algorithm , key (lock) , data mining , stability (learning theory) , computer security , operating system
Machine learning technology is indispensable for big data. In machine learning, data on a large scale can improve the accuracy of the model. However, complex machine learning algorithms require the key technology of distributed memory computing in time and performance. Big data memory computing can implement the parallel operation of the algorithm, which is conducive to the processing of big data sets by the machine learning algorithm. Hence, a nonlinear machine learning algorithm implemented in the big data memory environment is proposed in this paper, where data compression, biased sampling or loading based on the implementation is optimized. To fully configure resources for the script running by batch, we also implemented a machine learning framework to schedule the optimized algorithm mentioned above. The experimental results showed that the mean error of the three algorithms after optimization was reduced by 40%, and the mean time was reduced by 90%.