Open AccessDesign and Implementation of Real Time Data Processing System Based on Spark Streaming
Author(s) -
Haibo He
Publication year - 2021
Publication title -
xue xi yu jiao yu
Language(s) - English
Resource type - Journals
ISSN - 2251-2802
DOI - 10.18282/l-e.v10i3.2451
Subject(s) - spark (programming language) , computer science , big data , the internet , data processing , order (exchange) , data science , database , world wide web , operating system , finance , economics , programming language
With the rapid development of Internet technology, people have more abundant ways to obtain information through the Internet.More and more information is spread in the network, which will produce a large amount of data.The big data type is more complicated. In order to implement data implementation, the industry is generally combined with different data types and business scenes, special development and design of different processing subsystems.This method has a large research cost and learning cost, and a unified computing system platform is relatively lacking, which further makes developers unable to maintain and expand the system formed by different sets of technology systems.Based on the above situation, this paper designs a universal real-time data analysis and processing system based on Spark Streaming.