Uma arquitetura de referência para o processamento distribuído de stream de dados em soluções analíticas de near real-time

A REFERENCE ARCHITECTURE FOR DISTRIBUTED PROCESSING STREAMS OF DATA FOR NEAR REAL-TIME ANALYTICS Author: Daniel da Cunha Rodrigues de Souza Supervisor: Rafael Timóteo de Sousa Júnior Co-Supervisor: Edison Pignaton de Freitas Programa de Pós-graduação em Engenharia Elétrica Brasília, March of 2015 The current requirement of low latency processing for high volume of data streams is pushing the limits of the traditional data processing architectures. A new class of applications called Distributed Stream Processing Systems (DSPS) has emerged to facilitate such large scale real time data analytics. Nevertheless the diversity of architectures, data models and APIs introduced by the use of these systems resulted in a greater complexity to the development of data processing systems. In this context, a reference architecture to data stream processing for near real-time analytics is proposed in this work. This proposal is based on a layered architecture pattern, with clearly defined responsibilities providing a strong reference model, to improve the maintainability and reuse for data stream processing systems. In order to evaluate the proposed architecture and its framework, a case study is used in which two probabilistic algorithms are applied: the HyperLogLog and the Count-Min Sketch.