English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Context: The combination of distributed stream processing with microservicearchitectures is an emerging pattern for building data-intensive softwaresystems. In such systems, stream processing frameworks such as Apache Flink,Apache Kafka Streams, Apache Samza, Hazelcast Jet, or the Apache Beam SDK areused inside microservices to continuously process massive amounts of data in adistributed fashion. While all of these frameworks promote scalability as acore feature, there is only little empirical research evaluating and comparingtheir scalability. Objective: The goal of this study to obtain evidence aboutthe scalability of state-of-the-art stream processing framework in differentexecution environments and regarding different scalability dimensions. Method:We benchmark five modern stream processing frameworks regarding theirscalability using a systematic method. We conduct over 740 hours of experimentson Kubernetes clusters in the Google cloud and in a private cloud, where wedeploy up to 110 simultaneously running microservice instances, which processup to one million messages per second. Results: All benchmarked frameworksexhibit approximately linear scalability as long as sufficient cloud resourcesare provisioned. However, the frameworks show considerable differences in therate at which resources have to be added to cope with increasing load. There isno clear superior framework, but the ranking of the frameworks depends on theuse case. Using Apache Beam as an abstraction layer still comes at the cost ofsignificantly higher resource requirements regardless of the use case. Weobserve our results regardless of scaling load on a microservice, scaling thecomputational work performed inside the microservice, and the selected cloudenvironment. Moreover, vertical scaling can be a complementary measure toachieve scalability of stream processing frameworks.

Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud