English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Distributed data stream processing systems(DSPS) are widely used in real-time massive data processing scenarios for its characteristics of real-time and high throughout. In the real-world DSPS, the fluctuating arrival rate of the input data leads the consuming computing resource of DSPS to be time-variable. To guarantee the performance of DSPS, the accurate prediction of DSPS’s consuming resources is necessary. In this paper, we proposed approaches to make the online prediction of computing resources that DSPS consumes. We monitor the usage of computing resources such as CPU and memory in a DSPS, and use temporal data streams clustering algorithm and linear regression method to make online prediction of CPU resources and memory resources respectively. Our prediction approaches are proved efficient and quickly enough. Introduction In a DSPS, the workload at a time is variable since the amount of input data is variable over time, which causes the demand of computing resources is variable. Fig. 1 shows the variation of user visits of twitter in two weeks [1], from which we can see that the user’s access behavior is periodical. Based on this, we did an experiment to monitor the usage of CPU and memory resource with the fluctuation arriving data set as input. We conduct the experiment in JStorm [2] and use SequenceTopology as benchmark. Figure 1: user visits in Twitter Figure 2: CPU variation curve Figure 3: Memory variation curve Fig. 2 and Fig. 3 shows that different type of resource shows different usage characteristic with different data arrival speed. With the increase of input data speed the demand of CPU resource increase while the usage of memory resource varies little with it. However, memory shows strong relationship with running time. To predict the usage of resource, Khoshkbarforoushha [3] made an offline model to predict both CPU and memory, while as we conclude above that the demand of CPU and memory depends on different variate, which make the prediction inaccuracy. In this paper, we propose two online adaptive algorithms to predict the usage of CPU and memory respectively. Our contributions are as follows: • We found the different variation characteristics of CPU resource and memory resource with input 2nd Workshop on Advanced Research and Technology in Industry Applications (WARTIA 2016) © 2016. The authors Published by Atlantis Press 1824 data speed and time. • We propose two algorithms to predict the usage CPU and memory resource online. • We conduct an experiment in real application scenario and make performance evaluation toward these two resource prediction approaches. CPU Resource Prediction In our prediction of CPU resource, the current usage amount of CPU should be captured to predict the usage of it in the next window. We define the CPU usage set is 1 2 { , ,..., }( (1, )) n C c c c n t = ∈ , which is a continuously updated real-time stream that a temporal data streams clustering algorithm is necessary. Therefore we propose a clustering algorithm [4] to acquire the usage amount of CPU in the next window by finding the most similar item from historical data set. We transfer set C into resource variation set 2 3 ' { , ,..., } n C c c c = ∆ ∆ ∆ , among which n c ∆ is calculated by Equation -1 n n n c c c ∆ = . The current usage variation of CPU is n c ∆ ,we cut off C' according to the length l and form data stream sub-sequence 2 3 { , ,..., } n l R R R R − = as the point in clustering space. Each element in sub-sequence is the dimension of point in the clustering, as Eq. 1 shows. l is a variation representing the size of sliding window. The procedure of clustering is defined as Eq. 2:

Resource Estimation in Distributed Data Stream Processing Systems