zStream: towards a low latency micro-batch streaming system

초록

Conventional micro-batch streaming systems introduce high tail latency for the following two reasons. First, they trigger data processing on a fixed time interval, without taking into account the data ingestion rate and data analysis processing rate. Second, in a multi-query environment, their scheduling methods do not consider the real-time characteristics of the application and each query. This paper presents zStream, a low latency micro-batch streaming system implemented on Apache Spark. The main idea of zStream's design is to use a reference value called deadline for each query to constrain latency around a specific time. Considering the deadline information of each query, zStream controls the admission of every micro-batch dynamically and schedules them by their priorities, significantly reducing the average and tail latency while maintaining throughput. Experimental results show that the tail latency of zStream is reduced by a maximum of 34% compared to the original Spark. Also, zStream reduces the average latency per query by up to 48% while stably maintaining throughput compared to the original Spark.

키워드