Streamrag: A Lock-Aware and Traffic-Aware Query Coordinator in Stream-Based Rag Systems

초록

Stream-based retrieval augmented generation (RAG) systems integrate stream processing engines (SPEs) with real-time document retrieval to support dynamic indexing and search over unstructured datasets. However, efficiently executing queries in these systems is challenging, as seamless coordination between SPEs and vector databases is essential for maintaining low latency and high throughput. The lack of mutual awareness between these components results in two major performance bottlenecks. First, SPEs are unaware of ongoing indexing operations in vector databases, leading to metadata lock contention when indexing and search operations overlap, which increases query latency. Second, vector databases lack visibility into query traffic from SPEs and rely solely on internal metrics for scaling. As a result, they respond reactively to traffic spikes, often leading to instance overload and delayed query processing. To address these issues, we propose STREAMRAG, a lock-aware and traffic-aware query coordination mechanism that facilitates real-time exchange of metadata lock statuses and query traffic metrics between SPEs and vector databases. By optimizing query routing and enabling proactive instance scaling, STREAMRAG enhances the performance and scalability of real-time RAG systems. Experimental results demonstrate that STREAMRAG reduces tail latency by up to 4x at the 99th percentile and significantly improves overall system performance under varying traffic conditions.

키워드