We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.PL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Programming Languages

Title: Stream Processing With Dependency-Guided Synchronization (Extended Version)

Abstract: Real-time data processing applications with low latency requirements have led to the increasing popularity of stream processing systems. While such systems offer convenient APIs that can be used to achieve data parallelism automatically, they offer limited support for computations that require synchronization between parallel nodes. In this paper, we propose *dependency-guided synchronization (DGS)*, an alternative programming model for stateful streaming computations with complex synchronization requirements. In the proposed model, the input is viewed as partially ordered, and the program consists of a set of parallelization constructs which are applied to decompose the partial order and process events independently. Our programming model maps to an execution model called *synchronization plans* which supports synchronization between parallel nodes. Our evaluation shows that APIs offered by two widely used systems -- Flink and Timely Dataflow -- cannot suitably expose parallelism in some representative applications. In contrast, DGS enables implementations with scalable performance, the resulting synchronization plans offer throughput improvements when implemented manually in existing systems, and the programming overhead is small compared to writing sequential code.
Comments: 41 pages. Non-extended version to appear at Principles and Practice of Parallel Programming (PPoPP), February 2022
Subjects: Programming Languages (cs.PL); Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as: arXiv:2104.04512 [cs.PL]
  (or arXiv:2104.04512v3 [cs.PL] for this version)

Submission history

From: Caleb Stanford [view email]
[v1] Fri, 9 Apr 2021 17:50:53 GMT (247kb,D)
[v2] Sat, 17 Apr 2021 05:12:14 GMT (203kb,D)
[v3] Mon, 3 Jan 2022 08:11:19 GMT (1293kb,D)

Link back to: arXiv, form interface, contact.