References & Citations
Computer Science > Distributed, Parallel, and Cluster Computing
Title: Selecting Efficient Cluster Resources for Data Analytics: When and How to Allocate for In-Memory Processing?
(Submitted on 6 Jun 2023 (v1), last revised 7 Jun 2023 (this version, v2))
Abstract: Distributed dataflow systems such as Apache Spark or Apache Flink enable parallel, in-memory data processing on large clusters of commodity hardware. Consequently, the appropriate amount of memory to allocate to the cluster is a crucial consideration.
In this paper, we analyze the challenge of efficient resource allocation for distributed data processing, focusing on memory. We emphasize that in-memory processing with in-memory data processing frameworks can undermine resource efficiency. Based on the findings of our trace data analysis, we compile requirements towards an automated solution for efficient cluster resource allocation.
Submission history
From: Jonathan Will [view email][v1] Tue, 6 Jun 2023 13:37:18 GMT (129kb,D)
[v2] Wed, 7 Jun 2023 15:28:20 GMT (106kb,D)
Link back to: arXiv, form interface, contact.