We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.DC

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Distributed, Parallel, and Cluster Computing

Title: A Random Sample Partition Data Model for Big Data Analysis

Abstract: Big data sets must be carefully partitioned into statistically similar data subsets that can be used as representative samples for big data analysis tasks. In this paper, we propose the random sample partition (RSP) data model to represent a big data set as a set of non-overlapping data subsets, called RSP data blocks, where each RSP data block has a probability distribution similar to the whole big data set. Under this data model, efficient block level sampling is used to randomly select RSP data blocks, replacing expensive record level sampling to select sample data from a big distributed data set on a computing cluster. We show how RSP data blocks can be employed to estimate statistics of a big data set and build models which are equivalent to those built from the whole big data set. In this approach, analysis of a big data set becomes analysis of few RSP data blocks which have been generated in advance on the computing cluster. Therefore, the new method for data analysis based on RSP data blocks is scalable to big data.
Comments: 9 pages, 7 figures
Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Data Structures and Algorithms (cs.DS); Data Analysis, Statistics and Probability (physics.data-an); Machine Learning (stat.ML)
DOI: 10.1109/TII.2019.2912723
Cite as: arXiv:1712.04146 [cs.DC]
  (or arXiv:1712.04146v2 [cs.DC] for this version)

Submission history

From: Salman Salloum [view email]
[v1] Tue, 12 Dec 2017 06:49:28 GMT (1067kb,D)
[v2] Sat, 20 Jan 2018 10:59:15 GMT (2286kb)

Link back to: arXiv, form interface, contact.