We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL
< prev | next >
new | recent | 1912

Change to browse by:

cs
cs.DC
cs.LG

References & Citations

DBLP - CS Bibliography

listing | bibtex

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Computer Science > Computation and Language

Title: Large-scale text processing pipeline with Apache Spark

Authors: Alexey Svyatkovskiy, Kosuke Imai, Mary Kroeger, Yuki Shiraito
(Submitted on 2 Dec 2019)
Abstract: In this paper, we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas.
We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark dataframes and Scala application programming interface. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.
Subjects: Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
Journal reference: Published in Proceedings of Big NLP workshop at the IEEE Big Data Conference 2016
Cite as: arXiv:1912.00547 [cs.CL]
  (or arXiv:1912.00547v1 [cs.CL] for this version)

Submission history

From: Alexey Svyatkovskiy [view email]
[v1] Mon, 2 Dec 2019 02:12:15 GMT (387kb,D)
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Link back to: arXiv, form interface, contact.