We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

astro-ph.IM

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Astrophysics > Instrumentation and Methods for Astrophysics

Title: Development of a High Throughput Cloud-Based Data Pipeline for 21 cm Cosmology

Abstract: We present a case study of a cloud-based computational workflow for processing large astronomical data sets from the Murchison Widefield Array (MWA) cosmology experiment. Cloud computing is well-suited to large-scale, episodic computation because it offers extreme scalability in a pay-for-use model. This facilitates fast turnaround times for testing computationally expensive analysis techniques. We describe how we have used the Amazon Web Services (AWS) cloud platform to efficiently and economically test and implement our data analysis pipeline. We discuss the challenges of working with the AWS spot market, which reduces costs at the expense of longer processing turnaround times, and we explore this tradeoff with a Monte Carlo simulation.
Comments: Accepted for publication in Astronomy and Computing
Subjects: Instrumentation and Methods for Astrophysics (astro-ph.IM)
Cite as: arXiv:2009.10223 [astro-ph.IM]
  (or arXiv:2009.10223v2 [astro-ph.IM] for this version)

Submission history

From: Ruby Byrne [view email]
[v1] Mon, 21 Sep 2020 23:48:28 GMT (1255kb,D)
[v2] Tue, 2 Mar 2021 23:07:45 GMT (1007kb,D)

Link back to: arXiv, form interface, contact.