We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

astro-ph.IM

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Astrophysics > Instrumentation and Methods for Astrophysics

Title: Adding Workflow Management Flexibility to LSST Pipelines Execution

Authors: Michelle Gower (1), Mikolaj Kowalik (1), Nate B. Lust (2), James F. Bosch (2), Tim Jenness (3) ((1) NCSA, (2) Princeton University, (3) Vera C. Rubin Observatory)
Abstract: Data processing pipelines need to be executed at scales ranging from small runs up through large production data release runs resulting in millions of data products. As part of the Rubin Observatory's pipeline execution system, BPS is the abstraction layer that provides an interface to different Workflow Management Systems (WMS) such as HTCondor and PanDA. During the submission process, the pipeline execution system interacts with the Data Butler to produce a science-oriented execution graph from algorithmic tasks. BPS converts this execution graph to a workflow graph and then uses a WMS-specific plugin to submit and manage the workflow. Here we will discuss the architectural design of this interface and report briefly on the recent production of the Data Preview 0.2 release and how the system is used by pipeline developers.
Comments: 4 pages, submitted to Astronomical Data Analysis Software and Systems XXXII, October 2022
Subjects: Instrumentation and Methods for Astrophysics (astro-ph.IM); Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as: arXiv:2211.15795 [astro-ph.IM]
  (or arXiv:2211.15795v1 [astro-ph.IM] for this version)

Submission history

From: Tim Jenness [view email]
[v1] Mon, 28 Nov 2022 22:01:37 GMT (18kb,D)

Link back to: arXiv, form interface, contact.