Data Collection and Monitoring Across Heterogeneous Workflows in Pegasus

During this installment of Pegasus Office hours, we will be presenting a data collection pipeline we have developed to enable researchers of the Laser Inferometer Gravitational-Wave Observatory (LIGO) to easily monitor and analyze complex scientific workflows.

First, we discuss the need for such data collection and monitoring solutions. This includes an introduction into our work: Domain-Aware Management of Heterogeneous Workflows: Active Data Management for Gravitational-Wave Science Workflows.

Following that will be an overview of the data collection pipeline itself along the components used: Elasticasearch, Logstash, Kibana, RabbitMQ, and Grafana.

Finally, we provide a containerized solution so that you can get up and running collecting data from your Pegasus workflow runs. A demonstration will be held covering the setup of this pipeline, configuring Pegasus to publish data, and how to use the dashboard we have implemented.