Скачать или смотреть EN.605.788.81.FA22 - Apache Airflow with integration of EMR - McDonough Meghan

Project: Apache Airflow with integration of EMR

Goal:
The goal of this project is to explore Amazon EMR (previously called Amazon Elastic
MapReduce) to host an Apache Airflow instance to schedule and run Hadoop jobs.

Description:
Amazon EMR is a managed cluster platform that simplifies running big data frameworks such as Apache Hadoop and Spark on AWS framework. Since this runs on AWS, this project will dive into its elastic nature, utilizing its built in Hadoop tools, and its intuitive nature on use.

In addition to hosting a Hadoop cluster on Amazon EMR, an Apache Airflow instance will be spun up for scheduling and orchestration of data pipelines and or workflows. A workflow is represented as a Directed Acyclic Graph or (DAG) that contains individual pieces of work called Tasks which can be arranged in any manner.

Finally, a DAG will be implemented for all the previous assignments so each job can be triggered at a moment’s notice. If time allows, another MR program will be written.