Running Spark jobs on Amazon EMR Serverless

Get an overview of how to run Apache Spark jobs in EMR Serverless from the AWS Console, CLI, and using Amazon Managed Workflows for Apache Airflow (MWAA).

Also see how to use the new CloudWatch Metrics to monitor EMR Serverless usage, Live Dashboard UI, and package your PySpark jobs with virtual environments.

Table of Contents:

00:00 - Intro
02:01 - Create application in the console
02:47 - Pre-initialized Capacity
05:43 - Running jobs from the console
07:19 - Spark History Server
09:47 - Running jobs in the CLI
12:31 - CloudWatch Dashboard
15:08 - Live Spark UI
16:42 - Running EMR Serverless jobs with Airflow
22:25 - Budling Python dependencies
23:43 - Custom Python version