Скачать или смотреть Visual Flow, open-source ETL tool powered by Apache Spark - Dmitri Pavlov, Alexander Shevtsov

Thank you to all of of our ApacheCon@Home 2021 sponsors, including:

STRATEGIC
---------------
Google

PLATINUM
--------------
Apple
Huawei
Instaclustr
Tencent Cloud

GOLD
-----------
Aiven OY
AWS
Baidu
Cerner
Didi Chuxing
Dremio
Fiter
Gradle
Red Hat
Replicated A presentation from ApacheCon@Home 2021
https://apachecon.com/acah2021

We want to present Visual Flow – a cloud-native open-source ETL tool based on Apache Spark unified analytics engine with a Graphical User Interface. It combines the best features of Spark, Kubernetes and Argo into a single system with the following features:

Portability, flexibility and multi-cloud compatibility
Increased developer productivity
High availability, performance and fault tolerance
Cost-effective
Open-source

Visual Flow provides a Graphical User Interface to create ETL jobs and combine them in Data Processing Pipelines, run/schedule them, and monitor execution. It is a multi-tenant system. You can create multiple projects in a single cluster. Each project is represented as a namespace, which allows allocating resources and providing access to the users in different roles.
Visual Flow does not require a database. It is possible to create all objects as native Kubernetes resources.
Visual Flow leverages Kubernetes authorization to manage users and their roles within a project.
Visual Flow provides the ability to create parameters (e.g., connection info) and reuse them in Jobs.