Lead Software Engineer at HappyFresh
"Dealing Workflow using Airflow"
Lead Engineer at HappyFresh, Open source enthusiast, Peugeot driver
Apache airflow is a platform to programmatically author, schedule and monitor workflows. It originally created by Airbnb and recently has become Apache Top-level Project. Many companies use Airflow to manage their workflows. One of them is HappyFresh. HappyFresh utilize Airflow to manage and monitor its ETL process. Another use cases that suitable for Airflow are: machine learning pipelines, data warehousing, orchestrating automated test, etc.
Airflow is written using Python, and it extensible. You just need to define your entire workflow using python code, or you can extend Airflow using your own python modules.
This talk will discuss about Airflow, define DAG for workflow, and how to scale Airflow for huge number of workflow. This is suitable for all audience levels, especially who wants to manage their workflow.