awesome-apache-airflow

Curated list of resources about Apache Airflow

Github星跟踪图

Awesome Apache Airflow contrib badge

This is a curated list of resources about Apache Airflow. Please feel free to contribute any items that should be included. Items are generally added at the top of each section so that more fresh items are featured more prominently.

Contents

Airflow deployment solutions

Introductions and tutorials

Best practices, lessons learned and cool use cases

Books, blogs, podcasts, and such

Slide deck presentations and online videos

Libraries, Hooks, Utilities

  • Airflow plugins - Central collection of repositories of various plugins for Airflow, including mailchimp, trello, sftp, GitHub, etc.
  • fileflow - Collection of modules to support large data transfers between Airflow operators through either local file system or S3. This addresses a gap where data is too large for XCOMs but too small or inconvenient for loading directly in the operator. Built by Industry Dive.
  • fairflow - Library to abstract away Airflow's Operators with functional pieces that transform the data from one operator to another.
  • airflow-maintenance-dags - Clairvoyant has a repo of Airflow DAGs that operator on Airflow itself, clearing out various bits of the backing metadata store.
  • test_dags - a more complete solution for DAG integrity tests (first Circle of Data’s Inferno are the first.
  • dag-factory - A library for dynamically generating Apache Airflow DAGs from YAML configuration files.
  • whirl - Fast iterative local development and testing of Apache Airflow workflows.
  • airflow-code-editor - A plugin for Apache Airflow that allows you to edit DAGs in browser.
  • Pylint-Airflow - A Pylint plugin for static code analysis on Airflow code.
  • afctl - A CLI tool that includes everything required to create, manage and deploy airflow projects faster and smoother.

Meetups

Commercial Airflow-as-a-service providers

  • Google Cloud Composer - Google Cloud Composer is a managed service built atop Google Cloud and Airflow.
  • Qubole - Qubole is mainly known as a service-and-support company for Apache Hive, but also provides Airflow as a component of its platform.
  • Astronomer.io - Astronomer provides complete ETL lifecycle solutions and appears to be entirely focused on providing Airflow-based products.

Non-English resources

License

CC0

To the extent possible under law, Jakob Homan has waived all copyright and related or neighboring rights to this work.

主要指标

概览
名称与所有者jghoman/awesome-apache-airflow
主编程语言Shell
编程语言Shell (语言数: 1)
平台
许可证
所有者活动
创建于2018-01-11 07:14:19
推送于2024-08-20 21:13:45
最后一次提交2024-08-20 14:13:45
发布数0
用户参与
星数3.8k
关注者数160
派生数496
提交数357
已启用问题?
问题数6
打开的问题数2
拉请求数113
打开的拉请求数0
关闭的拉请求数5
项目设置
已启用Wiki?
已存档?
是复刻?
已锁定?
是镜像?
是私有?