![]() ![]() ![]() We did not want the hassle of bringing up a distributed infrastructure involving a distributed broker and a set of remote workers. As an early adopter, we were looking for a workflow scheduler that was easy to install, maintain, and run in the cloud. We run airflow in both QA and Production, which essentially means that the above architecture is replicated in 2 environments. The two airflow components (webserver and scheduler) run on a single machine, whereas the database is running on a shared database instance. More modest installations can use a LocalExecutor and get a fair amount of mileage out of that. a metadata database (mysql or postgres)Īll of this can run on a single box, scale at will. ![]() In a scalable production environment, Airflow has the following components InfoQ spoke to Airflow’s creator, Maxime Beauchemin, and Agari’s Data Architect and one of the framework’s early adopters Siddharth Anand, to discuss Airflow including where it can be of use and what’s planned for the future.Ĭould you give us a high level overview of Airflow’s architecture? As well as a command line interface, the tool provides a web-based UI which allows you to visualize your pipelines dependencies, monitor progress, trigger tasks and so on. The platform has mechanisms to interact with Hive, Presto, MySQL, HDFS, Postgres and S3, and hooks are provided to allow the system to be made more extensible. For Airbnb, this includes use-cases across multiple departments such as data warehousing, growth analytics, email targeting, A/B testing and so on. Directed Acyclic Graphs or else DAGs) per a periodic schedule. The platform is written in Python, as are any workflows that run on it.Īirflow is a tool that allows developers of workflows to easily author, maintain, and run workflows (a.k.a. Airflow is being used internally at Airbnb to build, monitor and adjust data pipelines. Airbnb recently open-sourced Airflow, its own data workflow management framework, under the Apache license. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |