Apache Airflow, Data Engineering, Kubernetes
[13/23] Systems Distribution: A Cost-Effective Unified Data Platform for Small Organizations
Systems Distribution There are four different ways of executing tasks in Airflow. When a DAG is scheduled, each…
Apache Airflow, Data Engineering, Dev Ops
[12/23] Testing: A Cost-Effective Unified Data Platform for Small Organizations
Testing Apache Airflow An important aspect to take care of when writing software is testing. Test-driven development is…
Apache Airflow, Data Engineering
[11/23] Workflow: A Cost-Effective Unified Data Platform for Small Organizations
In order to build data pipelines between our data sources and data warehouse, the workflow of a DAG…
Apache Airflow, Data Engineering
[10/23] Operators: A Cost-Effective Unified Data Platform for Small Organizations
In Airflow, the way of creating a task is by using an operator. Since a DAG describes how…