Apache Airflow, Data Engineering, Kubernetes

[13/23] Systems Distribution: A Cost-Effective Unified Data Platform for Small Organizations

Systems Distribution There are four different ways of executing tasks in Airflow. When a DAG is scheduled, each…

Apache Airflow, Data Engineering, Dev Ops

[12/23] Testing: A Cost-Effective Unified Data Platform for Small Organizations

Testing Apache Airflow An important aspect to take care of when writing software is testing. Test-driven development is…

Apache Airflow, Data Engineering

[11/23] Workflow: A Cost-Effective Unified Data Platform for Small Organizations

In order to build data pipelines between our data sources and data warehouse, the workflow of a DAG…

Apache Airflow, Data Engineering

[10/23] Operators: A Cost-Effective Unified Data Platform for Small Organizations

In Airflow, the way of creating a task is by using an operator. Since a DAG describes how…

1 2 3