Apache Airflow, Data Engineering, Dev Ops

[16/23] Deployment Requirements: A Cost-Effective Unified Data Platform for Small Organizations

Requirements Highly Available An essential requirement is that the architecture must be highly available. That means if a…

Apache Airflow, Data Engineering, Security

[15/23] Security: A Cost-Effective Unified Data Platform for Small Organizations

Security It is crucial to take care of the security of the pipelines when using Airflow in production.…

Apache Airflow, Data Engineering, Dev Ops, ElasticSearch

[14/23] Monitoring: A Cost-Effective Unified Data Platform for Small Organizations

Monitoring Monitoring the Airflow instances and DAGs are a top priority in production. Without a monitoring system, there…

Apache Airflow, Data Engineering, Kubernetes

[13/23] Systems Distribution: A Cost-Effective Unified Data Platform for Small Organizations

Systems Distribution There are four different ways of executing tasks in Airflow. When a DAG is scheduled, each…

1 2 3 5