Amazon Web Services, Apache Airflow, Data Engineering, Dev Ops, Kubernetes

[17/23] Deployment Architecture: A Cost-Effective Unified Data Platform for Small Organizations

Deployment Architecture The schema of the architecture used for deploying the ETL pipeline is illustrated in the figure…

Apache Airflow, Data Engineering, Dev Ops

[16/23] Deployment Requirements: A Cost-Effective Unified Data Platform for Small Organizations

Requirements Highly Available An essential requirement is that the architecture must be highly available. That means if a…

Apache Airflow, Data Engineering, Dev Ops, ElasticSearch

[14/23] Monitoring: A Cost-Effective Unified Data Platform for Small Organizations

Monitoring Monitoring the Airflow instances and DAGs are a top priority in production. Without a monitoring system, there…

Apache Airflow, Data Engineering, Dev Ops

[12/23] Testing: A Cost-Effective Unified Data Platform for Small Organizations

Testing Apache Airflow An important aspect to take care of when writing software is testing. Test-driven development is…

1 2