Apache Airflow (AA) is an open-source software. This type of software comes with common issues related to DevSecOps operational management.
The AA open-source project is mainly concerned with the workflow engine. It leaves the deployment strategy, security, and maintenance to the companies that want to use AA as their workflow manager.
To provide production-ready AA cloud deployment, Translucent Computing uses two different management approaches: self-managed and cloud-managed.
Before we look at each one, here’s how we define both:
Self-managed: managing a system within a Kubernetes cluster
Cloud-managed: managing a system through a cloud provider-managed service
Self-managed AA assumes that your organization has the resources and skills to manage the AA’s secure deployment and maintenance. Beyond the security, depending on the industry, deployment is further complicated by regulations related to data and privacy. To adhere to robust security standards, compliance and regulations, Translucent adds additional technologies to support AA, including:
All these tools must be deployed and configured. Translucent found that the most efficient way to deploy and configure the cloud infrastructure required for AA is with CI/CD pipeline that uses infrastructure as code (IaC).
Managing secure and production-ready AA puts a strain on DevSecOps teams. With Google Cloud Composer, we now have an option to use a cloud-managed service to reduce the burden on the DevSecOps team. Google Cloud Composer marries the Google Cloud Platform with the AA.
The Google Cloud Platform provides enterprise features including security, deployments, and AA management. The Google Cloud Composer manages the AA metadata store and web server, and provides security tools and observability tools while using Kubernetes to execute AA workflows.
The AA management approach depends on use cases and DevSecOps. While you do have complete control when you self-manage AA, the complexity of a secure production-ready system requires investment in DevSecOps and continuous investment in maintaining the system.
However, for Translucent, the main benefit of a self-managed AA is cloud independence. We can provide a secure and production-ready workflow manager in any cloud and on-premises built on the Kubernetes cluster.
At the same time, Cloud Composer is also attractive for most Translucent use cases, because of benefits like deployment speed, managed environment, predictable cost, and code portability from other AA.
November 26th, 2021⟵ Back
June 9th, 2022How Translucent is Changing The Cloud Native Industry