Overview

Using Terraform and Helm, and following Rodel van Rooijen's Medium article, I create an Airflow cluster on GKE without using gcloud or kubectl commands.

The Terraform and Helm code is available in the GitHub repository below.

github.com

Please note that there are several differences from the original article.

The Airflow Web UI will be exposed to the internet.
You do not need to manually create a Kubernetes Persistent Volume (kubectl apply). A Compute Disk is automatically created by the Kubernetes Persistent Volume Claim. For details, see Persistent volumes and dynamic provisioning.
Logs are only available during the Pod's lifecycle. If you want to persist logs, see Airflow Helm: Manage logs.
Compute Engine Disk is not used for managing DAG files. If you want to manage DAG files efficiently, see Airflow Helm: Manage DAGs files.

What to Prepare Before terraform apply

To run terraform apply, you need the following command-line tools:

gcloud
docker
kubectl
helm

Also, please note the following:

The Terraform code deletes the ~/.kube directory to overwrite GKE credentials. Back up the ~/.kube directory beforehand if needed.
Before applying Terraform, the GCP project must already be created and all necessary GCP APIs must be enabled.

Caveats

This code is not production-level in the following respects:

The default VPC network should be replaced with a custom VPC.
The Cloud SQL password should be managed using a more secure method such as Secret Manager, rather than a Terraform variable.
The Terraform state file should be stored in GCS (Google Cloud Storage), not in a local environment.
Managing DAG files with git-sync is more convenient.
Logs are only available during the Pod's lifecycle.

I do think this Airflow environment can be useful for things like DAG development.

Pain Points

There are two types of Airflow Helm charts: the official Helm chart and the community version.

Not knowing this, I researched them in a confused state and ended up with a muddled understanding, which cost me a lot of time.

This repository uses the official Helm chart.

Observations

Notes on things I noticed during development.

Compute Engine Disk does not support the ReadWriteMany access mode for PVC (Persistent Volume Claim).
The standards-rwo Storage Class does not create a Compute Engine Disk immediately after a PVC is created.

References

Deploying Airflow on GKE using Helm

Workload Identity in GKE with Terraform

Dynamic Provisioning and Storage Classes in Kubernetes

Persistent volumes and dynamic provisioning

GitHub: airflow-helm/charts

Apache Airflow ETL in Google Cloud

Alternative: link Kubernetes ServiceAccounts to IAM

Deploying Airflow on Google Kubernetes Engine with Helm

Deploying Airflow on Google Kubernetes Engine with Helm — Part Two

Airflow Helm: Production Guide