Meridian Stack

Running Airflow on two vCPUs

2026-10-12 · Jonas Weber · 5 min read

A small research group wanted Airflow, not a platform team. Their budget bought two vCPUs and 4 GB. Everyone told them to use a managed service. They asked us to make it work anyway, and it does — with a few settings that are not in the quickstart.

Where the memory goes

The scheduler is not the problem. The webserver is, because gunicorn workers each load the whole DAG directory. On a small box, one worker and a hard DAG parse timeout beat four workers that swap.

[webserver]
workers = 1
worker_refresh_interval = 600
[core]
min_file_process_interval = 60
dag_file_processor_timeout = 60
parallelism = 4

The scheduler

With fewer than twenty DAGs, the scheduler is mostly idle. The killer is the metadata database: SQLite works until it does not, so we moved it to Postgres on the same box with a small shared_buffers and no autovacuum during the workday.

What we did not do

We did not install a second scheduler, a celery worker farm, or a redis broker. For this workload, LocalExecutor with a parallelism of four finished every job within the SLA and stayed under 60% memory.

Eight months later it is still running, the box has not been resized, and the team writes DAGs instead of maintaining infrastructure. Sometimes the boring answer is the right one.