Airflow

The Airflow page (/airflow) shows what each Airflow DAG and task costs in BigQuery. It works for self-hosted Airflow, Astronomer and Cloud Composer. It needs no setup and works on every plan, including the trial.

How SlashPeak finds Airflow jobsCopy Anchor Link

BigQueryInsertJobOperator (in apache-airflow-providers-google) marks each BigQuery job it starts:

• It adds the job label airflow-dag with the DAG id.
• It adds the job label airflow-task with the task id.
• It names the job airflow_<dag>_<task>_<time>_<hash>, unless the DAG sets its own job_id.

SlashPeak reads these on every scan. A job with either label, or with the airflow_ job name, counts as Airflow. A job whose SQL comment names Airflow counts too.

Airflow writes the ids in lower case. It skips both labels when a DAG id or task id breaks BigQuery's label rules, for example when it is longer than 63 characters or has a dot.

Jobs with no DAG labelCopy Anchor Link

Some Airflow jobs have the airflow_ job name or an Airflow comment, but no airflow-dag label. The page puts their cost in one No DAG label row.

Other operators and dbtCopy Anchor Link

Other ways to run BigQuery from Airflow do not add the labels. Examples are a PythonOperator with the BigQuery client, or dbt run from a BashOperator. To see these jobs per DAG, add the two labels to each job yourself:

job_config = bigquery.QueryJobConfig(
    labels={"airflow-dag": dag_id.lower(), "airflow-task": task_id.lower()},
)

dbt jobs stay on the dbt page, even when Airflow starts them.

The Airflow pageCopy Anchor Link

• Spend: Airflow job cost per day, one area per DAG. The breakdown shows cost By DAG and By task.
• Most expensive: DAGs by cost, tasks by cost, and tasks by bytes billed. When every Airflow job runs on reservations, the last list shows slot-hours instead.
• DAGs: every DAG that ran BigQuery jobs in the window, most expensive first. Open a DAG to see its tasks, their jobs, failures and last run. Open a task to see its destination table and its latest SQL.

The page has the same window, project, region and user filters as the Usage pages.

Find the job of a slow taskCopy Anchor Link

The task list shows each task's latest SQL. Use it to find the query on the Findings and Usage pages. Findings name Airflow as the workload source.