Dev blog

How to execute DBT Core Commands with Airflow dags

This is an additional step for our local data stack, which includes a Dev container, dbt, PostgreSQL, Metabase, and Airflow. With Airflow, we can schedule the execution of our project.

I will explain how to run the dbt run, dbt clean, and dbt compile commands. First, I will explain how to add the Airflow image to our Docker Compose file, how to add the dbt project to Airflow, and finally, how to execute the dbt Core commands.

You can check the code in my GitHub repository.

How to add the aiflow docker image to the local stack of DBT, postgres and Metabase ?

I based my Docker Compose file on the one provided in the Airflow documentation. I modified it by changing the PostgreSQL database name, adding the metanet1 network, using a custom Docker image, and adding a Dockerfile. In addition, I mounted our dbt project in the /opt/airflow/dbt directory with the following volume mapping: ../:/opt/airflow/dbt/

x-airflow-common:
  &airflow-common
  # In order to add custom dependencies or upgrade provider distributions you can use your extended image.
  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  build:
    context: ..
    dockerfile: .devcontainer/DockerFile2
  image: custom-airflow:3.3.0
  # build: .
  env_file:
    - .env
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@airflow-postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql+psycopg2://airflow:airflow@airflow-postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: 65c53be52b00b00f259f696992eabba0dc13608e2df7bc8e450745e3ebca804c56f2f3f33c51f215f762aa7c3c7e23c1b3cd548a1c834c49390b73b5c6b360e8
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
    AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'
    AIRFLOW__API_AUTH__JWT_SECRET: ${AIRFLOW__API_AUTH__JWT_SECRET:-airflow_jwt_secret}
    AIRFLOW__API_AUTH__JWT_ISSUER: ${AIRFLOW__API_AUTH__JWT_ISSUER:-airflow}
    # yamllint disable rule:line-length
    # Use simple http server on scheduler for health checks
    # See https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/check-health.html#scheduler-health-check-server
    # yamllint enable rule:line-length
    AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true'
    # WARNING: Use _PIP_ADDITIONAL_REQUIREMENTS option ONLY for a quick checks
    # for other purpose (development, test and especially production usage) build/extend Airflow image.
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
    # The following line can be used to set a custom config file, stored in the local config folder
    AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'
  volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
    - ../:/opt/airflow/dbt/
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    &airflow-common-depends-on
    redis:
      condition: service_healthy
    airflow-postgres:
      condition: service_healthy
  networks:
    - metanet1

services:
  dbt_service:
    build:
      context: ..
      dockerfile: .devcontainer/DockerFile
    volumes:
      - ..:/dbt:cached
    command: sleep infinity

    user: root
    networks:
      - metanet1    

  db:
    image: postgres:17.0
    restart: always
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "15432:5432"
    networks:
      - metanet1

  metabase:
    image: metabase/metabase:latest
    container_name: metabase
    hostname: metabase
    volumes:
      - /dev/urandom:/dev/random:ro
    ports:
      - "3000:3000"
    environment:
      MB_DB_TYPE: postgres
      MB_DB_DBNAME: metabaseappdb
      MB_DB_PORT: 5432
      MB_DB_USER: metabase
      MB_DB_PASS: mysecretpassword
      MB_DB_HOST: postgres
    
    networks:
      - metanet1
    healthcheck:
      test: ["CMD", "curl", "--fail", "-I", "http://localhost:3000/api/health"]
      interval: 15s
      timeout: 5s
      retries: 5
    depends_on:
      - postgres

  postgres:
    image: postgres:16
    container_name: postgres
    hostname: postgres
    environment:
      POSTGRES_USER: metabase
      POSTGRES_DB: metabaseappdb
      POSTGRES_PASSWORD: mysecretpassword
    volumes:
      - ./pg_data:/var/lib/postgresql/data
    networks:
      - metanet1

  airflow-postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - airflow-postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 10s
      retries: 5
      start_period: 5s
    restart: always
    networks:
      - metanet1    

  redis:
    # Redis is limited to 7.2-bookworm due to licencing change
    # https://redis.io/blog/redis-adopts-dual-source-available-licensing/
    image: redis:7.2-bookworm
    expose:
      - 6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 30s
      retries: 50
      start_period: 30s
    restart: always
    networks:
      - metanet1    

  airflow-apiserver:
    <<: *airflow-common
    command: api-server
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/monitor/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-dag-processor:
    <<: *airflow-common
    command: dag-processor
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob --hostname "$${HOSTNAME}"']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    healthcheck:
      # yamllint disable rule:line-length
      test: ["CMD-SHELL", 'celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}" || celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    environment:
      <<: *airflow-common-env
      # Required to handle warm shutdown of the celery workers properly
      # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
      DUMB_INIT_SETSID: "0"
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-apiserver:
        condition: service_healthy
      airflow-init:
        condition: service_completed_successfully

  airflow-triggerer:
    <<: *airflow-common
    command: triggerer
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    # yamllint disable rule:line-length
    command:
      - -c
      - |
        if [[ -z "${AIRFLOW_UID}" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
          echo "If you are on Linux, you SHOULD follow the instructions below to set "
          echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
          echo "For other operating systems you can get rid of the warning with manually created .env file:"
          echo "    See: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#setting-the-right-airflow-user"
          echo
          export AIRFLOW_UID=$$(id -u)
        fi
        one_meg=1048576
        mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
        cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
        disk_available=$$(df / | tail -1 | awk '{print $$4}')
        warning_resources="false"
        if (( mem_available < 4000 )) ; then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
          echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
          echo
          warning_resources="true"
        fi
        if (( cpus_available < 2 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
          echo "At least 2 CPUs recommended. You have $${cpus_available}"
          echo
          warning_resources="true"
        fi
        if (( disk_available < one_meg * 10 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
          echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
          echo
          warning_resources="true"
        fi
        if [[ $${warning_resources} == "true" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
          echo "Please follow the instructions to increase amount of resources available:"
          echo "   https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#before-you-begin"
          echo
        fi
        echo
        echo "Creating missing opt dirs if missing:"
        echo
        mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Airflow version:"
        /entrypoint airflow version
        echo
        echo "Files in shared volumes:"
        echo
        ls -la /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Running airflow config list to create default config file if missing."
        echo
        /entrypoint airflow config list >/dev/null
        echo
        echo "Files in shared volumes:"
        echo
        ls -la /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID:-50000}:0"
        echo
        chown -R "${AIRFLOW_UID:-50000}:0" /opt/airflow/
        echo
        echo "Change ownership of files in shared volumes to ${AIRFLOW_UID:-50000}:0"
        echo
        chown -v -R "${AIRFLOW_UID:-50000}:0" /opt/airflow/{logs,dags,plugins,config}
        echo
        echo "Files in shared volumes:"
        echo
        ls -la /opt/airflow/{logs,dags,plugins,config}

    # yamllint enable rule:line-length
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_MIGRATE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
      _PIP_ADDITIONAL_REQUIREMENTS: ''
    user: "0:0"

  airflow-cli:
    <<: *airflow-common
    profiles:
      - debug
    environment:
      <<: *airflow-common-env
      CONNECTION_CHECK_MAX_COUNT: "0"
    # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
    command:
      - bash
      - -c
      - airflow
    depends_on:
      <<: *airflow-common-depends-on

  # You can enable flower by adding "--profile flower" option e.g. docker-compose --profile flower up
  # or by explicitly targeted on the command line e.g. docker-compose up flower.
  # See: https://docs.docker.com/compose/profiles/
  flower:
    <<: *airflow-common
    command: celery flower
    profiles:
      - flower
    ports:
      - "5555:5555"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

volumes:
  airflow-postgres-data:
  postgres-db-volume:  
  postgres-data:

networks:
  metanet1:
    driver: bridge
Code language: YAML (yaml)

How to add DBT project to Airflow docker images ?

I added Dockerfile2, where I install dbt Core and the dbt-postgres adapter on top of the Airflow image. I also define the path to the dbt profiles file.

FROM apache/airflow:3.3.0

USER airflow
RUN pip3 install --no-cache-dir \
    "apache-airflow==${AIRFLOW_VERSION}" \
    "dbt-core==1.10.15" \
    "dbt-postgres== 1.9.1"

ENV DBT_PROFILES_DIR=/opt/airflow/dbt/
Code language: Dockerfile (dockerfile)

How to execute the dbt core commands

I created three DAGs for running dbt commands. In each DAG, I used the BashOperator to navigate to the dbt project directory and execute the corresponding command.

dbt run

from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator

DBT_ROOT = "/opt/airflow/dbt"
DBT_PROJECT_DIR = f"{DBT_ROOT}/dbt_proj"
DBT_PROFILES_DIR = DBT_ROOT

with DAG(
    dag_id="dbt_run_only",
    start_date=datetime(2026, 7, 1),
    schedule=None,
    catchup=False,
    tags=["dbt"],
) as dag:

    dbt_run = BashOperator(
        task_id="dbt_run",
        bash_command=f"cd {DBT_PROJECT_DIR} && dbt run",
    )


Code language: Python (python)
Execution screenshoot
execution Logs
::group::Log message source details
/opt/airflow/logs/dag_id=dbt_run_only/run_id=manual__2026-08-20T13:19:32.592487+00:00/task_id=dbt_run/attempt=1.log
::endgroup::
[2026-08-20T13:19:41.718322Z] INFO - ::group::Pre Execute
Task Identity ti_id=01a01f53-aea3-7f8d-af15-c00bfd65714d dag_id=dbt_run_only task_id=dbt_run run_id=manual__2026-08-20T13:19:32.592487+00:00 try_number=1 map_index=-1
[2026-08-20T13:19:43.300981Z] INFO - DAG bundles loaded: dags-folder, example_dags, apache-airflow-providers-common-sql-example-dags, apache-airflow-providers-standard-example-dags
[2026-08-20T13:19:43.306382Z] INFO - Filling up the DagBag from /opt/airflow/dags/dag_dbt_run.py
[2026-08-20T13:19:43.718638Z] WARNING - The `airflow.operators.bash.BashOperator` attribute is deprecated. Please use `'airflow.providers.standard.operators.bash.BashOperator'`. category=UserWarning 
[2026-08-20T13:19:43.749983Z] INFO - Worker startup parse complete bundle_name=dags-folder  bundle_version=null  dag_file=dag_dbt_run.py  bundle_prepare_ms=29  dag_file_parse_ms=445 
[2026-08-20T13:19:44.668050Z] INFO - ::endgroup::
[2026-08-20T13:19:44.721121Z] INFO - Task instance is in running state
[2026-08-20T13:19:44.722591Z] INFO -  Previous state of the Task instance: TaskInstanceState.QUEUED
[2026-08-20T13:19:44.723997Z] INFO - Current task name:dbt_run
[2026-08-20T13:19:44.725293Z] INFO - Dag name:dbt_run_only
[2026-08-20T13:19:44.758579Z] INFO - Tmp dir root location: /tmp
[2026-08-20T13:19:44.794173Z] INFO - Running command: ['/usr/bin/bash', '-c', 'cd /opt/airflow/dbt/dbt_proj && dbt run']
[2026-08-20T13:19:44.800611Z] INFO - Output:
[2026-08-20T13:20:33.623383Z] INFO - 13:20:33  Running with dbt=1.10.15
[2026-08-20T13:20:39.150961Z] INFO - 13:20:39  Registered adapter: postgres=1.9.1
[2026-08-20T13:20:46.675589Z] INFO - 13:20:46  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
[2026-08-20T13:20:46.675866Z] INFO - There are 1 unused configuration paths:
[2026-08-20T13:20:46.676112Z] INFO - - models.dbt_proj.bronze
[2026-08-20T13:20:49.563501Z] INFO - 13:20:49  Found 13 models, 4 seeds, 1 analysis, 8 sources, 476 macros
[2026-08-20T13:20:49.592317Z] INFO - 13:20:49
[2026-08-20T13:20:49.605201Z] INFO - 13:20:49  Concurrency: 2 threads (target='dev')
[2026-08-20T13:20:49.606194Z] INFO - 13:20:49
[2026-08-20T13:20:54.517794Z] INFO - 13:20:54  2 of 13 START sql table model silver.services .................................. [RUN]
[2026-08-20T13:20:54.698080Z] INFO - 13:20:54  1 of 13 START sql table model silver.patients .................................. [RUN]
[2026-08-20T13:20:58.895483Z] INFO - 13:20:58  2 of 13 OK created sql table model silver.services ............................. [SELECT 4 in 4.24s]
[2026-08-20T13:20:58.895752Z] INFO - 13:20:58  3 of 13 START sql table model silver.services_weekly ........................... [RUN]
[2026-08-20T13:20:58.895902Z] INFO - 13:20:58  1 of 13 OK created sql table model silver.patients ............................. [SELECT 1000 in 4.10s]
[2026-08-20T13:20:58.896036Z] INFO - 13:20:58  4 of 13 START sql table model silver.staff ..................................... [RUN]
[2026-08-20T13:21:00.003146Z] INFO - 13:20:59  4 of 13 OK created sql table model silver.staff ................................ [SELECT 126 in 1.04s]
[2026-08-20T13:21:00.025220Z] INFO - 13:21:00  5 of 13 START sql table model silver.staff_schedule ............................ [RUN]
[2026-08-20T13:21:00.046937Z] INFO - 13:21:00  3 of 13 OK created sql table model silver.services_weekly ...................... [SELECT 208 in 1.15s]
[2026-08-20T13:21:00.059092Z] INFO - 13:21:00  6 of 13 START sql table model gold.dim_patients ................................ [RUN]
[2026-08-20T13:21:00.855652Z] INFO - 13:21:00  6 of 13 OK created sql table model gold.dim_patients ........................... [SELECT 1000 in 0.78s]
[2026-08-20T13:21:00.860461Z] INFO - 13:21:00  7 of 13 START sql table model gold.dim_services ................................ [RUN]
[2026-08-20T13:21:00.923404Z] INFO - 13:21:00  5 of 13 OK created sql table model silver.staff_schedule ....................... [SELECT 6552 in 0.89s]
[2026-08-20T13:21:01.008081Z] INFO - 13:21:01  8 of 13 START sql table model gold.patients .................................... [RUN]
[2026-08-20T13:21:01.536896Z] INFO - 13:21:01  7 of 13 OK created sql table model gold.dim_services ........................... [SELECT 4 in 0.67s]
[2026-08-20T13:21:01.541893Z] INFO - 13:21:01  9 of 13 START sql table model gold.dim_rol ..................................... [RUN]
[2026-08-20T13:21:02.176903Z] INFO - 13:21:02  8 of 13 OK created sql table model gold.patients ............................... [SELECT 4 in 1.14s]
[2026-08-20T13:21:02.181438Z] INFO - 13:21:02  10 of 13 START sql table model gold.dim_staff .................................. [RUN]
[2026-08-20T13:21:02.902849Z] INFO - 13:21:02  9 of 13 OK created sql table model gold.dim_rol ................................ [SELECT 3 in 1.34s]
[2026-08-20T13:21:03.059517Z] INFO - 13:21:03  11 of 13 START sql table model gold.staff ...................................... [RUN]
[2026-08-20T13:21:03.166109Z] INFO - 13:21:03  10 of 13 OK created sql table model gold.dim_staff ............................. [SELECT 126 in 0.98s]
[2026-08-20T13:21:03.260180Z] INFO - 13:21:03  12 of 13 START sql table model gold.services_weekly ............................ [RUN]
[2026-08-20T13:21:04.222647Z] INFO - 13:21:04  12 of 13 OK created sql table model gold.services_weekly ....................... [SELECT 48 in 0.96s]
[2026-08-20T13:21:04.247544Z] INFO - 13:21:04  13 of 13 START sql table model gold.staff_schedule ............................. [RUN]
[2026-08-20T13:21:04.512518Z] INFO - 13:21:04  11 of 13 OK created sql table model gold.staff ................................. [SELECT 12 in 1.46s]
[2026-08-20T13:21:04.825301Z] INFO - 13:21:04  13 of 13 OK created sql table model gold.staff_schedule ........................ [SELECT 126 in 0.57s]
[2026-08-20T13:21:05.197488Z] INFO - 13:21:05
[2026-08-20T13:21:05.219536Z] INFO - 13:21:05  Finished running 13 table models in 0 hours 0 minutes and 15.59 seconds (15.59s).
[2026-08-20T13:21:05.844460Z] INFO - 13:21:05
[2026-08-20T13:21:05.845862Z] INFO - 13:21:05  Completed successfully
[2026-08-20T13:21:05.847338Z] INFO - 13:21:05
[2026-08-20T13:21:05.848769Z] INFO - 13:21:05  Done. PASS=13 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=13
[2026-08-20T13:21:09.443181Z] INFO - Command exited with return code 0
[2026-08-20T13:21:09.449787Z] INFO - ::group::Post Execute
[2026-08-20T13:21:09.474129Z] INFO - Pushing xcom ti=RuntimeTaskInstance(id=UUID('01a01f53-aea3-7f8d-af15-c00bfd65714d'), task_id='dbt_run', dag_id='dbt_run_only', run_id='manual__2026-08-20T13:19:32.592487+00:00', try_number=1, dag_version_id=UUID('019fb426-6fef-786e-90ad-f49e50367df2'), map_index=-1, hostname='0f1ff81ff000', context_carrier={'traceparent': '00-e9f9d7b1a7d5901bb1294eacdda705af-e8e58f3738419533-00'}, queue='default', task=<Task(BashOperator): dbt_run>, bundle_instance=LocalDagBundle(name=dags-folder), max_tries=0, start_date=datetime.datetime(2026, 8, 20, 13, 19, 41, 940345, tzinfo=datetime.timezone.utc), end_date=None, state=<TaskInstanceState.RUNNING: 'running'>, is_mapped=False, rendered_map_index=None, sentry_integration='sentry_sdk.integrations.celery.CeleryIntegration') 
[2026-08-20T13:21:10.214122Z] INFO - ::endgroup::
[2026-08-20T13:21:10.230555Z] INFO - Task instance in success state
[2026-08-20T13:21:10.232671Z] INFO -  Previous state of the Task instance: TaskInstanceState.RUNNING
[2026-08-20T13:21:10.233135Z] INFO - Task operator:<Task(BashOperator): dbt_run>
Code language: PHP (php)

dbt clean

from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator

DBT_ROOT = "/opt/airflow/dbt"
DBT_PROJECT_DIR = f"{DBT_ROOT}/dbt_proj"
DBT_PROFILES_DIR = DBT_ROOT

with DAG(
    dag_id="dbt_clean",
    start_date=datetime(2026, 7, 1),
    schedule=None,
    catchup=False,
    tags=["dbt"],
) as dag:

    dbt_run = BashOperator(
        task_id="dbt_c",
        bash_command=f"cd {DBT_PROJECT_DIR} && dbt clean",
    )
Code language: Python (python)
Execution screenshoot
execution Logs
::group::Log message source details
/opt/airflow/logs/dag_id=dbt_clean/run_id=manual__2026-08-20T13:25:40.180370+00:00/task_id=dbt_c/attempt=1.log
::endgroup::
[2026-08-20T13:25:41.563631Z] INFO - ::group::Pre Execute
Task Identity ti_id=01a01f59-4940-7786-896a-56c0b0cc9a92 dag_id=dbt_clean task_id=dbt_c run_id=manual__2026-08-20T13:25:40.180370+00:00 try_number=1 map_index=-1
[2026-08-20T13:25:42.364088Z] INFO - DAG bundles loaded: dags-folder, example_dags, apache-airflow-providers-common-sql-example-dags, apache-airflow-providers-standard-example-dags
[2026-08-20T13:25:42.365630Z] INFO - Filling up the DagBag from /opt/airflow/dags/dag_dbt_clean.py
[2026-08-20T13:25:42.377905Z] WARNING - The `airflow.operators.bash.BashOperator` attribute is deprecated. Please use `'airflow.providers.standard.operators.bash.BashOperator'`. category=UserWarning 
[2026-08-20T13:25:42.378792Z] INFO - Worker startup parse complete bundle_name=dags-folder  bundle_version=null  dag_file=dag_dbt_clean.py  bundle_prepare_ms=20  dag_file_parse_ms=13 
[2026-08-20T13:25:42.847470Z] INFO - ::endgroup::
[2026-08-20T13:25:42.848922Z] INFO - Tmp dir root location: /tmp
[2026-08-20T13:25:42.849898Z] INFO - Running command: ['/usr/bin/bash', '-c', 'cd /opt/airflow/dbt/dbt_proj && dbt clean']
[2026-08-20T13:25:42.855552Z] INFO - Output:
[2026-08-20T13:25:42.865153Z] INFO - Task instance is in running state
[2026-08-20T13:25:42.865891Z] INFO -  Previous state of the Task instance: TaskInstanceState.QUEUED
[2026-08-20T13:25:42.866434Z] INFO - Current task name:dbt_c
[2026-08-20T13:25:42.866863Z] INFO - Dag name:dbt_clean
[2026-08-20T13:25:56.274444Z] INFO - 13:25:56  Running with dbt=1.10.15
[2026-08-20T13:25:57.607501Z] INFO - 13:25:57  Checking /opt/airflow/dbt/dbt_proj/target/*
[2026-08-20T13:25:57.676268Z] INFO - 13:25:57  Cleaned /opt/airflow/dbt/dbt_proj/target/*
[2026-08-20T13:25:57.680676Z] INFO - 13:25:57  Checking /opt/airflow/dbt/dbt_proj/dbt_packages/*
[2026-08-20T13:25:57.681651Z] INFO - 13:25:57  Cleaned /opt/airflow/dbt/dbt_proj/dbt_packages/*
[2026-08-20T13:25:57.681879Z] INFO - 13:25:57  Finished cleaning all paths.
[2026-08-20T13:25:59.644336Z] INFO - Command exited with return code 0
[2026-08-20T13:25:59.645256Z] INFO - ::group::Post Execute
[2026-08-20T13:25:59.645861Z] INFO - Pushing xcom ti=RuntimeTaskInstance(id=UUID('01a01f59-4940-7786-896a-56c0b0cc9a92'), task_id='dbt_c', dag_id='dbt_clean', run_id='manual__2026-08-20T13:25:40.180370+00:00', try_number=1, dag_version_id=UUID('019fb427-a737-7243-b1a5-34b0880c90c2'), map_index=-1, hostname='0f1ff81ff000', context_carrier={'traceparent': '00-77ff37de4fab71c51cd5e69f93e75916-654f5cca8d15605c-00'}, queue='default', task=<Task(BashOperator): dbt_c>, bundle_instance=LocalDagBundle(name=dags-folder), max_tries=0, start_date=datetime.datetime(2026, 8, 20, 13, 25, 42, 261101, tzinfo=datetime.timezone.utc), end_date=None, state=<TaskInstanceState.RUNNING: 'running'>, is_mapped=False, rendered_map_index=None, sentry_integration='sentry_sdk.integrations.celery.CeleryIntegration') 
[2026-08-20T13:25:59.957146Z] INFO - ::endgroup::
[2026-08-20T13:25:59.968432Z] INFO - Task instance in success state
[2026-08-20T13:25:59.969501Z] INFO -  Previous state of the Task instance: TaskInstanceState.RUNNING
[2026-08-20T13:25:59.974140Z] INFO - Task operator:<Task(BashOperator): dbt_c>
Code language: PHP (php)

dbt compile

from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator

DBT_ROOT = "/opt/airflow/dbt"
DBT_PROJECT_DIR = f"{DBT_ROOT}/dbt_proj"
DBT_PROFILES_DIR = DBT_ROOT

with DAG(
    dag_id="dbt_compile",
    start_date=datetime(2026, 7, 1),
    schedule=None,
    catchup=False,
    tags=["dbt"],
) as dag:

    dbt_run = BashOperator(
        task_id="dbt_com",
        bash_command=f"cd {DBT_PROJECT_DIR} && dbt compile",
    )

Code language: Python (python)
Execution screenshoot
execution Logs
::group::Log message source details
/opt/airflow/logs/dag_id=dbt_compile/run_id=manual__2026-08-20T13:29:01.916499+00:00/task_id=dbt_com/attempt=1.log
::endgroup::
[2026-08-20T13:29:03.994629Z] INFO - ::group::Pre Execute
Task Identity ti_id=01a01f5c-5d15-79cd-a5e2-e1bf6558d78a dag_id=dbt_compile task_id=dbt_com run_id=manual__2026-08-20T13:29:01.916499+00:00 try_number=1 map_index=-1
[2026-08-20T13:29:04.227646Z] INFO - DAG bundles loaded: dags-folder, example_dags, apache-airflow-providers-common-sql-example-dags, apache-airflow-providers-standard-example-dags
[2026-08-20T13:29:04.370546Z] INFO - Filling up the DagBag from /opt/airflow/dags/dag_dbt_compile.py
[2026-08-20T13:29:04.564613Z] WARNING - The `airflow.operators.bash.BashOperator` attribute is deprecated. Please use `'airflow.providers.standard.operators.bash.BashOperator'`. category=UserWarning 
[2026-08-20T13:29:04.566260Z] INFO - Worker startup parse complete bundle_name=dags-folder  bundle_version=null  dag_file=dag_dbt_compile.py  bundle_prepare_ms=196  dag_file_parse_ms=207 
[2026-08-20T13:29:04.920835Z] INFO - ::endgroup::
[2026-08-20T13:29:04.921855Z] INFO - Tmp dir root location: /tmp
[2026-08-20T13:29:04.929030Z] INFO - Task instance is in running state
[2026-08-20T13:29:04.933185Z] INFO - Running command: ['/usr/bin/bash', '-c', 'cd /opt/airflow/dbt/dbt_proj && dbt compile']
[2026-08-20T13:29:04.934115Z] INFO - Output:
[2026-08-20T13:29:04.955340Z] INFO -  Previous state of the Task instance: TaskInstanceState.QUEUED
[2026-08-20T13:29:04.955583Z] INFO - Current task name:dbt_com
[2026-08-20T13:29:04.955703Z] INFO - Dag name:dbt_compile
[2026-08-20T13:29:23.783417Z] INFO - 13:29:23  Running with dbt=1.10.15
[2026-08-20T13:29:26.506918Z] INFO - 13:29:26  Registered adapter: postgres=1.9.1
[2026-08-20T13:29:27.058902Z] INFO - 13:29:27  Unable to do partial parsing because saved manifest not found. Starting full parse.
[2026-08-20T13:29:41.051408Z] INFO - 13:29:41  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
[2026-08-20T13:29:41.051758Z] INFO - There are 1 unused configuration paths:
[2026-08-20T13:29:41.052031Z] INFO - - models.dbt_proj.bronze
[2026-08-20T13:29:41.816813Z] INFO - 13:29:41  Found 13 models, 1 analysis, 4 seeds, 8 sources, 476 macros
[2026-08-20T13:29:41.831353Z] INFO - 13:29:41
[2026-08-20T13:29:41.841440Z] INFO - 13:29:41  Concurrency: 2 threads (target='dev')
[2026-08-20T13:29:41.844039Z] INFO - 13:29:41
[2026-08-20T13:29:50.621403Z] INFO - Command exited with return code 0
[2026-08-20T13:29:50.622572Z] INFO - ::group::Post Execute
[2026-08-20T13:29:50.623158Z] INFO - Pushing xcom ti=RuntimeTaskInstance(id=UUID('01a01f5c-5d15-79cd-a5e2-e1bf6558d78a'), task_id='dbt_com', dag_id='dbt_compile', run_id='manual__2026-08-20T13:29:01.916499+00:00', try_number=1, dag_version_id=UUID('01a01be2-6945-7491-b756-be65453befe6'), map_index=-1, hostname='0f1ff81ff000', context_carrier={'traceparent': '00-a97f9dde995057e6caea37c232e80957-39e79b669c59979b-00'}, queue='default', task=<Task(BashOperator): dbt_com>, bundle_instance=LocalDagBundle(name=dags-folder), max_tries=0, start_date=datetime.datetime(2026, 8, 20, 13, 29, 3, 818724, tzinfo=datetime.timezone.utc), end_date=None, state=<TaskInstanceState.RUNNING: 'running'>, is_mapped=False, rendered_map_index=None, sentry_integration='sentry_sdk.integrations.celery.CeleryIntegration') 
[2026-08-20T13:29:51.095283Z] INFO - ::endgroup::
[2026-08-20T13:29:51.118714Z] INFO - Task instance in success state
[2026-08-20T13:29:51.119100Z] INFO -  Previous state of the Task instance: TaskInstanceState.RUNNING
[2026-08-20T13:29:51.152396Z] INFO - Task operator:<Task(BashOperator): dbt_com>
Code language: PHP (php)

Posted

in

,

by

julioVM

Tags:

,

Comments

Leave a Reply