dbt State Aware Orchestration

In many data organizations today, data pipelines are not as efficient as they could be. Whenever a pipeline runs, everything is rebuilt. Even if there is no new data in the sources and no changes have been made to the models, the pipeline runs anyway. This leads to a major waste of both time and money.

Wouldn’t it be better to rebuild only what is necessary? Ideally, we should only process the parts of the DAG where something has actually changed and simply reuse the models that are already up-to-date.

This is exactly what dbt state-aware orchestration (SAO) powered by new Fusion engine does.

State Aware Orchestation

State-aware orchestration is a dbt Fusion-powered innovation that fundamentally rewrites how data pipelines work by only running what actually needs to run.

dbt now understands the difference between your current project state and the previous one. It identifies changes and only runs what is actually affected by them.

With state-aware orchestration, dbt automatically detects changes in your code or data. This allows it to decide exactly which models need to be rebuilt every time a job runs, leading to:

• Savings in compute costs.
• Reduced overall pipeline runtime.

It was one of the biggest announcements from dbt Coalesce 2025!

SAO example

Let’s look at a simple dbt DAG that consists of 4 sources and 9 models.

Imagine that only the orders_training source has new data. All other sources remain unchanged, and no code has been modified.

With State-Aware Orchestration: dbt detects that only one source is fresh. It identifies the specific downstream path and runs only the 3 relevant models. The other 6 models are skipped and reused as they are.

The Traditional Way: Your orchestrator would rebuild all 9 models, even though most of them would produce the exact same results as the last run.

Let’s take a look at how the DAG looks within the dbt Cloud environment.

dbt cloud logs: all models

3 models affected by a change will run, providing a clear log message explaining why.

6 unaffected models are reused and accompanied by a clear log message.

Prerequisites

• You need a dbt Enterprise or Enterprise+ account.
• The environment where you run state-aware orchestration must be updated to use the dbt Fusion engine.
• You must have a deployment environment set up. Currently, state-aware orchestration is only supported for Production or Staging environments. It does not work with ‘General’ or ‘Development’ environment types.

How does dbt detect Code Changes?

To detect code changes, dbt uses a file called the manifest.json. Think of this file as a detailed “DNA map” of your entire project. It contains a unique fingerprint (a hash) for every model, macro, and test.

When a state-aware job runs, dbt performs a State Comparison:

It compares the fingerprints for every model. If the fingerprints do not match, dbt knows the code has changed and marks that model to be rebuilt.

It looks at the manifest.json from your last successful production run.

It generates a new manifest.json for your current code.

How does dbt detect Data Changes?

By default, dbt uses the warehouse metadata to check if data sources or upstream models are fresh.

dbt detects data changes by looking at your warehouse metadata ( eg.: INFORMATION_SCHEMA). It checks when your source tables were last modified and compares that to the time of your last successful dbt run. If the data in the warehouse is newer than your last run, dbt knows it needs to rebuild the model. If not, it simply reuses the model to save time and money.

Customization

You can definitely leverage state aware orchestration and fine tune it per your needs with additional set up using these parameters:

Source level/Upstream models (Mesh)

Parameters you can use on source level:
loaded_at_field: By default, dbt identifies changes using the models’ metadata stored in your warehouse. However, if you don’t want to rely on that, and you have a timestamp column in your sources that indicates what data is fresh and what isn’t, you can configure dbt to use that field instead.

source YAML

When you define a loaded_at_field, you are changing the “source of truth” for freshness. Instead of dbt asking the data warehouse metadata for the “last modified” time of the table file, dbt runs a query to find the maximum value in your specific column: UPDATED_ON.

Models

Parameters on model level that you can use are:build_after and updates_on.

We have the same DAG with 9 models, but let’s focus on the sales model, the very last step in our pipeline. The business only needs to see updated sales data once every 24 hours, but our dbt job runs every few hours to keep other models fresh.

Without any special settings, the sales model would rebuild every time new data arrives even if it is not necessary. To prevent this, I adjusted the YAML file for the sales model and set the build_after parameter to 24 hours.

build_after

What happens in practice? Imagine the last run of our pipeline was only 3 hours ago. Now, the job runs again and there is new data in some of our sources (orders). dbt checks the sales model and sees that it was already refreshed 3 hours ago.

Because it has not been 24 hours yet, dbt will not trigger a rebuild. Instead, the sales model is simply reused. You can see this clearly in the logs with the message:

“did not meet build_after of 1 day.”

Lineage

LogsLogs

In addition to timing, you can also control the logic of the rebuild using the updates_on parameter.

Imagine I want the sales model to refresh only if all upstream models have new data, not just one, but every single one of them. By setting updates_on: all, I am telling dbt to be more strict.

What happens in practice? In our example, only the orders source was updated, while the other sources stayed the same. Because the “all” condition was not met, the sales model was not rebuilt and was simply reused. You can see this clearly in the logs, which will state that the condition did not pass.

Alternatively, you can set this parameter to any. In this case, the sales model will rebuild if even one of its sources has new data. This is the default behavior for dbt, but being able to switch to all gives you the power to prevent unnecessary runs when your final models require a complete data refresh to be useful.

Lineage

Logs

Note: In the YAML example, you might notice that it was set the build_after parameter to only 4 minutes. This was a temporary setup used specifically to demonstrate how the updates_on parameter works for this article. In a real production environment, you would typically set this to a much longer duration (like 12 or 24 hours), but for testing purposes, 4 minutes allowed me to verify the results without waiting all day!

How it works step-by-step:

• The Check: dbt executes a query: SELECT MAX(UPDATED_ON) FROM our_source_tables.
• The Comparison: dbt compares that maximum timestamp to the completion time of your last successful production job.
• The Decision: * If MAX(UPDATED_ON) > Last Run Time, dbt marks the source as Fresh. And then, downstream models are triggered.
• Instead of a field, you can write your own sql condition with this parameter: loaded_at_query.
• You can either define loaded_at_field or loaded_at_query but not both.

Smart Testing

Beyond optimizing models, the dbt Fusion engine also makes your testing suite much more intelligent.

In traditional dbt, every individual test often runs as its own separate SQL query. If you have 10 columns with “not null” tests, that translates to 10 different queries to your warehouse. With Fusion, dbt can “aggregate” (combine) multiple tests into a single query, reducing testing costs significantly.

Moreover, Fusion performs a deep analysis of your column-level lineage. It can recognize when a column is simply being “passed through” from one model to another without any transformations. For example, if you tested a column in your staging model and it hasn’t changed at all in your downstream model, Fusion knows that testing it a second time is redundant. dbt skips the downstream test because it already knows the data is valid from the upstream check.

Business Impact

One of the most powerful aspects of the dbt Fusion engine is how it simplifies your daily operations. By understanding the state of your data sources, dbt removes the stress of perfectly timing your pipeline runs. You can schedule your jobs as often as you like, confident that the system will only process what has actually changed.

This intelligence leads to massive cost reductions. Simply turning on state-aware orchestration provides an average 10% reduction in compute costs. By leveraging advanced configurations like updates_on and build_after etc., teams can save an additional 15%.

Finally, Fusion optimizes your testing by combining multiple tests into single queries and skipping redundant checks on columns that haven’t changed. When you combine smarter model runs with these testing optimizations, the total impact can reach up to a 29% reduction in costs (estimation).

Estimated cost reductions with Fusion and SAO, dbt Labs

dbt Labs themselves were able to reach out to 64 % cost reduction as they were running around 1500 at different schedules.

dbt Labs’ results

Conclusion

The great news is that state-aware orchestration is turned on automatically for all new jobs. This allows you to see the benefits of this feature right away. However, this only happens if you use the Fusion engine and have an Enterprise license.

If you are already using dbt Cloud and you switch to Fusion, your old jobs will not change. They will not have state-aware orchestration turned on automatically. You must go into the settings and turn it on yourself for those jobs.

If you have a specific job where you do not want to use this feature, you can easily turn it off in the settings.

State-aware orchestration enabled

• State-aware orchestration is currently available for SQL models only.
• Python models are not currently supported.

Moreover, jobs are now aware of each other. In the past, if two different jobs tried to update the same table at the same time, it could cause errors or “concurrent write” conflicts.

State-aware orchestration prevents this by recognizing when a table is already being updated, eliminating a significant pain point for many data teams. This ensures your data pipelines are not just faster and cheaper, but also much more stable.

Katerina Protivenskyi

Katerina is a Senior Data Engineer at Revolt BI with a strong focus on modern data platforms and data architecture.

She has experience designing and implementing scalable data models, building ELT pipelines in dbt, and orchestrating workflows with tools like Apache Airflow. She specializes in data transformation and integration, ensuring data quality and consistency, and automating data processes. Katerina helps companies work with data effectively, set up robust data infrastructure, and prepare their data for advanced analytics and AI.

Want to see how dbt works?

Or would you like to train your internal team with it? Get in touch with us!