dbt Labs Cranks the Performance Dial with New Fusion Engine

(JavierLizarazo/Shutterstock)

Dbt became one of the most used data tools thanks to its extensibility and simplicity. Now the folks behind dbt are taking the tool to the next level with the launch of a new Rust-powered Fusion engine, which dbt Labs CEO Tristan Handy says will bring a 30x performance boost over dbt Core, among other enhancements. The company also launched a trio of new and updated tools to enhance data analyst productivity.

The new dbt Fusion engine originates with dbt Labs’ acquisition of SDF Labs at the beginning of the year. SDF Labs described their technology “a multi-dialect SQL compiler, transformation framework, and analytical database engine packaged into a single CLI.” It turns out that it’s much more than that.

For starters, dbt Fusion is fast. According to Handy, the dbt Fusion engine that’s based on SDF Labs’ technology is, out of the box, 30 times faster at parsing dbt code than the aging Python engine at the heart of the free dbt Core product and all of the other commercial products from dbt Labs based on dbt Core.

But 30x is just an estimate, Handy says. “That’s like code that we’re sprinting to get out the door and is not fully optimized,” he tells BigDATAwire. “I think that that number will actually go up.”

All that underlying speed will drive a better user experience for the analytics engineers that dbt Labs targets with its products. That better UX–delivered through a new plug-in for VS Code that dbt Labs also announced–will be critical, as dbt Labs brings more conversational AI capabilities to the product.

dbt Fusion also understands a wide variety of SQL dialects, Handy says. So regardless of what dialect you’re writing in–whether it’s Snowflake SQL, Databricks SQL, Google Cloud BigQuery SQL, or Amazon Redshift SQL–dbt Fusion will be able to instantly spot errors in it.

The new dbt Fusion engine is 30x faster at parsing dbt code than the dbt Core product, according to dbt Labs

“It will show you a red squiggly line. You mouse over it, it shows you an error message. It tells you exactly what’s wrong and how to fix it,” Handy says. “This is the kind of developer experience that developers are used to in all modern programming languages, but SQL has never had.”

Future Fusion Benefits

Over the next six weeks, dbt Labs will support all four of the previously mentioned SQL data warehouses with dbt Fusion, with more coming in the future. Dbt Fusion should work pretty much out-of-the-box with existing code written with dbt Core, Handy says. Plus, all of the products that dbt Labs has built atop its core dbt Core product will get better as a result of Fusion.

“Every one of them is going to get faster. It means that our editor is going to be much more interactive, better error messages, etc. “There’s a lot of neat stuff, though, that we have never been able to build in the past that we will now be able to build.”

For instance, thanks to Fusion, dbt users in the future will be able to run an emulated version of their data warehouse directly on their laptop. That will allow them to develop better transformations on a subset of their data while saving them money, Handy says.

“What Fusion can do is fully emulate the underlying data platform and allow you to actually have a coherent local development environment where you’re not actually talking to the underlying platform at the time. You have a reduced copy of the data that you can test everything locally,” Handy says. “That’s going to be a big boon to the developer experience. And it also is going to save a lot of money.”

According to dbt Labs estimates, that feature alone will save customers 10% on the cost of their data warehouse.

New dbt Tools

dbt Labs also launched a series of new and updated tools for dbt users, dbt Insights, an AI-powered query tool for quick analysis and sharing; including dbt Canvas, a lightweight, drag-and-drop tool used for data model and data pipeline development; and a new release of dbt Catalog, its data asset discovery tool.

While dbt is primarily about data pipelines and the metadata associated with data pipelines, sometimes a user has to explore their data a little bit to build those pipelines, Handy says. That’s what drove the development of dbt Insights.

dbt Labs CEO Tristan Handy is a 2022 Datanami Person to Watch

“We wanted people to be able to explore their data inside of a dbt interface,” he says. “Snowflake and Databricks both have kind of lightweight, exploratory experiences that users love, and so now we’ll have our own.”

What’s interesting about dbt Canvas, Handy says, is that it builds upon the graphical pipeline building user interfaces that developers have been using for decades by overlaying a conversational AI interface on top of it.

“It turns out that we are able to single shot that experience quite effectively because of all of the context that dbt already has about your data,” Handy says. “So it’s kind of magical. You show up in Canvas and there’s a blank screen, and then you just enter a prompt and say ‘Here’s what I want.’ You hit enter and then bam! You’re just greeted with a visual pipeline that you can then tweak on your own.”

These two new tools and upgraded data catalog are aimed at bringing more users into the dbt experience, which features native DevOps integration for data management that other analytics and development tools lack, Handy says.

“Those are all things that will make the dbt workflow accessible to data analysts who have traditionally been working in working on these other tools,” he says.  “The central data teams that love dbt so much can actually go to them and say ‘Hey, look, I’ve got better tooling for you. Why don’t you join the party?’”

End of the Line for dbt Core

The launch of dbt Fusion marks a high performance new beginning for dbt, but it also spells the beginning of the end for dbt Core. The old product will continue to be supported, but it won’t be seeing as many new features and enhancements, as dbt Fusion is the new strategic future for dbt Labs.

Handy was retrospective about the change.

“We originally built dbt Core back in 2016. Since then we’ve added a bunch of features to it. But the core technology is the same. It’s Python-based. It has some consistent design principles that it continues to apply to this day. And it turns out that all technology has a lifecycle,” he says.

“We’ve pushed core forwards for nine years, and the technology in there does not have a lot further to go,” he says. “There’s a bunch of stuff that we wanted to bring to the dbt community that we just couldn’t with this code base.

Internally, dbt Labs had started to work on figuring out the path forwards. As they were researching possible paths, SDF Labs came out of stealth with a data transformation engine that already supported dbt. That got Handy’s attention.

“They essentially built the engine that we would have wanted to build. It was already targeting the dbt community. It already knew how to speak dbt language. And so this is kind of how the acquisition happened,” he says. “Over the last five and a half months, we have been sprinting like hell to get this thing actually integrated into dbt technology.”

From this point forward, dbt Fusion will be the engine that powers both dbt Labs’ open as well as the private commercial products. Dbt Fusion, like dbt Core, will have a source-available license, which means its free for anybody to use as they like, but dbt Labs retains full control over the development.

Handy is particularly impressed with the work of SDF Labs CTO and Co-founder Wolfram Schulte, who is the brother of the company’s CEO and Co-Founder, Lukas Schulte. Wolfram Schulte was a longtime principal software engineer at Microsoft Research. After the Cambridge Analytica scandale, Wolfram was hired by Meta to programmatically figure out where all the personally identifiable information (PII) was traveling throughout Facebook data pipelines.

“When you’re literally creating over a million tables, it’s very hard to keep track of where PII is going to. It’s too many to go and hand classify everything,” Handy says. “And so what Wolfram did was build a multi-dialect parser that could understand all of those pipelines. They then took that technology and it formed the basis of what has become Fusion.”

Related Items:

dbt Labs Report Reveals How AI Is Boosting Data Budgets and Team Growth

Qlik and dbt Labs Make Big Data Integration Acquisitions

Still Too Much Duct Tape in Data Transformation, dbt Labs’ Handy Says