Log In to continue learning
Data Engineering
Intermediate
Topics
Data Engineering
Data Processing
Deep Learning
Search and Retrieval
How's it going, Drew? Hey, Joe. Hey, good. So, yeah, today we have Drew Banin, co-founder of dbtLabs. But for people who don't know who you are, do you want to give a quick intro? Yeah, sure thing. My name's Drew. I'm one of the co-founders at dbtLabs, as you say. I'm based out of Philadelphia, PA, and we've been working on dbt for about eight years now. We've been doing this. So, first question. I'm sure the learner might be wondering, what is dbt? Great question. I do an onboarding session for all of our new employees, and we call it, What is dbt? And the funny thing about this session is that it's an hour long, and I spend the first 50 minutes talking about everything but dbt. And so I like to set the stage for, you know, a modern data platform. And so that involves a data warehouse or something like it. BI tools connected to your data platform, so data warehouse, data ingestion, and kind of getting all your data in one place. Then only once we've introduced all of this can we talk about dbt. So I'll short-circuit that, like, 60-minute talk and try to do it, like, one minute, if that's okay. I would say that dbt is the thing that applies business logic rules to your data to convert that data into information. So if we think about data, it's numbers, it's strings, it's dates. But are the time stamps in UTC or Pacific time? And are the numbers that represent revenue in U.S. dollars or Canadian dollars or euros? And so we think about business logic as, right, these are rules. So with dbt, you can codify those rules in SQL and Python. You can version control that logic and adopt a software engineering-based workflow for managing changes to the logic over time. And then dbt will help you apply that logic to your data inside of your data platform. So we don't take your data out of, you know, your data warehouse, just say, and transform it and put it back in. What we do is we instruct the data warehouse to transform your data in place. And so folks like this because it's kind of better secured and governed with the data not leaving, you know, your data platform. So that's the 30,000-foot view. There's so much we could talk about beyond that, but at a high level, it's about business logic. Sure. And for the learner, maybe give some perspective on what life was like before dbt came on the scene. What was the workflow like for an analyst or a data engineer or an analytics engineer? Oh, yeah. So, you know, the word analytic or the job description or title, depending on how you think about it, analytics engineer kind of came out of dbt and the paradigm that we promote with dbt. So pre-dbt, there were kind of two things that happened. And the first one was the Wild West. So this was people with random SQL scripts stored on their hard drive. In fact, one place I used to work when our one data analyst left the company, he sent a zip file of 50 or something like that, SQL scripts to his boss, who was the CFO. And so that's like not a good offboarding plan, right? Like that is kind of chaotic. So pre-dbt, people would just kind of do this stuff ad hoc. You'd create tables. You've run random insert and delete statements to fix data. And there wasn't really like an audit trail of all the transformations applied to your data to apply them. When it just happened live in the warehouse. Post-dbt, what we see happening is that people like, you know, better control their code. And we sort of made a community of practice or at least help facilitate a community of practice around this work. And so these are folks who call themselves analytics engineers. And, you know, we affectionately call them purple people where we think of, you know, the red people are the engineers and the blue people are the business folks. And as a good analytics engineer, you're right in the middle. So you understand the business context. You understand the technology. And you can help these, you know, two groups come together and create the right data that helps you solve the right problems at the right time. Yeah, that's something I think that the learners take away is that the world. Before something like dbt, I've worked in that world. It was very cumbersome. You know, you'd version control SQL scripts, but somebody changes. I mean, I'll change the SQL script. And pretty soon you have. I don't even know how many SQL scripts out there and just got the big mess. So thanks for doing that. That's awesome. So in terms of how learners should think about approaching dbt, you know, if they're new to it, what are some recommendations that you have? Well, you know, I'll tell you, we've always sort of thought about. dbt as being analogous to something like Ruby on Rails, if you're if anyone's familiar. And so the idea is it is very much a framework for applying this business logic. There's a lot of I'm going to say convention in the same way that there's convention in Ruby on Rails, maybe a little less than Ruby on Rails. But it is if you like follow the dbt way, then easy things will be easy and hard things will be possible. That's kind of how we think about it. So I'd say if you find that you're really fighting against dbt at the outset, it might be like a signal that you should check up on our documentation or, of course, the course and make sure that you're kind of using the right mental model to approach dbt. Now, inevitably, what happens is you need to break outside the confines of the framework because data is complicated and every business is different, different. So we do have a lot of escape hatches where you can inject your own logic and do custom stuff along the way. But I would say the starting point, it should be really easy to get started with dbt. You make one SQL file, you take dbt build, you're off and running. And then from there, you can start to think about testing your data and writing documentation and eventually the semantic layer and things like that kind of built on top of that core modeling capability. Or just start with models and maybe test from there and go from there. What are some best practices you recommend in terms of building models and maintaining models in dbt? Sure. I mean, if folks are doing this kind of at scale in an organization, we always recommend using a SQL style guide or something like it. So the goal there is to make sure that everyone's code, regardless of team or who the individual is, everyone's code looks pretty similar. Having conventions for how you name columns and how you split apart logic into sort of staging tables and then maybe intermediate transformations and then more like marts or dimensional models that can be really handy for keeping things organized at scale. Beyond that, I would say, you know, a lot of software engineering best practices just apply in general. So keep your code relatively modular. Don't try to do too much stuff in one file, you know, because we kind of think of a SQL model is like a function. Don't try to do too much stuff in one function or file. Make sure you test your code as you go. And especially, you know, we just really support for unit testing. So there's actually new types of ways that you can write tests in dbt. But yeah, modularity, you know, get in the habit of making pull requests and kind of reviewing your code before it goes out to prod. Set up, you know, CI/CD or something like that. So deploys are automated. And then there are also things that you can kind of roll your own way if you need to. Interesting. This brings up an interesting question. I'm sure the learners will have as well. But you keep mentioning software engineering practices. I was a software engineer. So a lot of things you mentioned is familiar testing and deployment and so forth. How much do you think a learner should actually go and maybe study some of the principles of software engineering? Gosh, it's always a funny question because I studied computer science in college and I wouldn't recommend it. I'm just kidding. Yeah, listen, I mean, some of these fundamentals, they're kind of invariant over time and trends. And so like SQL is a decades old technology. Of course, it's evolved, but the fundamentals are still about the same in software. You know, Git is relatively new as technologies go. I think, well, gosh, I'm going to feel bad if I get this way wrong. But isn't it like less than 20 years old? I think it is. I think, yeah, the last Torvalds rolled it out randomly one day, I think. Yeah, great. But like there was version control before Git and the same kinds of rules applied. Like keep your changes pretty small, make them testable, like write your code in a way that you can test it. To be honest with you, this actually isn't something that they teach when you study computer science in college, at least they didn't where I did. Sometimes they put it in kind of a different set of classes and they call it software engineering. And it's a lot more about the testing and the methodology and the scaling. So it's not something I have a tremendous amount of formal experience with myself. But I do know that if you've written code in any capacity before, you can kind of understand when things are like spaghetti in a big mess or sort of well architected. And really the punchline is think hard about what you're trying to do up front and make sure you're solving the right problems. I think people get in trouble when they just start writing code. And then, as they say, 10 hours of writing code can save you one hour of planning. Keep that in mind. That's such a good quote. Is there anything else that you would like learners to know about dbt? Yeah, I mean, we're so big on community. A lot of a lot of the energy behind dbt, you know, comes from the community. And so we've got meetups all across the globe. We've got places like our community slack. We have a conference in October every year that we call coalesce community conference. So I would say, you know, beyond the course itself, if you're looking for ways to get connected to other people doing this work, definitely check out the dbt community and and feel free to join. We'd love to have you. Awesome. Well, thanks for your time, Drew. It's been great talking to you about dbt and hopefully the learners learn something new about dbt. Yeah, thank you. It's my pleasure.
