Log In to continue learning
Data Engineering
Intermediate
Topics
Data Engineering
Data Processing
Deep Learning
Search and Retrieval
I'm here today with Ben Rogojan, also known as Seattle Data Guy. So it's good to see you, Ben. How are you doing? Hey. Good to see you again, Joe. I'm doing great. Doing great. Awesome. Yeah. So really excited to talk to you about data engineering, but for the learners who don't know who you are, do you want to give a quick intro? Yeah. So since we're talking about data engineering, I've been working, I think, about almost a decade at this point in the data world. Originally, I think like most people who came around in the 2012, 2015 kind of era, I was like, data science is what I'm going to do. I switched a lot of my courses, even in college, to being a lot more bioinformatics or statistics based, just for that reason. Then I came out, started doing some data science work, and I think I just eventually realized I like more of the software, the programming side of things. And I accidentally found data engineering, honestly, when I was looking for a new job. I was just typing in the skills that I had, SQL, Python, automation, data warehousing, different things, and got my first official role as a data engineer at a healthcare analytics startup. And then from there, just kind of kept growing my skillset. I eventually got a job at Facebook after a few years at the startup, and then worked there for a few more years before, at this point now, I just do kind of consulting in the data engineering, data infrastructure space. And you make awesome content about data engineering. Yes, yes, I try. I try. It's always challenging to keep up with all the stuff that changes, so you're always like, oh, man. It's so hard, isn't it? It's so hard. Yeah, I have an intro to Databricks video that's done really well, and now I'm like, I need to redo it all at this point. So much has changed, so yeah. That's the nature of it, though. Data engineering, I would say that the one constant is change, oxymoronically, yeah. So walk me through your journey. So you started out as a data scientist, got into data engineering. What were your early days as a data engineer like? Like at the company that I work for, we did, again, healthcare kind of data analytics. We were basically just taking like 40 or 50 different insurance provider datasets that we would get access to for various reasons, sometimes because the company we were kind of owned by, they were just a giant insurance company. And then also, they were trying to broaden their services and offer it to like Amazon and Boeing and be like, hey, we'll do like healthcare analytics for your company, because most companies are like half or some level self-insured. So if something goes wrong, they actually end up paying a lot of it, not just the insurance company, because it would just be too expensive otherwise. And so if they can increase the health of their population, the view is that you reduce the risk and the burden there. But yeah, so we would just take all that data, standardize it, right, because the nice thing here is that claims data, which is what we were working on, it's like healthcare claims data, is decently standardized, minus the fact that all the data you get comes in different formats, but like the data that it has inside should be about the same. So that's when I learned about like all the various ways data can be separated, like position delimited fields or position space fields are always weird. I'm sure you've come across it occasionally where it's like, yeah, field zero or row or column zero to whatever, 50 is this field, and you have to like have a schema file just to keep track of all that. But yeah, like really just building a system around that, and then also building products on top of that. Because while I was there, we built like a product on fraud detection, we built something around opioids and tracking kind of that, and it was maybe provide too much in terms of opioids and trying to help that and give that to different things. So that's really where it kind of started. And we worked honestly just on PowerShell and SQL Server and like things that most people would be like, that's not sexy at all. But that's honestly where it started. Then you moved over to Facebook, which I'm guessing is a bit different than PowerShell and SQL Server. I mean, it is, but like at the same time, the problems like are different. One, Facebook's data infrastructure is very mature. So a lot of people sometimes find it boring because, you know, if you're wanting to work on like, quote unquote, a big data problem, there are some teams that will work on those big data problems and do have big data issues, but it's not everyone. And it's probably a very small percentage. So a lot of your infrastructure is well set up. I remember like just pushing code. I was like, oh, I'm just like building a Python script that really wraps around SQL. And then I'm just pushing it somewhere. And it's really more about like how I communicate and how I find like problems that are worth solving more than like my coding ability. Some people found it again. I think I don't know if I already said this, I found it boring because it was like, well, all the hard stuff is kind of solved, you know. That's an interesting comment about communication and finding problems to solve. So a big part of this course is identifying things that maybe the business is going to find valuable for you to work on, right? And so I think the temptation for data engineers is to figure out, you know, very complex systems to build and so forth. But oftentimes it's at odds with solving legitimate problems. Walk me through that framework, though, because I don't think this is something that's often taught. How did you figure out that you needed to not just be maybe awesome at coding, but also great at communicating and finding problems to solve? Yeah. I mean, in terms of how or why, I think one of the things that Facebook forces you to do is like, hey, there's like a thousand other data engineers. And if I just do what someone else tells me to do, it's very hard for me to differentiate, right? If I don't take some risk or do something that's like, you know, I think this is worth doing, everyone else will just define what I should be doing and I might not do something that actually impacts the business. And so usually what we do is spend a good part of maybe the first few, well, maybe before the first few months of a half is kind of how we broke down the year. So first part of like maybe in December or November, we'd like talk to all the teams that we communicate with and work with and be like, hey, what problems are you trying to deal with? And not just data, like what are your daily problems and what's going on? We would talk about what problems we solved the last few months and be like, okay, where can we find some correlation of like big problems? One of the ways my manager put it was like, you know, you don't get promoted for solving a thousand tasks. You get promoted for figuring out how to get rid of a thousand tasks, right? So if there's a big thing that keeps repeating over and over again, that should be a signal that's like, hey, why is this repeating and how do we get rid of this problem? And I think that's something that is why Facebook kind of has a very mature infrastructure even in itself. Like when you think about like they built Presto and Hive, it's like, well, yeah, because they looked at MapReduce and like, well, this is terrible, right? Like if we have to have really expensive engineers, it's really slow to put out code. We can't get things out quickly. You know, it's not great. So if we can put some layers on top of that, that automatically makes everyone's life better. So, yeah, I think that's kind of the mentality I often try to have when approaching problems is like, how do I get rid of these, this big problem, the bigger problem, not just the little daily problems that you're trying to deal with on an everyday basis. Yeah. That process of elimination of problems is something that, yeah, hopefully the learners take away from that. Maybe that's all the advice. What do you recommend to people who are interested in getting into data engineering? Do you have the Ben Rogojan path to success in data engineering or advice there? Yeah. I mean, I think, you know, your, it's funny, actually, someone asked me this question recently at one of the chats at Snowflake, and I was like, oh yeah, you can probably read Joe Reis' book just to get started, right? Like to understand like, hey, what exists, where can I kind of make your start? And one of the people on the panel was like, no, don't do that. Like learn more about the business first. And I understood the notion, right? I'm like, one of the important parts isn't just the technology side, but I think that's where it's all going to start regardless. It's like, you do need to start on the technology side, understand you're going to need to know SQL, Python, data warehousing, kind of data pipelines and how those exist and how to build them. And then kind of everything else will kind of come from that, right? Do you need to learn Snowflake, Databricks, BigQuery? Probably at some point you will, maybe, but I'd worry less on that and more about that foundation of like, how does this all work? Why am I even building a data pipeline? Like I think that was the question I didn't ask enough early on was like, yeah, why are we building data pipelines? Why am I literally taking data from one place to another just for, you know, the reason of analytics? I was asking Ethan recently because he posted about, he posted on someone's, a reference to like Airbyte. And I was like, how much do you think the industry spends just on essentially EL, right? Like just ingesting, like how much cost and time do you think we've spent? And he's like, he put out like 5 billion a year, which is probably not inaccurate just for ingestion. But yeah, like, like even this basic stuff and having that foundation and like asking why and like, why are we doing all this stuff and spending so much money I think is a great place to start. So build your solid skills and then start asking why you need them and what they actually do for the, for the business. Yeah. That's such good advice. Yeah. I often find that data pipelines, people want to build them for their own reason. You know, because again, you have the word engineer in your title, so you want to engineer things. Sometimes, oftentimes they're bridges to nowhere or pipelines to nowhere. So it's a, it's solid advice. Yeah. And I think that the comment on the panel is interesting. Like I think the book is good, but the person who made the comment, go learn about the business. I think they're absolutely correct too. You know, that's the kind of, but, but it's hard too. I mean, what, what do you, how do you recommend that an engineering or technically focused person learn about how to even look at an organization or a business? That's definitely a harder question probably to answer because like the easiest way is going to be to work at a company, right? To like actually go in and, and start when you're early on your career, it's almost hard to say like you should, because like early on in your career you want to start asking questions, but you like also want to focus on getting better, right? Like as, as an engineer, more than anything else, like you just like, Hey, get good at building pipelines, but start asking, you know, inquisitive questions about the business. How do we make money? You know, like what is this pipeline for? Start poking in that direction. And then the higher you kind of go up, I think the more you, you, you know, as you're confident in your technical skills, you can start shifting over and be like, yeah, really, really start digging into the business. Start talking to other people that aren't technical. Cause if you only talk to technical people, like we, we just kind of live in our own bubble sometimes and we're like, Oh yeah, here's the cool new tool. It solves this problem. But none of those problems generally have to do with the business. It's all like, Oh, we, I don't know, we ingest code or code. We ingest data faster. We can, you know, process more data. We can do all of this stuff. But really at some point you need to shift over and be like, but why? You know, and that, that all comes down to like, Hey, what are we giving the business to? What did it get? What is the benefit of data in real time? Is there a real benefit? Or are we just doing it because, you know, it sounds cool. Someone asked, etcetera. Real time. It's a forever. It's a forever kind of thing that we, I think we are all like, yes, the thing we're trying to do. And then you get there and you're like, but why? What does real time mean to you? Oh gosh. That's the other question. I mean, that's the question. I always love to ask this question. Yeah. It's like, what does that actually mean? I've had like calls with prospects where like, it'll end up being like, Oh, we really meant like 24, like every, you know, 12 to 24 hours, um, which I was like, well, that's very different, but okay. I get it. You know? The longest one I ever heard was, um, every month I'm not even making this up. So I was just like, whatever, however you want to define that, I guess it is what it is. Yeah. I think then part of the course we talk about the, the, the notion of a real time being this, this sort of, um, uh, it's, it's a very shifting definition. So you always need to ask like, what, what do you mean by this? I think this goes back to your, your point you're making too, of asking good questions of people and asking why, like, why do we need to do this? Uh, what's this pipeline for? Um, when you say real time, like what exactly do you mean by this? These are, uh, part of being a good, um, data engineer and it's good, uh, practitioner in general is asking good questions. So, yeah, that's awesome. Is there anything you're excited about, um, over the next year or two? There's been a lot of shifts, I guess, in the, in the world recently, obviously Tabular just got bought out, uh, which was, I mean, it's interesting. I, I hope we continue to maybe try to shift towards, you know, building reliable and usable data sets over the longterm. I think that's my, my hope, um, cause I don't know, even in a world where all data exists in maybe one homogenous data source somewhere, um, you know, you're still going to have that problem. I don't even know if companies will do that. I think that's the one thing I've thought about with the, with tabular data breaks. I was like, this works really well, like the multi-stack or multi-engine kind of, uh, data infrastructure in theory, but most companies like, you know, your Asia like entity has BigQuery, your American entity has Snowflake, your marketing team is using Redshift. And it's like, it's so everywhere that it's like, there's very few companies that I feel like will properly maybe implement that, but that's a different thing, I guess. Um, so yeah, I don't, I don't know. I think for me, my, my hope, maybe not excitement, my hope is it will just continue to review the fundamentals and, and, you know, implement them well, especially as we're trying to do everything AI. So it's like, well, if we want to make that better, we really need to, to, to take on some of these, like fundamentals of data management and data quality. So. I agree. Yeah. I think a lot of these topics are getting a second look now precisely because of that, right? There's a camp that says you can just throw AI onto everything, but, uh, as you know, a lot of corporate data sets are less than ideal, so it's a big, big issue. Um, well, thanks, Ben. Thanks for, uh, you know, I think educating the learners on a lot of great, um, ideas in data engineering, uh, for people who want to find out more about you, how can they do that? Yeah. Uh, you can find me on YouTube or Substack, uh, under the Seattle Data Guy. Um, so those are pretty, pretty much everywhere. I think I try to be a Seattle Data Guy, so you'll find me even some other places like LinkedIn and so on. Seattle Data Guy, even though you, uh, moved from Seattle a while ago. Denver Data Dude. Yeah. So awesome. Well, thanks, Ben. It's always good to chat with you. Yeah. Thank you so much, Joe. All right. Thank you.
