Log In to continue learning
Data Engineering
Intermediate
Topics
Data Engineering
Data Processing
Deep Learning
Search and Retrieval
So, I'm here with my really good friend, Zach Wilson. Hey, hey. So, Zach is... It's better you describe yourself, Zach. For sure. I mean, I have done a lot of data engineering for the last 10 years, like a bunch of big tech companies. I do a lot of content. I'm a teacher. That's what I've been doing over the last year or so. I have a boot camp called DataExpert.io. We're launching pretty soon. It's going to be pretty awesome. V4 is going to be really, really awesome. And so, I kind of did a big shift over the last year from engineer to entrepreneur. Yeah, that's been kind of my journey. What were some of the companies you worked at as a data engineer? Yeah, for sure. Well, I started in big tech at Facebook, and then I went to Netflix, and then I went to Airbnb. I did almost exactly two years at those three companies. Awesome. I spent almost the exact same amount of time at each one. It was kind of spooky how similar they were. I'm sure some of the learners for this course, they'll be interested in a couple things. Let's talk about maybe you take a course, you maybe take a boot camp, and maybe getting a job as a data engineer will start there. What would you recommend for people looking at getting their first job? And then we'll talk about career progression. Yeah, that's a good one. Data engineering is a weird field because if you're trying to get that first job, trying to find those junior roles, there isn't that many junior roles, generally speaking, for data engineering just because a lot of companies, they need one data engineer, and they don't want the one data engineer to be a junior data engineer. That's because they want that person to handle the pipelines. So that's a big thing that can happen. So finding that entry foothold role can be hard. So there's a couple different ways to go. Some people would go with an adjacent role where they start as a software engineer or they start as a data analyst. So for me in my career, my first role was actually a software engineering role where I did a lot of automations. It was way more software engineering focused than data engineering focused, but it had a lot of the same skills because there was a lot of overlap between backend and data and all these roles. The titles are more like labels on top, but the skills underneath are 70% overlap. So a lot of times there's a lot more of those plentiful roles in junior software engineer or junior data analyst. You're going to be able to get your foothold in on those roles a lot easier than with junior data engineer. But I don't know. That's probably one of the bigger things that I think is a good strategy. Okay. And if you're in an adjacent role, what's some ways that you would try and navigate towards data engineering? That's a good one. So for me, back in 2014, 2015, when I was interviewing, this is when I was interviewing for a role at Teradata back then. I worked at Teradata for a little bit. And when I was interviewing for that role, or when I was prepping for the interview for that role, I focused on a couple of different things. It has shifted, actually, what makes a good data engineer in 2015 and what makes a good data engineer now. What makes a good data engineer? Yeah, yeah, right? I think that there's a couple of things there. The big things that I was intimidated by in that interview were they were like, we're going to grill you on Linux commands. And I'm like, I know three. I know LS, CD. I'm like, I probably need to get a little bit better in this area. And that was definitely something that you need to know. You need to know just how to write a Bash script and all that stuff. That stuff is super important. Because that's how you kind of stitch all of the services together. But then you also need to know how to use the services like BigQuery or Airflow or Spark and those cloud services. And then those are kind of stitched together with Python and Bash. And so, yeah, Python, Bash, SQL, and then the cloud technologies. And then if you can kind of learn how to stitch all that together and build pipelines, that's how you're going to really make it as a data engineer. And what does it mean to make it as a data engineer? How do you know when you've... Yeah, that's a good one. How do you know you've become a good data engineer? Oh, that's a good one. Okay, I think there's a couple of things there. A big way, I think, at least for me, a signal in my career where I was like, okay, I am now a good data engineer is when I was no longer just focused on I need to work on this pipeline. And I'm thinking about the intricate details of how to build the pipeline. And I'm thinking more about why are we building this pipeline? What is the impact of this pipeline? And thinking bigger picture, right? And actually thinking about... Because I honestly feel like one of the things that's a big signal that you are a good data engineer is you know when to say to not build a pipeline. Right? Good point, yeah. Because there's just so many analytics teams out there that feel like they need the answer to every little question. And then there's data engineers out there who are like, yes, sir, I'll go build it. And then it's like they spend like two weeks to answer this thing that they use one time. And it's like, was that even worth it? Could you have just relied on intuition for that one number one time and then the data engineer can focus on things that are more repeatable? Right. Because I think that's a big problem in the space right now. Data engineers get burnt out because business is always just like, I want an answer. I want an answer. We need data. And then the data engineer is like, all right, I'll go and slap something together real quick. And then they're like, why is it bad? Why is there not good quality? And it's like, because you expected me to do it in a week. And so, yeah, that balance of like being able to say no, because that was something that I actually did bad through a lot of my career. Like even up through Facebook and Netflix, those two companies, I just said yes to everything for the most part. And then when I got to Airbnb, that was when I was like, you know what? I'm going to try and do this differently this time. I'm not going to just say yes to everything and burn myself in the face. I'm not going to do that anymore. And because of that, though, that's how my life became a lot more expansive, because that's what gave me the time back to make content, build a brand and get creative with stuff. Because it's like at Netflix, when I was thinking about it, when I was working at Netflix and I didn't have good boundaries, it was like, well, I can't do anything else. It's like, that is my life. And so that's where like, yeah, for sure. That's another big one is just like being able to live the good life as well, right? And not like try to suffer from so much burnout. Yeah, it's a big thing in tech in general. I think it's easy to work yourself really hard. I guess if you were to talk to your younger self, to talk to young Zach Wilson as he's getting that first job, what's some advice you'd give that younger version of yourself? Oh, that's good. That's good. I mean, I think there's a couple of things there. One is like your work will not speak for itself. That's not true. Like you do have to market yourself. You do have to say you have to show people your work and the good work that you're doing. It sucks. And I know a lot of engineers out there. They like that makes them feel gross because they're like, why can't my just good work speak for itself? Right. And like that's like but that's that's just like a magical land. Right. And that's definitely a mindset I had for a very long time. And I felt like and I actually felt like it has a different part about it that actually cuts in kind of a negative way as well. Like my work will speak for itself. Is that the undertones of that also say like, like my work doesn't need feedback or my work does. It's not as collaborative. Right. As like, hey, like I want to get more feedback from people. And that's one of the things that you can do to like point at me like, hey, this is this is a much better way to do things. Right. And like you can get more feedback from people as well if you are trying to sell your work and get more visibility. I think that's a big one. I think another big one is like learn Spark even sooner. I know that was like I had the vision for Spark when I was like in like 2013, 2014. Right. Right. When it came out, I knew it was going to be just monstrous. Right. And then like it took me like three or four years to actually get into the opportunities to like learn it and actually get good at it. But like, yeah, that was I had the vision. I knew I needed to learn it. But then it was just like finding that opportunity to learn. It took more time than I thought it would. Right. For sure. How do you go about learning and keeping up your skills with data engineering? That's good. That's a good. I mean, since I quit my job, that's that's been that's been a very, very different. It's a different ballgame. But like I'm going to I'm going to position this more like as in the previous nine years when I was working at data engineering role is there's a couple of things there. One is like you want to be thinking about how to push the boundaries on the three V's. Right. So volume, velocity, variety. Like if you should be thinking about like, OK, can. Like, for example, when I worked at Netflix, I was working like in like cybersecurity. And one of the things I was like, I was like, I want to work on one of the biggest pipelines that Netflix has. Right. And that was the network logs pipeline. Right. And that pipeline was like two petabytes a day. Right. And I was like, yeah, I want to work on that. I want to like. But like that was like outside of my job responsibilities because like I was more focused on like detection and like a completely different kind of area of cybersecurity. But like I was they were saying, hey, we need help here. And I was like, OK, I'll take a look at it. Like so sometimes it's like looking at adjacent things and trying to find teams that are going to work on that are doing things that like you want to get into. Right. Especially like if they are doing like more cutting edge stuff. Like that's the other reason why at Netflix it was so good. I was working on the detection team because like that was really when I got that first like like real time. Like wait, Kafka is crazy. Kafka is like this is like we don't do this like on a daily basis anymore. This is just like it's just running all the time. It's just here all the time now. And so like that was the other thing that really drew me to that role was like because when I was at Facebook, that was the problem. I was like, I just was writing freaking SQL queries and aggregate queries and making metrics. And it was just like a lot of the same. It was a lot of the same. And that was like another big reason why I was like, I need to go somewhere else. And that's when I went to Netflix. I picked up Scala, picked up Flink, all that stuff. And like so I think there's the two things that are like one is like recognizing when you're in a role that like you're not growing in. That's a big one. And then the other one is like when you're in a company that is supportive of your growth, finding those opportunities and seeking them out. Because if you rely on your manager to give you those opportunities, they might come. But like they're going to come like one like a tenth as much as if you seek them out. And because if you can seek them out and explain the value to your manager and then and for your manager, you're just essentially like, hey, manager, like I want to work on this. Like most of the time, they're just like if you have a manager who's not toxic, they're going to be like, OK, go go check it out or like spend some amount of time on it. Right. Yeah, for sure. Learning is that's that's that was a big priority for me for a long time. For sure. Super cool. You mentioned something interesting, too. And one thing we're working with the learners in this course early on is knowing how to identify and explain value to various people. You sort of alluded to this and we've talked that you've gone from working on being very reactive, I guess, in your work to being very proactive and trying to find valuable things to work on. How do you determine what's valuable to work on versus what isn't valuable? That's great. There's when I when I have kind of a set of heuristics for that, like there's one side of it, like depending on where you're at in your career, this can actually make a difference. Like if you are like senior, like junior to senior and you're in those three level like junior, mid senior, then like generally speaking, it's better to prioritize measurable impact. Like so in that case, like measurable impact being like I made this data model 30 percent more efficient through better modeling techniques or like by optimizing the pipelines or changing file formats or whatever. Like there's all those different like kind of cloud efficiency wins you can get or like on the other side, you have like I've created a new data set that gave visibility into experimentation that then caused this business metric to lift. And you can have that kind of measurable impact. It's very good to have those kind of more measurable impacts when you are like earlier in your career, when you're later in your career. It's like it's more about leveraged impact and like I call it like horizontal impact where like you it's less about like impact. It's less about doing like specific units of work. And it's more about like how do you change how work is getting done? It's like it's like how work is getting done, not what work is getting done. And like if you can impact each of your like I'll give you an example. Like so when I worked at Netflix, I was working on this project where like when I was deploying the pipeline, the spark pipeline, it would take like I would build the jar and then I would deploy it and it would take like eight minutes. And I was just like and then I would like it would take eight minutes and then I'd be like, wow, I forgot a comma. And I'm like, I know I have to wait eight more minutes. And I'm like and then I'm just like it was just maddening. It was so maddening me like the dev process for that was just like so painful that like what I did was like I looked at I'm like, why is this painful, though? Like what is happening here? And it turns out like that like that that spark pipeline, what it was doing was it was compiling all these like extra like Hadoop libraries and all this like it just ancient stuff that was also being uploaded. And like and I was like, OK, and then what it is, I wrote a little bit of a groovy script like because that's what you use for Gradle is a groovy script plugin that what it did was it removed that code. And then then the deploys instead of taking eight minutes took like one minute. And then I'm like, yes, I'm so happy. The part that was great about that was like it made me happy because now now my my development process is way more enjoyable. But it also where the big impact happens is when you take that and you're like, hey, team, adopt this. And like, hey, like now this is like now I'm saving like, you know, 10, 20 engineers. If I get it across the team, it's like 10, 20 engineers. Then it's like if you push even further, you can get it across the whole company. And then you think about like, OK, if I'm saving each engineer 30 minutes a day, that's like crazy. So the impact there is insane. And so like that, you want to be looking for impacts like that when you are growing in your career, especially beyond senior, because that's really like why you're there. Right. You're there for identifying those opportunities and you're there for exercising good judgment about like what to work on. Right. And being like kind of good about like prioritizing like this is more important than that. And that's going to be the big, big impact opportunities as you grow into those kind of like higher levels of engineering. Right. Cool. What do you think data engineering is going over the next five years? Oh, that's a good one. I think there's a couple of things there. Like, I think one is service owners are going to be better. Like I've even noticed this throughout my career, like as I've kind of grown, is that like service owners themselves are just getting a lot smarter about logging. And like if you if you get the logging contract right and then the fact data generation right, then like that is makes data engineering a lot easier, makes it a lot easier. And so like if they do that, like if service owners are actually owning like the data that their services are generating, then like that is a future world that I think of like where the data engineer and the back end engineer are going to be like, like morphed. They're going to like morph together a little bit. That's like one side. Like that's more like the like the there's like two titles. I actually had a big debate at Netflix when this was happening because I was a service owner at Netflix as well. I owned a service called Asset Inventory that what it did was like it had a graph database of everything in the cloud at Netflix. And like I had a big debate with my freaking manager and stuff about how like I'm not really a data engineer. And they actually changed my title because I was I was also like, I'm not a software engineer either. And then we made a compromise. I'm like, my title is software engineer comma data. That was my I'm like, that's my title. And so and I think that that title is actually going to be a title that is going to get bigger and like is going to be one of those things that's going to kind of expand. Because like really, it comes back to like the service owner is going to be the person who has the context anyways. Right. And it's like so then you don't have to like train another person to generate that data like they can do it themselves. I think that's one kind of area. The other one that's kind of interesting as well is kind of like a little bit further downstream on like more on like things that are closer to the business where you have like data engineers that are like more like the role that's kind of in that space is more like analytics engineer where it's like they're pulling in data sets that are already kind of curated and cleaned. But then they're turning them into metrics and dashboards and experiments and all that stuff. And I think that that role, the analytics engineer is also going to kind of expand out and get a lot bigger as well. And I mean, beyond that, I think it comes back to like, you know, LLM. I think a lot more focus on like unstructured data. Right. Where it's like, it's like, okay, give me like rows and columns just don't exist anymore. It's just like give me a giant blob of text or like that's that's good data. I think that new techniques are going to have to come out, though, because it's like, how do you determine if a giant blob of text is high quality? Right. And so like that's I think there's going to be a lot in that space that is like a lot of innovation that's going to need to happen in that space as well. Because LLM hallucination is something that I think people are starting to get a little bit sick of where they're like, can't we just trust this thing? Right. And so, yeah, for sure. I mean, what about you? What do you think data engineering is going? Oh, it's a good question, isn't it? There's always a vanilla answer. And it's that it will continue to exist. I think it'll I think it really depends on the types of companies that you're at, too. Whereas at, you know, big companies like Netflix, for example, I can definitely see it moving more towards this hybrid software and data fusion, because I don't know, there's not much of a difference, in my opinion, in most cases. And this is sort, you know, and it depends. Maybe at other types of companies, data engineering is more of a a defined role, where it's very data centric. But yeah, I think it just depends. You know, as we, as we talk about this right now, you know, AI is, I think the giant question mark in terms of what that does, I think it's going to do a lot. I actually think that sort of one possibility is the role of data engineers are morphing both upstream and downstream, really, you know, machine learning, software as you're defining data products, and then deploying quote data products. So I could see a few scenarios of where this plays out. I think it's probably like all of the above, because like every company is different, and they have different needs. And so I don't think there's universally one answer. But that's, yeah, one thing that I've noticed, one of the things that I've noticed as well, that it actually comes from Spark, it's an oddity of Spark, is that companies that have JVM backends are more likely to have that hybrid role of software engineer and data in you. But you have to. Because then it's like, well, and it's also because it's easy too, right? Where it's like, oh, the Spark job can just compile the backend and it just brings it in, right? And it's like, and that makes it very easy for those two things. That was a big thing I did at Airbnb was like, they have like pricing and availability libraries that are in the backend. And then I have the Scala Spark job that just compiles them. And then I run a bunch of data through it. And then it just pulls it out. And it's like, it's like, part of it is that too, I think just like what you're saying, but different companies, you know, not everyone is on the JVM, right? I mean, not even half are on the JVM. And so like, for companies that aren't, that don't, who can't have that like full like integration like that, like, I think data engineering as like a more separate role is probably going to be more the case than not. Yeah. Well, Zach, it's been a pleasure having you. Thank you. Yeah. And thanks a lot. And hopefully learners, you get a lot out of listening to Zach and feel free to, you know, and definitely follow Zach on all the socials. Yeah, that's true. All what... Check out Data Expert. Yeah. Oh, yeah. I'm everywhere. You can find me at exactly everywhere. E-C-Z-A-C-H-L-Y. That's going to be the, that's my, that's my name everywhere. So yeah, for sure. But Data Expert is my company. So then that's where I do a lot more of the teaching stuff as well. So... Awesome. All right. All right. Yeah. Yeah. Yeah.
