Log In to continue learning
Introduction to Data Engineering
17h36m
Topics
Data Science
I've seen the field of data engineering change over time. The original data engineers were software engineers whose primary work was building the software applications or organizations needed. And not that long ago, the data being generated by those software applications was mostly regarded as a sort of byproduct or exhaust. So what I mean by that is, let's say you have a software application running over here like this, and it's recording various events within this application to a log. Then this data recorded in the log might have been considered useful for things like troubleshooting or monitoring the health of the application. But it wasn't thought of as having much intrinsic value on its own. So I'll draw this little exhaust pipe coming out of the application, and then you'll have data being generated like the exhaust from a car or really just a byproduct of the application. Over time, however, as organizations began to recognize the intrinsic value of data and as a volume and variety of data being generated by software applications, uploaded by users or created through the digitization of various records, continued to grow. Those same software engineers became increasingly focused on building systems specifically for the purposes of ingesting, storing, transforming, and serving data for various use cases. With the emergence of data engineering as a central function within many organizations that work with data, the role of data engineer was born. In our book, Fundamentals of Data Engineering, my co-author Matt Housley and I propose the following definition for data engineering. Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high quality, consistent information that supports downstream use cases such as analysis and machine learning. Data engineering is the intersection of security, data management, data ops, data architecture, orchestration, and software engineering. Now, at this point in the course, I realize this definition might be a little confusing or at least sound like it requires some further explanation of the terms that appear here. Don't worry, though. In this course, we'll get into all the details of everything you see in this definition so you're comfortable with what this means and how it looks in practice. Apart from the definition of data engineering, to visualize the lifecycle of data engineering in our book, we put together this diagram. You'll be seeing this diagram a lot throughout these courses, and so I'd like to spend a little time introducing it right now. You can think of the data engineering lifecycle as comprising a series of stages. On the left here, you have data generation and source systems, and these source systems might be any kind of software application or user-generated data or sensor measurements or something else. One way or another, the lifecycle starts with data generation. Then, in the middle here, you have ingestion, transformation, storage, and serving. I've drawn a box around these stages to indicate that these are the stages of the lifecycle that will be your focus as a data engineer. You'll notice here that storage sits underneath ingestion, transformation, and serving and spans the entire width of the box. This is to indicate that data storage is an integral part of each of the stages above. In practice, the data systems you build may not look quite as simple as a lifecycle diagram might appear, but in my experience, it's helpful to think of the lifecycle in this way when it comes to visualizing the common elements of any data system. On the right side of the diagram, you have the end-use cases, and these are the ways in which stakeholders in your organization will actually derive value from the data. These include things like analytics, machine learning, or something called reverse ETL, which is basically sending the transformed or processed data back to the source systems to provide additional value for individuals in your organization who use these systems. We'll be using the term data pipeline frequently throughout this course to collectively describe the various steps data passes through on its way from generation and source systems on through to end-use cases. You can think of the data pipeline as a combination of architecture, systems, and processes that move data through the stages of the data engineering lifecycle. As a data engineer, you'll manage the data engineering lifecycle, beginning with getting data from source systems and ending with serving data for use cases, including analytics and machine learning. In simpler terms, your job as a data engineer is to get raw data from somewhere, turn it into something useful, and then make it available for downstream use cases. Within the definition of data engineering, I said that data engineering sits at the intersection of security, data management, data ops, data architecture, orchestration, and software engineering. In our book, Fundamentals of Data Engineering, we presented these six components of data engineering as what we will call the undercurrents of the data engineering lifecycle. With this diagram of the data engineering lifecycle, I'd like to write the undercurrents down below here. These undercurrents are not stages of the lifecycle like the elements above, but rather they span the entire lifecycle. The way to read this diagram now is that you have these stages of the data engineering lifecycle up here, including data generation, ingestion, storage, transformation, and serving. Then you have your undercurrents down below here, including security, data management, data ops, data architecture, orchestration, and software engineering. Now, each of these undercurrents is relevant to all the stages of the lifecycle above. In the second week of this course, we'll zoom in on each of the stages and undercurrents shown here to give you a clear understanding of how all these pieces fit together. As I mentioned in the previous video, it can be tempting as a data engineer to jump straight into implementation and experimentation with different tools and technologies. In doing so, it can be easy to lose sight of the higher-level goals of your work and how exactly you aim to provide value for your organization. Throughout these courses, we'll be looking at how you can think holistically about the entire data engineering lifecycle and the undercurrents, so that you can successfully transform stakeholder needs into requirements for your systems and deliver real value for your business. Before we move ahead, I think it's worthwhile to take a brief look at the history of data and data engineering to get a sense of how we got to where we are today. With that being said, the next video is optional. You won't need to remember any of this history to be successful in these courses. So, if you'd rather just jump right into how to get started in your work as a data engineer, then you can skip ahead. Otherwise, join me in the next video for a quick look at the history of data and data engineering.