Welcome to the second course of the data engineering specialization. In the first course, you got a high-level overview of the field of data engineering, the principles of good data architecture, and how to translate stakeholder needs into requirements and tooling for your data systems. In this course, you learn more about ingesting data from source systems, as well as data ops and the orchestration of end-to-end data pipeline. I'm here again with your instructor, Joe Rees, who is also co-author of the best-selling Fundamentals of Data Engineering. Joe, maybe you can say a little bit about what learners will see in this course. Sure thing, Andrew. So this course, as you said, includes a focus on the first two stages of the data engineering lifecycle, which are data generation in source systems and data ingestion from those source systems. So we'll start by taking a look at different types of source systems, maybe things like databases or object storage and streaming systems. And we'll get into the details of how you'll interact with these source systems in your work as a data engineer. After that, we'll look at ingesting data from source systems, as well as the data ops and orchestration aspects and building data pipelines. That sounds great. And you know, for so many AI systems, the data engineering or the data ingestion is like 80% of the work, and then the machine learning modeling is maybe 20% of the work. But people's attention on these two topics is often flipped, where 80% of the attention is on AI modeling and not enough attention, and best practices and tools, frankly, is on the data ingestion. In fact, for some of my early work, when I was at a very large tech company, I was responsible for the company's user data warehouse. And so every piece of data that touched an individual user was supposed to come into my data warehouse, and that created a lot of value for the company. But that intellectual work to design the database, the data schema, the ingestion system, the data ops maintainer, that turned out to be, you know, pretty massive undertaking. It is a massive undertaking. I've always found that when we're talking about ingesting data from source systems, this is everything. If you can't get the data, there's not much else you can do with it. That should seem simple enough. But as you point out, it feels like a lot of times ingestion is ignored or focused on other things. And so this is a fundamental thing you need to get correct. If you can't get the data, you can't really do anything else. So in this part of the course, we'll definitely be talking about, you know, understanding the source systems from which you're going to get your data, and different ways of ingesting it, as well as different ways of orchestrating these data pipeline workflows and monitoring them to make sure that you're preserving data quality as well as ingestion performance and other characteristics that are very important to your work as a data engineer. In fact, I find this to hold true for many different data workloads, from the structured tablet data to as the work process is more on structured data, like text and images. This seems to have stayed the same. And in fact, even for when I chat to my friends training the largest language models and kind of some of the leading AI teams, a lot of the time, not all, but a lot of the time is actually spent, you know, thinking about the data, and then some also on the model training. But the data ingestion occupies so much time for all of these AI workloads. That's really interesting. I guess what are they seeing in terms of the complexities with ingesting data? Because I imagine they're working at a very massive scale. Yeah, I think things like there is actually a lot of Internet data. So things like Common Crawl has a lot of data. But given the data, how do you ingest it, process it, screen it to be high quality? And then also, if you have gaps in the data, if you notice your model does one of these topics, how do you figure out where it doesn't do quite as well on? And where on earth do you go to get data, if it even exists, to fill in those gaps? So these are a lot of the intellectually challenging things that I find people on the ground training, even some of the largest OMMs, large multimodal models, and other large foundation models spend time worrying about. That's fascinating. And one of the things we're talking about in this course and that you'll be learning about is ingesting data from multiple types of source systems, right? So obviously we're going to talk about ingesting tabular data, but that's such a small subset of the total data universe these days. When we're talking about unstructured data sets from text all the way to images and video, this is increasingly becoming a much bigger universe than I would say we traditionally have used or thought about in the data world. So one of the things we'll be covering in this course, again, is not just structured data sets from databases, but we're also going to be getting into working with text, image data, and so forth. So to help you prepare as a data engineer, not just for the workloads of today, maybe if you're working in a data warehouse before the workloads of the future. Most of the value of data has probably been structured data up to this point. But as our ability to process unstructured data grows, we'll see if that shifts. Maybe it's already shifting. And the volume of unstructured data in the world is much greater than the volume of structured data in the world. So I think this would be even more challenges and even more jobs for data engineers. Exactly. So hopefully this is going to be a very exciting introduction to the data engineering lifecycle. Again, ingestion from source systems, orchestrating these workloads, and monitoring these workloads is so critical to your work as a data engineer. And again, as Andrew pointed out, it's only going to get more interesting, more exciting, and much bigger. So lots of exciting things. Let's go on to the next video to dive into all of these topics.