Sign in to continue learning
Building Multimodal Data Pipelines
Intermediate
1h1m
Topics
Data Processing
You just saw all the steps necessary to process video data with the VLM. So now let's actually do it. Let's go. All right, so let's go ahead and connect to our Snowflake environment. Same setup as before, we load our credentials, create a Snowflake session, and set up our learner schema. By now, this probably looks familiar. You'll see a confirmation that the session is connected and your schema is set. Before we jump into the data, let me explain how the video was processed. I ran a vision language model, Qwen_VLM, on the meeting videos ahead of time, using a container image that I pushed into Snowflake and ran on top of GPUs. And I did this in advance so that you don't have to wait for GPU provisioning and job execution. The results are ready for you to explore. So let's take a look at what I specified in the container spec. In this container spec, you'll see that I'm requesting four GPUs, and that's because these models are computationally expensive. The frame sampling rate is set to one frame every four seconds. That's a practical tradeoff between cost and coverage. And the prompt asks the model to return structured JSON with timestamps and descriptions for each segment it identifies. The results are written into a table called video_analysis, which is what we'll query next. Let's start by looking at how the VLM broke down the video. Here, we're calculating segment durations and how many segments per meeting part, as well as the average length of each segment and the total video time covered. Here you see the table with one row per meeting part, showing these stats. Here you'll notice 10 segments were created for each video meeting part. You can also see the average length of each segment duration and how that varies depending on the meeting part. For example, the segment durations for meeting parts a and c are shorter in length than the segments for meeting parts b and d. Now let's preview a few of those segments to see what the VLM actually extracted. Here we see a preview of the meeting part, the time window, and a preview of the description. You'll also see a few rows here with truncated descriptions so that you can get a gist of each segment. In the previous cell, you probably noticed that we extracted 10 segments per meeting part. Here's a preview of the segments for meeting part A. And one thing you'll notice is that the segments are not the same in length. For example, the first segment for meeting part A here is 25 seconds in length, while the second one is nearly a minute long. And that aligns with the prompt that we fed the VLM, which is to extract 10 major segments covering events that are happening in the video. The VLM has the flexibility to set the boundaries for the segments based on what it is seeing and how it is reasoning about what it is seeing. This is why you'll notice that the segment lengths vary in duration. Let's take a look at one description to see the level of detail that the VLM provides. Here's a preview for one segment in meeting part B, which lasts around two minutes in length. In the description column, we can see a preview. And it says the meeting begins with participants seated around a large oval table, each using laptops. One person at the head of the table appears to be leading the discussion. gesturing while speaking. Notice how detailed this is compared to the truncated previews that we saw in the previous cells. Not only do we have descriptions of each video segment that was extracted by the VLM, but we can also filter video segments by time. For example, let's say we want to look at just the first five minutes of each meeting part. Now, this is a simple but useful pattern where we can narrow down to a specific window and see what was happening during that moment in time. And you'll see that the segments fall within that early time window for each part. If you're analyzing videos that follow a common pattern, you can use time-based queries to extract important information. For example, if you know that your videos tend to start with someone at the very beginning reading out an agenda. You could query in this way to extract that important information and feed it to your downstream consumers or systems. Now let's do a keyword search across all three modalities, audio, slides and video, for the word presentation. This is a straightforward LIKE query, but it shows how having all of your data across the modalities together lets you search across them in a single query. You'll see the results here tagged by source type. So you can tell exactly which modality each match came from. Now let's move from keyword search to semantic search. The video descriptions already have a vector embeddings generated using the Snowflake Arctic embed model. This is the same embedding model that we used for audio and slide content in lab 2. Let's take a look at the vector embeddings. Here's a sample of five vector embeddings for the video segments. You can see the vector embedding here in the first column, as well as the segment that they are mapped to. With the vector embeddings, we can move from keyword search to semantic search. Here, we embed a query product pricing discussion and use cosine similarity to find the most relevant video segments. This finds matches even when the exact words don't appear in the description. You'll see the results ranked by similarity score, with the most relevant segments at the top. Now let's move on to Cross-Modal Semantic Search. Here we search across audio, slides, and video by meaning, all in one query. Again, we're using the same embedding model and the same vector space, so the results from all three modalities are directly comparable and ranked together. You'll see a mixed list of results from audio transcripts, slide text, and video descriptions. Again, all ranked by relevance to your query. Earlier in the course, we saw one modality ranking higher than the others, audio. But now, it's interesting to see that audio and video rank together, interspersed in the results. This means our cross-modal search is returning relevant segments regardless of modality that are semantically similar to our query. We can start to do some interesting things across time and meaning. For example, we're going to look at segments about design discussion but only in the first three minutes. This kind of compound query is unique to video. It lets you filter by when something happened and what it was about. You'll see here in the results only the segments that match both criteria, semantically relevant and within the time window. We can do interesting analysis when we combine time and meaning. For example, in this query, we're going to look for segments about design discussion, but only in the first three minutes. This kind of compound query is unique to video because you can filter by when something happened and what it was about. And the results, you'll see only the segments that match both criteria, semantically relevant and within the specified time window. And finally, we can answer questions like, how much total time was spent discussing pricing and budget? The data output by the VLM lets us to sum up the durations of all segments with high semantic similarity to our query. This turns our unstructured video data into quantifiable insights. Here in the results, you see a single number. The total number of minutes spent on that topic across all meeting parts. I came up with this query just because I thought it'd be a fun one to try out with the data that the VLM output. In this case, how much total time did we spend talking about pricing? The fact that I can even write this query is thanks to the VLM. These are the types of questions that you can ask once a VLM has processed your video. And with your own video data, you could ask even more specific questions that are tailored to your domain and use case. Let's step back and look at what VLMs have allowed us to build. Once a VLM processes video into structured segments with timestamps and descriptions, you can query video the same way you'd query any other data. You can do traditional queries by time. For example, show me the first five minutes of all meetings. You can do keyword queries like find every meeting that discussed budget. You can do semantic queries like: Find meetings where attendees debated different product design directions, where the exact words might not appear, but the meaning matches. And finally, you can combine these. For example, meetings that discuss pricing in the first three minutes. And that's the power of what you just built. We took unstructured temporal data in video and added enough structure so that familiar search and processing patterns just work.
