Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings.
We'd like to know you better so we can create more relevant courses. What do you do for work?
Instructor: Josh Starmer
Learn how the attention mechanism in LLMs helps convert base token embeddings into rich context-aware embeddings.
Understand the Query, Key, and Value matrices, what they are for, how to produce them, and how to use them in attention.
Learn the difference between self-attention, masked self-attention, and cross-attention, and how multi-head attention scales the algorithm.
This course clearly explains the ideas behind the attention mechanism. It walks through the algorithm itself and how to code it in Pytorch. Attention in Transformers: Concepts and Code in PyTorch, was built in collaboration with StatQuest, and taught by its Founder and CEO, Josh Starmer.
The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper “Attention is All You Need” by Ashish Vaswani and others, revolutionized AI with their scalable design.
Learn how this foundational architecture works to improve your intuition about building reliable, functional, and scalable AI applications.
What you’ll do:
Anyone who has basic Python knowledge and wants to learn how the attention mechanism in LLMs like ChatGPT works.
Introduction
The Main Ideas Behind Transformers and Attention
The Matrix Math for Calculating Self-Attention
Coding Self-Attention in PyTorch
Self-Attention vs Masked Self-Attention
The Matrix Math for Calculating Masked Self-Attention
Coding Masked Self-Attention in PyTorch
Encoder-Decoder Attention
Multi-Head Attention
Coding Encoder-Decoder Attention and Multi-Head Attention in PyTorch
Conclusion
Course access is free for a limited time during the DeepLearning.AI learning platform beta!
Keep learning with updates on curated AI news, courses, and events, as well as Andrew’s thoughts from DeepLearning.AI!