Gain in-depth knowledge of the steps to pretrain an LLM, encompassing all the steps, from data preparation, to model configuration and performance assessment.
We'd like to know you better so we can create more relevant courses. What do you do for work?
Instructors: Sung Kim, Lucy Park
Gain in-depth knowledge of the steps to pretrain an LLM, encompassing all the steps, from data preparation, to model configuration and performance assessment.
Explore various options for configuring your modelâs architecture, including modifying Metaâs Llama models to create larger or smaller versions and initializing weights either randomly or from other models.
Learn innovative pretraining techniques like Depth Upscaling, which can reduce training costs by up to 70%.
In Pretraining LLMs youâll explore the first step of training large language models using a technique called pretraining. Youâll learn the essential steps to pretrain an LLM, understand the associated costs, and discover how starting with smaller, existing open source models can be more cost-effective.
Pretraining involves teaching an LLM to predict the next token using vast text datasets, resulting in a base model, and this base model requires further fine-tuning for optimal performance and safety. In this course, youâll learn to pretrain a model from scratch and also to take a model thatâs already been pretrained and continue the pretraining process on your own data.
In detail:
After taking this course, youâll be equipped with the skills to pretrain a modelâfrom data preparation and model configuration to performance evaluation.
This course is ideal for AI enthusiasts, data scientists, and machine learning engineers who want to learn the complete process of pretraining LLMs. Basic knowledge of Python and large language models is recommended.
Introduction
Why Pre-training
Data Preparation
Packaging Data for Pretraining
Model Initialization
Training in Action
Evaluation
Conclusion
Course access is free for a limited time during the DeepLearning.AI learning platform beta!
Keep learning with updates on curated AI news, courses, and events, as well as Andrewâs thoughts from DeepLearning.AI!