Sign in to continue learning
Natural Language Processing with Attention Models
26h58m
The T5 model could be used on several NLP tasks, and it also uses similar training strategies to BERT. Concretely, it makes use of transfer learning and masked language modeling. The T5 model also uses transformers when training. So let's take a look at how you can use this model in your own applications. So you're going to understand how T5 works, then you're going to recognize the different types of attention used, and see an overview of the model architecture. So T5 transformer is known as the text-to-text transformer, and you can use it in classification, you can use it for question answering to answer a question, you can use it for machine translation, you can use it for summarization, and you can use it for sentiments. And there are other applications that you can use T5 for, but we'll focus on these for now. So the model architecture and the way the pre-training is done, first of all, is you have an original text, something like this, where you have, thank you for inviting me to your party last week. So you mask the certain words, so for inviting me last, and then you replace it with these tokens, like brackets X, brackets Y, so brackets X corresponding to for inviting, and brackets Y corresponding to last. And your targets are going to be brackets X for inviting, brackets Y last, and these tokens or these brackets, they keep going in increments order. So then it's brackets Z, then maybe brackets A, brackets B, and so forth. So each bracket corresponds to a certain target. The model architecture and the different transformer architecture variants that we're going to consider for the attention part here. So we start with the basic encoder decoder representation. So you can see over here, you have fully visible attention in the encoder, and then causal attention in the decoder. And then you have the general encoder decoder representation, just as a notation. So light gray lines correspond to causal masking, and dark gray lines correspond to the fully visible masking. So on the left, as I said again, it's the standard encoder decoder architecture. In the middle over here, what we have, we have the language model, which consists of a single transformer layer stack, and it's being fed the concatenation of the inputs and the targets. So it uses causal masking throughout, as you can see, because they're all gray lines. And you have X1 going inside over here, get its X2, X2 goes into the model, X3, and so forth. Now, over here to the right, we have prefix language model, which corresponds to allowing fully visible masking over the inputs, as you can see here in the dark arrows, and then causal masking in the rest. So as you can see over here, it's doing causal masking. So the model architecture, it uses encoder decoder stack. It has 12 transformer blocks each. So you can think of it as a dozen eggs, and then 220 million parameters. So in summary, you've seen prefix language model attention, you've seen the model architecture for T5, and you've seen how the pre-training is done similar to BERT, but we just use masked language modeling here. You now have an overview of the transformer model. You know how to train it, and you've seen that you can use it on multiple tasks. In the next video, I'll be talking about a few training strategies for this model. See you there!