Sign in to continue learning
Once you are done training your model and have optimized the cost function, you still need to actually get the word embeddings. In this video, I will show you several methods to extract the word embeddings from the weights learned by your model. You have already implemented forward propagation, back propagation, and gradient descent to train the neural network of a continuous bag-of-words model. But how do you get the word embeddings out of your trained neural net? As you may remember, word embeddings are not directly output by the training process. They are a byproduct of the process. I'll now explain how you can extract word embeddings from a trained neural net. Remember, these are vectors that carry the meaning of the words of the vocabulary, based on the context words in the corpus. Once you've trained your neural network after iterating through all of your training data, possibly several times, you can extract three alternative word embedding representations. The first possibility is to consider each column of W1 as the column vector embedding vector of a word of the vocabulary. Recall that matrix W1 has v number of columns, so it has one column for each word in the vocabulary. The mapping between columns of W1 and words uses the same order as the input rows. For example, with the corpus I'm happy because I'm learning, if the five rows of the input vector or matrix corresponds to am, because, happy, I, and learning, then in W1 the first column will be the word embedding column vector for am, the second for because, and so forth. The second option to extract word embeddings is to use each row of W2 as the word embedding row vector for the corresponding word. Matrix W2 has v rows, one row for each word in the vocabulary. Again, the order is the same as the input vector or matrix. So with our sample corpus and inputs, the first row would be the word embedding row vector for am, the second for because, and so on. The third and final option is to take the average of the two previous representations. So if you want word embedding column vectors, you would average W1 and a transpose of W2 to obtain W3, a new n by v matrix. You can then extract the word embedding vectors from each column of W3 as you did previously. So with our visual example, the word embedding for am would be the first column of W3, which would be the average of the values of the first column of W1 and of the first row of W2. In this week's assignment, you'll be averaging W1 transposed and W2 to extract the word embeddings as row vectors. Now that you know how to train these word vectors, in next week's video you'll learn how to test them. Specifically, you'll learn about two types of evaluation metrics. The first type is intrinsic evaluation, and the second type is extrinsic evaluation.