Sign in to continue learning
Convolutional Neural Networks
35h41m
In terms of designing content architectures, one of the ideas that really helps is using a one-by-one convolution. Now, you might be wondering, what does a one-by-one convolution do? Isn't that just multiplying by a number? That seems like a funny thing to do. Turns out, it's not quite like that. Let's take a look. Here's a one-by-one filter. I put a number 2 there. And if you take this 6x6 image, 6x6x1, and convolve it with this one-by-one-by-one filter, you end up just taking the image and multiplying it by 2. So, 1, 2, 3 ends up being 2, 4, 6, and so on. And so, a convolution by a one-by-one filter doesn't seem particularly useful. You just multiply it by some number. But that's the case of 6x6x1 channel images. If you have a 6x6x32 instead of by 1, then a convolution with a one-by-one filter can do something that makes much more sense. And in particular, what a one-by-one convolution will do is it will look at each of the 36 different positions here, and it will take the element-wise product between 32 numbers on the left and the 32 numbers in the filter, and then apply a value nonlinearity to it after that. So, to look at one of the 36 positions, maybe one slice through this volume, you take these 36 numbers, multiply it by one slice through the volume like that, and you end up with a single real number, which then gets plotted in one of the outputs like that. And in fact, one way to think about the 32 numbers you have in this one-by-one-by-32 filter is as if you have one neuron that is taking as input 32 numbers, multiplying each of these 32 numbers in one slice in the same position, height and width, but these 32 different channels, multiplying them by 32 weights, and then applying a value nonlinearity to it, and then outputting the corresponding thing over there. And more generally, if you have not just one filter, but if you have multiple filters, then it's as if you have not just one unit, but multiple units that are taking as input all the numbers in one slice, and then building them up into an output that is 6 by 6 by number of filters. So, one way to think about a one-by-one convolution is that it is basically having a fully connected neural network that applies to each of the 62 different positions, and what this fully connected neural network does is it inputs 32 numbers and it outputs number of filters outputs. So I guess, depending on your notation, this is really a NC of L plus 1, if that's the next layer. And by doing this at each of the 36 positions, each of the 6 by 6 positions, you end up with an output that is 6 by 6 by the number of filters. And this can carry out a pretty non-trivial computation on your input volume. And this idea is often called a one-by-one convolution, but it's sometimes also called network-in-network. And it's described in this paper by Minlin, Qiangchen, and Shuichen Yan. And even though the details of the architecture in this paper aren't used widely, this idea of a one-by-one convolution, or this sometimes called network-in-network idea, has been very influential and has influenced many other neural network architectures, including the Inception Network, which we'll see in the next video. But to give you an example of where one-by-one convolution is useful, here's something you could do with it. Let's say you have a 28 by 28 by 192 volume. If you want to shrink the height and width, you can use a pooling layer, so we know how to do that. But what if the number of channels has gotten too big and you want to shrink that? How do you shrink it to a 28 by 28 by 32 dimensional volume? Well, what you can do is use 32 filters that are one-by-one. And technically, each filter would be of dimension one-by-one by 192, because the number of channels in your filter has to match the number of channels in the input volume. But you use 32 filters, and the output of this process will be a 28 by 28 by 32 volume. So this is a way to let you shrink NC as well, whereas pooling layers are used just to shrink NH and NW, the height and width of these volumes. And we'll see later how this idea of one-by-one convolutions allows you to shrink the number of channels and therefore save on computation in some networks. But of course, if you want to keep the number of channels at 192, that's fine too. And the effect of a one-by-one convolution is it just adds non-linearity. It allows you to learn a more complex function of your network by adding another layer. The input is 28 by 28 by 192, and output is 28 by 28 by 192. So you've now seen how a one-by-one convolution operation is actually doing a pretty non-trivial operation and allows you to shrink the number of channels in your volumes, or keep it the same, or even increase it if you want. In the next video, you'll see that this can be used to help build up to the inception network. Let's go on to the next video.