Log In to continue learning
Neural Networks and Deep Learning
24h32m
Why does your neural network need a nonlinear activation function? It turns out that for your neural network to compute interesting functions, you do need to pick a nonlinear activation function. Let's see why. So here's the four-prop equations for the neural network. Why don't we just get rid of this, get rid of the function g, and set a1 equals z1. Alternatively, you could say that g of z is equal to z. Sometimes this is called the linear activation function. Maybe a better name for it would be the identity activation function, because it just outputs whatever was input. For the purpose of this, what if a2 was just equal to z2? It turns out if you do this, then this model is just computing y, or y-hat, as a linear function of your input features, x. To take the first two equations, if you have that a1 is equal to z1 is equal to w1 x plus b, and if then a2 is equal to z2 is equal to w2 a1 plus b, then if you take this definition of a1 and plug it in there, you find that a2 is equal to w2 times w1 x plus b1. Let's clean that up a bit. So this is b1 plus b2, and so this simplifies to w2 w1 x plus w2 b1 plus b2. So this is just, let's call this w prime b prime. This is just equal to w prime x plus b prime. If you were to use linear activation functions, or we also call them identity activation functions, then the neural network is just outputting a linear function of the input. And we'll talk about deep networks later, neural networks with many, many layers, many, many hidden layers. And it turns out that if you use a linear activation function, or alternatively, if you don't have an activation function, no matter how many layers your neural network has, all it's doing is just computing a linear activation function. So you might as well not have any hidden layers. Some of the cases that I briefly mentioned, it turns out that if you have a linear activation function here, and a sigmoid function here, then this model is no more expressive than standard logistic regression, without any hidden layer. So I won't bother to prove that, but you could try to do so if you want. The problem is that a linear hidden layer is more or less useless, because the composition of two linear functions is itself a linear function. So unless you throw a non-linearity in there, then you're not computing more interesting functions, even as you go deeper in the network. There is just one place where you might use a linear activation function, g of z equals z, and that's if you are doing machine learning on a regression problem. So if y is a real number, so for example, if you're trying to predict housing prices, so y is not 0 or 1, but it's a real number. Anywhere from, I don't know, $0 is the price of a house, up to however expensive houses get. I guess maybe houses can be potentially millions of dollars. So however much houses cost in your data set. But if y takes on these real values, then it might be okay to have a linear activation function here, so that your output, y hat, is also a real number, going anywhere from minus infinity to plus infinity. But then the hidden units should not use linear activation functions. They could use ReLU, or tanh, or leaky ReLU, or maybe something else. So the one place you might use a linear activation function is usually in the output layer. But other than that, using a linear activation function in a hidden layer, except for some very special circumstances relating to compression that I won't want to talk about, using a linear activation function is extremely rare. Oh, and of course, if you're actually predicting housing prices, as we saw in the week 1 video, because housing prices are all non-negative, perhaps even then you could use a ReLU activation function so that your outputs, y hat, are all greater than or equal to 0. So I hope that gives you a sense for why having a non-linear activation function is a critical part of neural networks. Next, we're going to start to talk about gradient descents. And to do that, to set up for our discussion for gradient descents, in the next video, I'm going to show you how to estimate or how to compute the slope or the derivatives of individual activation functions. So let's go on to the next video.