Sign in to continue learning
Advanced Learning Algorithms
34h24m
Let's take a look at how you can choose the activation function for different neurons in your neural network. We'll start with some guidance for how to choose it for the output layer. It turns out that depending on what the target label or the ground truth label Y is, there will be one fairly natural choice for the activation function for the output layer. And we'll then go and look at the choice of the activation function also for the hidden layers of your neural network. Let's take a look. You can choose different activation functions for different neurons in your neural network. And when considering the activation function for the output layer, it turns out that there'll often be one fairly natural choice depending on what is the target or the ground truth label Y. Specifically, if you are working on a classification problem where Y is either 0 or 1, so a binary classification problem, then the sigmoid activation function will almost always be the most natural choice because then the neural network learns to predict the probability that Y is equal to 1, just like we had for logistic regression. So my recommendation is if you're working on a binary classification problem, use sigmoid at the output layer. Alternatively, if you're solving a regression problem, then you might choose a different activation function. For example, if you're trying to predict how tomorrow's stock price will change compared to today's stock price, well, it can go up or down. And so in this case, Y would be a number that can be either positive or negative. And in that case, I would recommend you use the linear activation function. Why is that? Well, that's because then the output of your neural network, f of X, which is equal to A3 in the example above, would be G applied to Z3. And with the linear activation function, G of Z can take on either positive or negative values. So Y can be positive or negative. Use the linear activation function. And finally, if Y can only take on non-negative values, such as if you're predicting the price of a house, that can never be negative, then the most natural choice would be the ReLU activation function, because as you see here, this activation function only takes on non-negative values, either zero or positive values. So when choosing the activation function to use for your output layer, usually, depending on what is the label Y you're trying to predict, there'll be one fairly natural choice. And in fact, the guidance on this slide is how I pretty much always choose my activation function as well for the output layer of a neural network. How about the hidden layers of a neural network? It turns out that the ReLU activation function is by far the most common choice in how neural networks are trained by many, many practitioners today. Even though we had initially described neural networks using the sigmoid activation function, and in fact, in the early history of the development of neural networks, people use sigmoid activation functions in many places, the field has evolved to use ReLU much more often and sigmoids hardly ever. With the one exception that you do use a sigmoid activation function in the output layer if you have a binary classification problem. So why is that? Well, there are a few reasons. First, if you compare the ReLU and the sigmoid activation functions, the ReLU is a bit faster to compute because it just requires computing max of 0, z, whereas the sigmoid requires taking an exponentiation and an inverse and so on, so it's a little bit less efficient. But the second reason, which turns out to be even more important, is that the ReLU function kind of goes flat. Only in one part of the graph, here on the left, is completely flat. Whereas the sigmoid activation function, it kind of goes flat in two places. It goes flat to the left of the graph and it goes flat to the right of the graph. And if you're using gradient descent to train a neural network, then when you have a function that is flat in a lot of places, gradient descent will be really slow. I know that gradient descent optimizes the cost function J of WB rather than optimizes the activation function, but the activation function is a piece of what goes into computing, and that results in more places in the cost function J of WB that are flat as well and with a smaller gradient and it slows down learning. I know that that was just an intuitive explanation, but researchers have found that using the ReLU activation function can cause your neural network to learn a bit faster as well, which is why for most practitioners, if you're trying to decide what activation function to use for the hidden layer, the ReLU activation function has become now by far the most common choice. And in fact, if I'm building a neural network, this is how I choose activation functions for the hidden layers as well. So to summarize, here's what I recommend in terms of how you choose the activation functions for your neural network. For the output layer, use a sigmoid if you have a binary classification problem, linear if Y is a number that can take on positive or negative values, or use ReLU if Y can take on only positive values or zero positive values or non-negative values. Then for the hidden layers, I would recommend just using ReLU as the default activation function. And in TensorFlow, this is how you would implement it. Rather than saying activation equals sigmoid as we had previously, you can then for the hidden layers, that's the first hidden layer, the second hidden layer, ask TensorFlow to use the ReLU activation function. And then for the output layer, in this example, I've asked it to use the sigmoid activation function, but if you wanted to use the linear activation function instead, that's the syntax for it. Or if you wanted to use the ReLU activation function, that shows the syntax for it. With this richer set of activation functions, you'd be well positioned to build much more powerful neural networks than just once using only the sigmoid activation function. By the way, if you look at the research literature, you sometimes hear of authors using even other activation functions, such as the tanh activation function or the leaky ReLU activation function or the swish activation function. Every few years, researchers sometimes come up with another interesting activation function, and sometimes they do work a little bit better. For example, I've used the leaky ReLU activation function a few times in my work, and sometimes it works a little bit better than the ReLU activation function you've learned about in this video. But I think for the most part, and for the vast majority of applications, what you learned about in this video would be good enough. Of course, if you want to learn more about other activation functions, feel free to look on the Internet. And there are just a small handful of cases where these other activation functions could be even more powerful as well. With that, I hope you also enjoy practicing these ideas, these activation functions in the optional labs and in the practice labs. But this raises yet another question. Why do we even need activation functions at all? Why don't we just use the linear activation function or use no activation function anywhere? It turns out this does not work at all. And in the next video, let's take a look at why that's the case and why activation functions are so important for getting your neural networks to work.