What is PReLU and ELU activation function?

PReLU(Parametric ReLU) –

PReLU is vital to the success of deep learning. It solves the problem with activation functions like sigmoid, where gradients would often vanish. This approach is finding more and more success in deep learning environments. But, we can still improve upon ReLU. Leaky ReLU was introduced, which does not zero out the negative inputs as ReLU does. Instead, it multiplies the negative input by a small value (like 0.02) and keeps the positive input as is. This has shown a negligible increase in the accuracy of our models.

One of the disadvantages of ReLU is the fact that it might not be able to adapt well to sudden changes, because of its slope. This is where PReLU comes in – it can learn the slope parameter using backpropagation and prevents this problem.

Feed-forward networks only need to learn one slope parameter for each layer. In Convolutional Neural Networks, either the slope parameters can be learned for each layer or Compared to the number of weights & biases that need to be learned, this is relatively insignificant.

ELU(Exponential LU)

ELU activation functions are more computationally expensive than PReLU activation functions. They have a shallower slope, which can be advantageous for certain types of networks.

Exponential Linear Units are used to increase (make closer to zero) the mean activation of each layer. An alpha constant value is an important parameter which needs to be a positive number.

The ELU algorithm has been shown to provide more accurate results than ReLU and also converges faster. ELU and ReLU are both the same for positive input values, but for negative input values ELU smoothly “eases” down to 0.0 (i.e., -alpha) whereas ReLU sharply drops to 0.5 (-half of alpha).

Popular Posts

Spread the knowledge
 
  

62 thoughts on “What is PReLU and ELU activation function?

  1. Way cool! Some extremely valid points! I appreciate you penning this post
    and also the rest of the website is also very good.

  2. I’ve learn some excellent stuff here. Certainly worth bookmarking for revisiting.
    I surprise how so much attempt you set to create such a wonderful informative website.

  3. As I website possessor I believe the content matter here is rattling magnificent , appreciate it for your hard work. You should keep it up forever! Best of luck.

  4. Having read this I thought it was very informative. I appreciate you taking the time and effort to put this article together. I once again find myself spending way to much time both reading and commenting. But so what, it was still worth it!

  5. I like the valuable info you provide for your articles. I?ll bookmark your weblog and check once more here regularly. I am somewhat sure I?ll be informed many new stuff right right here! Good luck for the following!

  6. Simply desire to say your article is as surprising. The clarity on your publish is simply nice and that i could suppose you’re knowledgeable on this subject. Fine together with your permission let me to clutch your feed to keep up to date with approaching post. Thanks 1,000,000 and please carry on the gratifying work.

  7. Definitely believe that which you stated. Your favorite justification appeared to be on the net the simplest thing to understand of. I say to you, I definitely get irked while folks think about concerns that they just do not realize about. You controlled to hit the nail upon the top and outlined out the entire thing without having side effect , folks could take a signal. Will likely be again to get more. Thanks

  8. It is best to take part in a contest for among the finest blogs on the web. I’ll recommend this website!

  9. Great goods from you, man. I’ve be mindful your stuff previous to and you’re simply extremely fantastic. I really like what you have acquired right here, really like what you’re saying and the way wherein you are saying it. You’re making it enjoyable and you still take care of to keep it smart. I can not wait to read much more from you. That is really a great site.

  10. Simply want to say your article is as surprising. The clarity on your post is just spectacular and i can assume you’re an expert on this subject. Fine along with your permission allow me to grasp your RSS feed to keep updated with forthcoming post. Thank you 1,000,000 and please carry on the rewarding work.

  11. I really enjoyed reading your post, especially because it addressed my issue. It helped me a lot and I hope it can help others too.

  12. I really enjoyed reading your post, especially because it addressed my issue. It helped me a lot and I hope it can help others too.

  13. Heya i am for the first time here. I came across this board and I find It really useful & it helped me out a lot. I hope to give something back and aid others like you aided me.

  14. This is very interesting, You’re a very skilled blogger. I’ve joined your rss feed and look forward to seeking more of your fantastic post. Also, I’ve shared your site in my social networks!

  15. I was just searching for this information for a while. After six hours of continuous Googleing, finally I got it in your web site. I wonder what is the lack of Google strategy that don’t rank this type of informative websites in top of the list. Usually the top sites are full of garbage.

  16. I am usually to running a blog and i really recognize your content. The article has really peaks my interest. I am going to bookmark your web site and preserve checking for brand new information.

Leave a Reply

Your email address will not be published. Required fields are marked *