summary of activation function in deep learning

1. Introduction

The neaural network is linear without non-linear activation function, we add non-linear transformation for complex task, such as language translation or image classification. However, there are lot of activation functions, what's the diffetence between them and what are the characteristics of corresponding activation function? To analyze their features, we write this document.

2. features of different activation function

2.1 setp function

mathematically

\begin{array}{l}f(x)=1, \text { if } \mathrm{x}>=0 \\f(x)=0, \text { if } \mathrm{x}<0\end{array}

diagram

description:

2.2 Identity

mathematically

$$f(x)=x$$

diagram

2.3 Relu

mathematically
$$f(x)=max(0,x)$$

diagram

2.4 Sigmoid

mathematically
$$f(x)=\frac{1}{1+e^{-x}}$$
diagram

2.5 tanh

mathematically
$$f(x)=\frac{(e^x-e^{-x})}{(e^x+e^{-x})}$$

diagram

2.6 Leaky ReLU

mathematically
$$\begin{array}{l}f(x)=ax,x<0\\ f(x)=x,\text{ otherwise }\end{array}$$
diagram

2.7 PReLU (Parameteric Rectified Linear Unit)

mathematically
$$f\left(y_i\right)=\{\begin{array}{ll}{y}_i,& \text{ if }{y}_i>0\\ a_i{y}_i,& \text{ if }{y}_i\le 0\end{array}$$
diagram

2.8 RReLU (Randomized Leaky Rectified Linear Unit)

mathematically
$$\begin{array}{c}{y}_{ji}=x_{ji}\text{ if }{x}_{ji}\ge 0\\ y_{ji}=a_{ji}{x}_{ji}\text{ if }{x}_{ji}<0\end{array}where{\alpha }_{ji}\sim U(l,u),l<u\text{ and }l,u\in [0,1)$$
$$U(l,u)\; means\; uniform\; distribution.$$

diagram

Reference

1 [what, why and which??activation functions]( https://medium.com/@snaily16/what-why-and-which-activation-functions-b2bf748c0441#:~:text=4.1 Linear or Identity Activation ,signal%20proportional%20to%20the%20input. "what, why and which??activation functions")
2 ReLU paper
3 PReLU paper
4 RReLU paper