S-形函数广泛应用于ANN 的激活函数

 

 

 

 

 

 

{\displaystyle f(x)={\frac {1}{1+e^{-x}}}}
 {\displaystyle f(x)=\tanh x={\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}}
 {\displaystyle f(x)=\arctan x}
 {\displaystyle f(x)=\operatorname {gd} (x)=\int _{0}^{x}{\frac {1}{\cosh t}}\,dt}
{\displaystyle f(x)=\operatorname {erf} (x)={\frac {2}{\sqrt {\pi }}}\int _{0}^{x}e^{-t^{2}}\,dt}
{\displaystyle f(x)=(1+e^{-x})^{-\alpha },\quad \alpha >0}
 {\displaystyle f(x)={\begin{cases}\left(\int _{0}^{1}{\big (}1-u^{2}{\big )}^{N}\ du\right)^{-1}\int _{0}^{x}{\big (}1-u^{2}{\big )}^{N}\ du\quad &|x|\leq 1\\\operatorname {sgn}(x)&|x|\geq 1\\\end{cases}}\,\quad N\geq 1}
 {\displaystyle f(x)={\frac {x}{\sqrt {1+x^{2}}}}}.

The integral of any continuous, non-negative, "bump-shaped" function will be sigmoidal, thus the cumulative distribution functions for many common probability distributions are sigmoidal. One such example is the error function, which is related to the cumulative distribution function (CDF) of a normal distribution.

Many natural processes, such as those of complex system learning curves, exhibit a progression from small beginnings that accelerates and approaches a climax over time. When a specific mathematical model is lacking, a sigmoid function is often used.[2]

 

A wide variety of sigmoid functions have been used as the activation function of artificial neurons, including the logistic and hyperbolic tangent functions.

 

 

Activation Functions

The activation ops provide different types of nonlinearities for use in neural networks. These include smooth nonlinearities (sigmoid, tanh, elu, softplus, and softsign), continuous but not everywhere differentiable functions (relu, relu6, crelu and relu_x), and random regularization (dropout).

All activation ops apply componentwise, and produce a tensor of the same shape as the input tensor.

posted @ 2017-09-19 14:27  papering  阅读(564)  评论(0)  编辑  收藏  举报