Mathematical Functions

aka ELU, Exponential Linear Unit

A mathematical function that is x if x > 0 and alpha * (exp(x) - 1) if x < 0 where alpha controls the value to which an ELU saturates for ne…

A mathematical function denoted by f(x)=exp or e^{x}.

aka GELU, Gaussian Error Linear Unit

A mathematical function that computes x * P(X <= x) where P(X) ~ N(0 1) weighting inputs by their value rather than gating inputs by their s…

A mathematical function that is a faster approximation of the sigmoid activation using a piecewise linear approximation.

A mathematical function that has the form f(x) = a + bx.

A mathematical rule that gives the value of a dependent variable corresponding to specified values of one or more independent variables.

aka ReLU, Rectified Linear Unit

A mathematical function that returns max(x 0) the element-wise maximum of 0 and the input tensor.

aka SELU, Scaled Exponential Linear Unit

A mathematical function that multiplies scale (> 1) with the output of the ELU function to ensure a slope larger than one for positive input…

aka tore

A mathematical function that applies the sigmoid activation function sigmoid(x) = 1 / (1 + exp(-x)) always returning a value between 0 and 1…

A mathematical function where the elements of the output vector are in range (0 1) and sum to 1 and each vector is handled independently.

A mathematical function that is softplus(x) = log(exp(x) + 1).

A mathematical function that is softsign(x) = x / (abs(x) + 1).

A mathematical function that is x*sigmoid(x) a smooth non-monotonic function that consistently matches or outperforms ReLU on deep networks.

aka hyperbolic tangent

A mathematical function that is the hyperbolic tangent activation function.