Deep learning visualizer

Activation Functions Visualizer

Explore how activation functions transform neuron values.

TypeNon-linear TransformUsed inHidden & Output LayersInputxOutputf(x)

Activation Function

Activation Function: ReLU

ReLUSigmoidTanhLeaky ReLUSoftmaxGELU
xf(x)-5-3-1135-3-2-10123f(1.50) = 1.50Derivative d/dxx1

Neuron Example (using ReLU)

Inputs

x10.60x1.20x2-0.40x-0.80x31.00x0.46
Weighted sumz = sum(w_i*x_i) + bz = 1.52
Activation (ReLU)a = f(z)a = 1.52
Next layer input1.52Passed forward

Activation Steps

  1. Compute weighted sum z = sum(w_i * x_i) + b
  2. Apply activation function to weighted sum
  3. Pass output to the next layer
  4. Optionally compute derivative
  5. Continue forward or backward pass

Activation Functions Summary

FunctionDescriptionFormulaRangeCommon use
ReLUFast default activation for deep neural networks.max(0, x)[0, infinity)Hidden layers
SigmoidUseful for binary classification outputs.1 / (1 + e^-x)(0, 1)Binary output
TanhCommon in recurrent and older dense networks.tanh(x)(-1, 1)RNNs, hidden layers
Leaky ReLUA ReLU variant that avoids completely zero gradients.max(0.1x, x)(-infinity, infinity)Deep nets
SoftmaxUsed in the output layer for multi-class classification.e^x_i / sum(e^x_j)(0, 1)Multi-class output
GELUPopular in transformers and modern NLP architectures.x * Phi(x)(-infinity, infinity)Transformers