An activation function is a decision making function that determines the presence of particular neural feature. Non-linearity is needed in activation functions because its aim in a neural network is to produce a nonlinear decision boundary via non-linear combinations of the weight and inputs.
What is ReLU activation function?
In the context of artificial neural networks, the rectifier is an activation function defined as the positive part of its argument: , The rectifier is, as of 2018, the most popular activation function for deep neural networks. A unit employing the rectifier is also called a rectified linear unit (ReLU).
What does a neuron compute activation function?
Neuron Output, y: The artificial neuron computes its output according to the equation shown below. This is the result of applying the activation function to the weighted sum of the inputs, less the threshold. This value can be discrete or real depending on the activation function used.