Sigmoid activation function

In general, a sigmoid function is real-valued and differentiable, having a non-negative or non-positive first derivative, one local minimum, and one local maximum.

Sigmoid functions are often used in artificial neural networks to introduce nonlinearity in the model.

A neural network element computes a linear combination of its input signals, and applies a sigmoid function to the result. A reason for its popularity in neural networks is because the sigmoid function satisfies a property between the derivative and itself such that it is computationally easy to perform.

Derivatives of the sigmoid function are usually employed in learning algorithms

REF: https://excel.ucf.edu/classes/2007/Spring/appsII/Chapter1.pdf
Sigmoid function produces similar results to step function in that the output is between 0 and 1. The curve crosses 0.5 at z=0, which we can set up rules for the activation function, such as: If the sigmoid neuron’s output is larger than or equal to 0.5, it outputs 1; if the output is smaller than 0.5, it outputs 0.

Sigmoid function does not have a jerk on its curve. It is smooth and it has a very nice and simple derivative of σ(z) * (1-σ(z)), which is differentiable everywhere on the curve.

REF: https://towardsdatascience.com/multi-layer-neural-networks-with-sigmoid-function-deep-learning-for-rookies-2-bf464f09eb7f
Many natural processes, such as those of complex system learning curves, exhibit a progression from small beginnings that accelerates and approaches a climax over time. When a specific mathematical model is lacking, a sigmoid function is often used.

REF: https://en.wikipedia.org/wiki/Sigmoid_function
The following is Python code to implement a sigmoid activation function

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.e**(-x))

x = np.linspace(-10, 10, 1000)

y = sigmoid(x)

plt.xlabel('$x$', fontsize=22); plt.ylabel('$y  = 1 / (1 + np.e**(-x))$', fontsize=22)
plt.plot(x, y,label = 'Sigmoid curve')
plt.legend(loc='upper left')
plt.plot(0.0, sigmoid(0), 'r.')
plt.yticks([0, 0.25, 0.5, 0.75, 1])
plt.show()  

REF: https://datascience.stackexchange.com/questions/30676/role-derivative-of-sigmoid-function-in-neural-networks