Artificial neural networks - a short introduction and a simple example

First we have to explain the exclusive OR operation. The exclusive OR (XOR) operation is a logical operator that results in true if exactly one of its operands has the value true. It can be described in Boolean algebra as follows:

p	q	p xor q
1	1	0
1	0	1
0	1	1
0	0	0

And what is an artificial neural network (ANN)? ANN is a mathematical model that tries to simulate the function and structure of the neural network in our brain. ANNs consists of a number of simple processors that are called neurons. The neurons are connected by weighted links that passes the signals from one neuron to another. The neuron can have many incoming signals with different values but there is always just one outgoing signal, an outgoing signal that can split into many but they all have the same value. The connections between the neurons are weighted, they have a numerical weight. The weight represents the strength and importance of the neurons input signal. And it is by adjusting the weights numerical value, through a number of iterations, that the ANN is able to learn. A neuron is a very simple processing unit that only has the means to compute its activation level based on the inputs and its numerical weights. The neurons also have an activation function and a threshold value. It computes the weighted sum of the input signals and compares it with a threshold value. If for example the neurons activation function is a sign function then the neurons output is -1 if the net input is less than the threshold, and +1 if the net input is greater than or equal to the threshold value. There are different types of activation functions, the step function, the sign function, the sigmoid function and the linear function.

An ANNs learning process can be divided into four different stages, initiation, activation, weight training and iteration (go back to second stage activation).  One iteration is called an epoch.

An ANN can have many layers of neurons. The two outer layers that are connected to the external environment are called input and output layer and between them there are sometimes hidden middle layers, se picture below. In our case we have to use a multi layered ANN because the XOR operation is not a linear separable function. (Single layered ANNs can only handle linear separable functions.)

Let’s now look at our example program. The following screenshot is from the MATLAB program “XOR_bp.m”:

The program uses an ANN to learn the Exclusive OR function. This ANN consists of four layers, one input layer, two middle layers and one output layer. The first middle layer uses a sigmoid function and the second middle layer a linear function. The letter W stands for a weighted connection and the letter b stands for bias. The ANN uses back-propagation which means that it sends error signals backwards through the network. An error signal occurs when the output signal differs from the desired. (Because of that the back-propagating ANN does not simulate the human brain.) When the error signal travels backwards through the ANN it adjusts the weights. An indicator of the ANNs performance are the sum of squared errors. In our example the program has set the goal of the sum of squared errors to 0.001. If it reaches that goal the ANN have converged. In the screenshot above we can see that we reached that goal in 199 iterations........ ...