Hybrid intelligent systems and evolutionary neural networks

A hybrid intelligent system is a system that uses two or more different artificial intelligence techniques. Hybrid intelligent systems are often based on something called soft computing. Systems based on soft computing can handle imprecise and uncertain information, much like the human mind. And like the human mind soft computing deals with words, not numbers. “Hard” computing on the other hand uses crisp values like binary ones and zeroes. Each of the different intelligent techniques has their weaknesses and strengths and by combining them we can build a system, a hybrid intelligent system, which is more powerful and has fewer weaknesses. And the four main techniques that, in different combination, are used when creating hybrid intelligent systems are expert systems, fuzzy systems, neural networks and genetic algorithm. They all have their strength and weaknesses. Fuzzy logic can handle imprecision, neural networks can be trained and genetic algorithm are good at optimisation.

One example of a hybrid intelligent system is the neural expert system which combines neural networks and rule based expert systems. An expert system can not learn, but if we use a neural network as its knowledge base, then we get a neural expert system that is capable of handling noisy and incomplete data. This ability is called approximate reasoning. The knowledge in a neural expert system is stored in the synaptic weights.  But expert neural systems are not perfect. One problem with neural expert systems are that they uses Boolean logic which means that continuous input data may lead to infinite number of rules.

A neuro-fuzzy system is a hybrid intelligent system that combines fuzzy logic and neural networks. Fuzzy logic deals with reasoning on a higher level using linguistic information from human experts. Neural networks on the other hand deals with raw data on a lower level. So when we combine them we get a system, a neuro-fuzzy system, that…

…is capable of learning.

…is good at parallel computing.

…represent knowledge in a human like way.

…have explanation abilities.

Another example of a hybrid intelligent system is the evolutionary neural network. They use both neural networks and evolutionary computing (e. g. genetic algorithms).Ordinary neural network has one big problem in that they can not guarantee an optimal solution. And that is because the back-propagation algorithm might converge to a set of weights that is not optimal. Another problem with neural networks is that it often can be a hard task to find the optimal topology. But genetic engineering, in combination with neural networks, can help us with optimisation and topology selection. The key steps when building an evolutionary neural network (that uses genetic algorithms) are as follows:

• Represent the problem as a chromosome. If we have a multilayered feedforward neural network we first assign random initial weights. Then we create a matrix with the from neuron number on the x-axis and the to neuron number on the y-axis. In the matrix we then place the corresponding weights. We use zeroes where there is no connection. Finally we extract the numbers (except the zeroes) in the matrix, from top left to right bottom, and place them in a chromosome.
  
  
• Define the fitness function. The fitness function is used for evaluating the chromosomes performance. Here we can use the sum of squared errors as a measurement of the chromosomes fitness. We feed the network with training data and calculate the squared sum. The smaller sum the fitter the chromosome is.
  
  
• Choose the genetic operators, crossover and mutation. The mutation operator chooses a random gene and adds a small random number to each of its weights. The crossover operator chooses two chromosomes, parents, and creates a new one, a child, based on materials from both the parents.
  
  
• Apply the genetic algorithm. Define the population size, the crossover and mutation probabilities and the number of generations.
  
  

With the description above we have a way of optimising the weights in our neural network by evolutionary learning. But we still have to define the topology of our neural network, and that can sometimes be hard! Genetic algorithm can help us here too. With a genetic search in a population of possible topologies we should be able to find a solution. There are several methods of encoding a topology into a chromosome. One of the simplest methods is to convert the network topology into a matrix with from neurons on the x-axis and to neurons on the y-axis. Fill the matrix with ones (when there is a link) and zeroes (when there is no link). Create the chromosome by reading the matrix. This method only works with feedforward networks. Now we can use a genetic algorithm to find a solution for our network topology. Here is the workflow for a genetic algorithm:

1. Define the number of iterations. Choose the population size and the probabilities for mutation and crossover.
2. Define a fitness function. A fitness function measures the chromosomes performance. We use the method of squared sums here.
3. Generate a random population of chromosomes.
4. Decode each of the chromosomes in the population into a neural network. Set initial weights and then train each of the neural networks. Calculate the fitness (squared sum of errors) of each network.
5. Pick a pair of chromosomes that we will apply genetic operators on. They should be picked according to their fitness. Higher fitness – higher probability.
6. Apply genetic operators. Crossover swaps rows between the two chromosomes. Mutation flips bits in the chromosome. There is usually a low probability for mutation (e. g. 0.005).
7. Add the offspring chromosomes (created in step 6) in the new population.
8. Go to step 5 and repeat until the new population equals the size of the initial population.
9. Replace the initial chromosome population with the new one.
10. Go to step 4 and repeat until the specified number of iterations (generations) has been executed.

And we can also use evolutionary computing in combination with fuzzy logic. If we do we get a fuzzy evolutionary system. Here evolutionary computing is primary used for creating fuzzy rules and adjusting membership function of fuzzy sets.