Natural Computing with Python: Learn to implement genetic and evolutionary algorithms to solve problems in a pythonic way

CHEPTER 1

Neural

Networks

Introduction

The history of the Neural Networks has its origins in the years the idea of neural networks learning. Between 1957 and 1958, however, Rosenblatt proposed the first true modern neural network scheme, that is, the perceptron able to recognize shapes and associate configurations.

The perceptron (described in the Perceptron section) exceeds the limitations of the binary structure proposed by McCulloch and Pitts, because it is equipped with variable synaptic weights, which are then able to learn.

Until the 1970s–80s, which is until the advent of modern computers, neural networks fell into the general disinterest of the scientific community, which recalls contact with the work of Werbos in 1974, which describes the mathematical bases for the training of multi-layer neural networks.

The most modern neural network structure is reached in 1986, when Rumelhart, Hilton, and Williams described the training algorithm for backpropagation of the error (introduced in MLP Python implementation section), thanks to which it is possible to modify the weights of the neuronal connections in a systematic way, as long as the response of the network does not become the same—or as close as possible—to the desired one.

A neural network actually presents itself as an adaptive system capable of modifying its structure (nodes and interconnections) based both on external data and internal information that connects and passes through the neural network during the learning phase and reasoning.

For this purpose, at the end of the chapter, the TensorFlow software library for machine learning is introduced, which provides tested and optimized modules useful in the implementation of algorithms for different types of tasks from language processing to image recognition.

Structure

Perceptron

Developing logic gates by perceptron

Activation functions

Linear and non-linear models

Sigmoid neuron

How neural networks learn

Neural network architecture

Supervised learning

Gradient descent

MLP Python implementation

TensorFlow

Logistic regression

Conclusion

Sitography

NOTE: The examples presented in this book were made for a Python 3.x version. However, as a matter of compatibility with all the examples that will be described in the following chapters, it is recommended to use Python 3.5.2 version, downloadable at the following link: https://www.python.org/downloads/release/python-352/

Biological neuron

Biological neurons are electrically active cells, and the human brain contains about 1011 neurons. They exist in different forms, although most of them have the shape shown in Figure 1.1. The main cell body is the soma; the dendrites represent the inputs of the neuron and the axon represents its output. Communication between neurons occurs at the junctions, called synapses.

Each neuron is typically connected to a thousand other neurons and, consequently, the number of synapses in the brain exceeds 1014. Each neuron can be found mainly in 2 states: active or rest. When the neuron is activated, it produces an action potential (electrical impulse) that is transported along the axon. Once the signal reaches the synapse, it causes the release of chemicals (neurotransmitters) that cross the junction and enter the body of other neurons.

Depending on the type of synapses, which can be exciters or inhibitors, these substances increase or decrease respectively the probability that the next neuron becomes active. At each synapse, a weight is associated which determines the type and extent of the exciter or inhibitor effect. Following is the diagram of a biological neuron:

Figure 1.1 : Biological neuron representation

So, in a nutshell, each neuron makes a weighted sum of the inputs coming from the other neurons and, if this sum exceeds a certain threshold, the neuron is activated.

Each neuron, operating at a time of millisecond, represents a relatively slow processing system; however, the entire network has a very large number of neurons and synapses that can operate in parallel and simultaneously, making the actual processing power very high.

Perceptron

One type of artificial neuron is the perceptron, developed between 1950s and 60 by the scientist Frank Rosenblatt. A perceptron takes in input different binary values x1, x2, .., xn and produces a single binary value of output y:

Figure 1.2 : Perceptron

Figure 1.2 shows a perceptron, which can have n input values x1, x2, ..., xn and a single output value y. Rosenblatt, in 1958, proposed simple rules for evaluating the output value, introducing the weights and matching each input value xj , a weight wj, a real value that represents the importance of input xj in the calculation of the value of exit y.

Figure 1.3 : Rosenblatt’s Perceptron

The output of the perceptron is a binary value and can only assume the values 0 and 1, depending on whether the sum of the inputs weighs less than or greater than a certain threshold value, parameter of the neuron that can assume any real value:

Here, the function f (x) is the so-called activation function. The previous equation shows the mathematical model of a perceptron, which can also be seen as a binary decision-maker that takes its decisions by weighing inputs. If the weight of an input xj is greater than the others, this value will be more closely considered in the weighted sum, whereas if the threshold value is decreased, it is more likely that the output of the neuron is 1 and vice versa.

Alternatively, to the threshold, the following relation can be considered:

Here the activation function is

We can think of bias as a measure of how easy it is to get an output of the perceptron equal to 1, it requires much less effort to activate a neuron having a very high bias and vice versa. If the bias is much less than zero then it would be very difficult to make the output of the perceptron equal to 1. In the following, we will always use the b value of the bias and no longer the threshold value, this is only a small change of notation but as you will see it will make further simplifications possible.

In summary, the perceptron has the following structure:

Figure 1.4 : Complete structure of a perceptron

Inputs are fed into the perceptron.

Weights are multiplied to each input.

Summation of the output and then addition of Bias.

Activation function is applied. Note that here a step function is used, but there are other more sophisticated activation functions such as sigmoid, hyperbolic tangent (tanh), rectifier (relu) and so on.

Output is either triggered as 1, or not, as 0. Note we use ŷ hat to label output produced by our perceptron model.

Developing logic gates by perceptron

So far, a method has been described to evaluate the results in order to make decisions, while another way in which the earners can be used is to calculate elementary logic functions, such as AND, OR, or NAND.

For example, suppose you have a perceptron with two inputs, each with weight 1 and one bias equal to −2 as shown in the following figure:

Figure 1.5 : Neural network AND gate

If you try to set the different values for x1 and x2 and analyze the value of the output y:

It can be seen that the perceptron behaves exactly like a logical AND gate.

If we had used a value of pair 2 for both w1 and w2 and a bias of 3, we would have implemented a NAND logic port and in fact, thanks to De Morgan’s laws related to the Boolean logic, we could use networks of perceptrons to compute any logical function.

The computational universality of the perceptrons could be reassuring and disappointing at the same time; it may be reassuring that this neuron looks as powerful as any computational device, but it may be disappointing to think that the perceptrons are nothing but a new type of NAND gate, not to mention the fact that we should design and set the weights and deviations manually, which in case of large networks would be an expensive process. Fortunately, there are the so-called learning algorithms that allow the automatic adjustment of weights and deviations of a network of artificial neurons.

These algorithms allow the use of artificial neurons in a completely different way from the logic gates. A neural network can simply learn to solve the problem, sometimes too complex to solve it through the explicit design of a network of logical gates.

Figure 1.6 shows a perceptron where x1, x2 are input signals, y is an output signal, w0 is a bias, and w1, w2 are weights:

Figure 1.6 : Logical gate neural network

The output is 1, only if the sum of inputs is over thresholds. In this case, the activation function is represented as follows:

Let us build a logic gate with this function. If w0=−1.5, w1=1, and w2=1, it will be the AND gate.

This is the truth table for AND, OR, and NAND:

Table 1.1 : AND OR NAND truth tables

Let us implement it with the following Python code:

Activation functions

In biological neurons, the action potential is transmitted in full once the potential difference to the membranes exceeds a certain threshold. In a certain sense, this is also true for artificial neurons, in particular to an artificial neuron. It calculates a linear function of its inputs and applies a non-linear function, called the activation function, to the result.

What would happen if the activation function were not applied? The neural network as a whole would be the composition of linear functions, which in turn is a linear function. In other words, a neural network of 1000 linear layers would have the same representative capacity as a network with a single linear layer. The activation function therefore serves to break the linearity and make the network a non-linear function.

In the next section, we will explain the difference between linear and non-linear models, and how the use of these models (or simply functions) is crucial for the implementation of artificial neural networks.

Linear and non-linear models

A neural network without an activation function is simply a regression model, that is, it tries to access data distribution with a straight line (see Figure 1.7). In this example, we can see that the line represents the distribution in a rather imprecise way. The level would always be the same as the previous one:

Figure 1.7 : Linear regression

The purpose of neural networks is to be a universal function approximator, that is, to be able to approximate any function, and to do this, it is necessary to introduce a factor of non-linearity, here which is the introduction of the activation functions.

Figure 1.8 : Non-linear regression

In the preceding graph, with a non-linear model it is possible to approximate the same data much more precisely. Thus, to be usable, an activation function must be non-linear.

The three most used activation functions are the step function, the sigmoid, and the ReLU function, which is described in the following sections.

Step function

The most intuitive function is the step function, in a sense, more similar to biological functioning. For all negative values, the answer remains 0, while it jumps to +1 as soon as the value reaches or even exceeds zero. The advantage is that it is simple to calculate, and it normalizes the output values by compressing them all in a range between 0 and +1.

However, this type of function is not really used because it is not differentiable at the point where it changes value (that is, there are no derivatives at that point). The derivative is nothing more than the slope of the tangent line at that point, and it is fundamental in deep learning, as it determines the direction in which to orientate for adjustments to values.

In short, we can say that this abrupt change of state makes it difficult to control the behavior of the network. A small change on a weight could improve the behavior for a given input but make it jump completely for other similar ones.

Sigmoid function

Sigmoid function is the most used activation function It has similarities with the step function, but the transition from 0 to +1 is more gradual, with an s-shaped trend. The advantage of this function, besides being differentiable, is to compress the values in a range between 0 and 1 and therefore be very stable even for large variations in values. The sigmoid has been used for a long time, but it still has its problems.

It is a function that has a very slow convergence, given that for very large input values the curve is almost flat, with the consequence that the derivative tends to zero. This poor responsiveness toward the ends of the curve tends to cause problems of vanishing gradient, which we will discuss later. Also not being zero-centered, the values in each learning step can only be all positive or all negative, which slows down the training process of the network.

This is a function that is no longer widely used in the intermediate layers, but still very valid in output for categorization tasks.

ReLU function

The ReLU (rectifier linear unit) function is a function that has become very popular lately, especially in intermediate layers. The reason is that it is a very simple function to calculate: it flattens the response to all negative values to zero, while leaving everything unchanged for values equal to or greater than zero.

This simplicity, combined with drastically reducing the problem of vanishing gradient, makes it a particularly attractive feature in intermediate layers, where the number of steps and calculations is important. In fact, calculating the derivative is very simple: for all the negative values it is equal to zero, while for the positive ones it is equal to 1. At the angular point in the origin instead the derivative is indefinite but is nevertheless set to zero by convention.

We can plot them, by the following Python code:

import numpy as np

import matplotlib.pylab as plt

def step(x):

return np.array(x > 0, dtype=np.int)

def sigmoid(x):

return 1 / (1 + np.exp(-x))

def relu(x):

return np.maximum(0, x)

x = np.arange(-5.0, 5.0, 0.1)

y_step = step(x)

y_sigmoid = sigmoid(x)

y_relu = relu(x)

plt.plot(x, y_step, label=’Step’, color=’k’, lw=1, linestyle=None)

plt.plot(x, y_sigmoid, label=’Sigmoid’, color=’k’, lw=1, ls=’--’)

plt.plot(x, y_relu, label=’ReLU’, color=’k’, lw=1, linestyle=’-.’)

plt.ylim(-0.1, 1.1) plt.legend()

Here, is the output for the three functions, which is coded above:

Figure 1.9 : ReLu, step, and sigmoid function

Sigmoid neuron

The sigmoidal neuron is represented in the same way as the perceptron, it can have an input x1, x2, ..., xn, the corresponding weights w1, w2, ...,wn, the