7.1 Introduction to Neural Networks

What Is a Neural Network?

A neural network is a structure made up of components called neurons, which are individual decision-making units that take some number of inputs $x_1,x_2,\dots ,x_n$ and produce an output $y$, as in Figure 7.2. How the output is determined by the inputs could be quite complex. Moreover, any two neurons may behave quite differently from each other on the same inputs.

Figure 7.2 Single Neuron in a Neural Network

The neural network itself may consist of hundreds, thousands, or even millions of neurons connected to each other in layers, or groups of neurons that all receive the same inputs from previous layers and forward signals in aggregate to the next layer. There are always at least two layers, the input layer (containing neurons that accept the initial input data) and output layer (containing the neurons that are used to interpret the answer or give classification information), together with some number of hidden layers (layers between the input and output layers). Figure 7.3 shows a simple neural network diagram for reference.

Figure 7.3 Neural Network Diagram. This neural network has four layers, two of which are hidden layers. There are three input neurons and one output neuron. In this example, all nodes in adjacent layers are connected, but some neural network models may not include all such connections (see, for example, convolutional neural networks in Introduction to Deep Learning).

The main purpose of a neural network is to classify complex data. Problems for which neural networks are especially well suited include the following:

• Image recognition, including facial recognition, identifying handwritten letters and symbols, and classifying parts of images. This is a huge area of innovation, with powerful tools such as TensorFlow developed by Google and PyTorch developed by Meta.
• Speech recognition, such as Google’s Cloud Speech-to-Text service, offers accurate transcription of speech and translation for various languages.
• Recommendation systems, which are used to serve ads online based on each user’s browsing habits, have been developed and used by Amazon, Meta, Netflix, and many other large companies to reach target markets.
• Anomaly detection has been developed to aid in fraud detection, data security, and error analysis/correction by finding outliers in large, complex datasets. An example is Microsoft’s Azure Anomaly Detector, which can detect anomalies in time series data.
• Autonomous vehicles and robotics, including Tesla’s Autopilot technology, are becoming more and more prominent as automation alleviates some of the routine aspects of daily life and leads to increased efficiencies in business, manufacturing, and transportation.
• Generative art, examples of which include visual art, music, video, and poetry, leverage vast stores of human creative output to produce novel variations.
• Predictive text, including natural language processing models such as ChatGPT (see Convolutional Neural Networks).

Neurons, Weights, Biases, and Activation Functions

The way neurons work in a neural network is conjectured to be similar, or analogous, to how neurons work in the brain. The input of the neural network is fed into the neurons of the input layer. Each neuron then processes it and produces an output, which is in turn pushed to the neurons in the next layer. Individual neurons only send signals, or activate, if they receive the appropriate input required to activate. (Activation is the process of sending an output signal after having received appropriate input signals.) After passing through some number of hidden layers, the output of the last hidden layer feeds into the output layer. Lastly, the output of this final layer is interpreted based on the nature of the problem. If there is a single output neuron, then the interpretation could be true if that neuron is activated or false if not. For neural networks used to classify input into various classes (e.g., number recognition), there is usually one output neuron per class. The one that is most activated would indicate the classification, as shown in Figure 7.4.

Figure 7.4 Output Neurons. In this figure, there are four output neurons, labeled A, B, C, and D. Since B has the highest activation level, the output of the neural network is B.

Each connection from one neuron to another has two parameters associated with it. The weight is a value $w$ that is multiplied to the incoming signal, essentially determining the strength of the connection. The bias is a value $b$ that is added to the weighted signal, making the neuron more likely (or less likely, if $b$ is negative) to activate on any given input. The values of $w$ and $b$ may be positive, negative, or zero. Once the weight and bias are applied, the result is run through an activation function, which is a non-decreasing function $f$ that determines whether the neuron activates and, if so, how strongly. (You’ll see this later in Figure 7.6.)

Consider the simplest case of all, a neuron with single input $x$ and output $y$. Then the flow of signal through the neuron follows the formula $y=f(wx+b)$. For example, suppose that the input value is $x=0.87$, with weight $w=0.53$ and bias $b=-0.12$. Then the signal would first be combined as $wx+b=\left(0.53\right)\left(0.87\right)+(-0.12)=0.3411$. This value would be fed as input to the activation function to produce $y=f\left(0.3411\right)$. In the next section, we will discuss various activation functions used in neural networks.

When there are multiple inputs, $x_1,x_2,\dots ,x_n$, each will be affected by its own weight. Typically, bias is thought of as a property of the neuron itself and so bias affects all the inputs in the same way. To be more precise, first, the inputs are multiplied by their individual weights, the result is summed, and then the bias is added. Finally, the activation function is applied to obtain the output signal.

$$y=f(w_1{x}_1+w_2{x}_2+\dots +w_n{x}_n+b)=f(\left(\sum _{i=1}^n{w}_i{x}_i\right)+b)$$

The weights and biases are parameters that the neural network learns during the training process, a topic we will explain in Backpropagation.

A typical neural network may have hundreds or thousands of inputs for each neuron, and so the equation can be difficult to work with. It would be more convenient to regard all the inputs $x_1,x_2,\dots ,x_n$ as parts of a single mathematical structure, called a vector. A vector is simply an ordered list of numbers, that is, ${x=(x}_1,x_2,\dots ,x_n)$. (Note: In this text we use boldface letters to represent vectors. Also, the components of a vector are listed within a set of parentheses. Some texts vary on these notational details.) The number of components in a vector is called its dimension, so for example the vector $(5,0,-2,1.2,\pi )$ has dimension 5. Certain arithmetic operations are defined on vectors.

• Vectors of the same dimension can be added: If $x={(x}_1,x_2,\dots ,x_n)$ and ${y=(y}_1,y_2,\dots ,y_n)$, then ${x+y=(x}_1+y_1,x_2+y_2,\dots ,x_n+y_n)$.
• Any real number can be multiplied to a vector: ${kx=k(x}_1,x_2,\dots ,x_n)={(kx}_1,k{x}_2,\dots ,k{x}_n)$.
• The dot product of two vectors of the same dimension results in a real number (not a vector) and is defined by:
  
  $$\begin{array}{rcl}x·y& =& {(x}_1,x_2,\dots ,x_n)·{(y}_1,y_2,\dots ,y_n)=\sum _{i=1}^n{x}_i{y}_i\\ & =& x_1{y}_1+x_2{y}_2+x_3{y}_3+\dots +x_n{y}_n\end{array}$$

If the inputs and weights are regarded as vectors, $x={(x}_1,x_2,\dots ,x_n)$ and $w={(w}_1,w_2,\dots ,w_n)$, respectively, then the formula may be re-expressed more concisely as:

$$y=f(w·x+b)$$

For example, if $w=(0.3,-0.1,0.9)$, $x=(1.2,0.4,-0.6)$, and $b=0.1$, then

$$w·x+b=(0.3\left)\right(1.2)+(-0.1)\left(0.4\right)+\left(0.9\right)(-0.6)+0.1=-0.12$$

So in this example, the output would be $y=f(-0.12)$, the exact value of which depends on which activation function $f\left(x\right)$ is chosen.

Types of Activation Functions

Activation functions come in many types. Here are just a few of the most common activation functions.

1. Step function, $f\left(x\right)=\{\begin{array}{l}0,\mathrm{if}x<c\\ 1,\mathrm{if}x\ge c\end{array}$. The value of $c$ serves as a threshold. The neuron only activates when the input is at least equal to a parameter $c$.
2. Sigmoid function, $\sigma \left(x\right)=\frac{1}{1+e^{-x}}$. Note, this is the same sigmoid function used in logistic regression (see Classification Using Machine Learning). Output values tend to be close to 0 when $x$ is negative and close to 1 when $x$ is positive, with a smooth transition in between.
3. Hyperbolic tangent (tanh) function, $\text{tanh }x=\frac{e^x-e^{-x}}{e^x+e^{-x}}$. Output values have the same sign as $x$, with a smooth transition through 0.
4. Rectified linear unit (ReLU) function, $\text{ReLU}\left(x\right)=\text{max}(0,x)$. The ReLU function is 0 for negative $x$ values and equal to the input when $x$ is positive.
5. Leaky ReLU function, $\text{LReLU}\left(x\right)=\text{max}(cx,x)$, for some small positive parameter $c$. Leaky ReLU acts much like ReLU except that the values get progressively more negative when $x$ gets more negative. Often, the optimal “leakiness” parameter $c$ is determined during the training phase.
6. Softplus function, $f\left(x\right)=\text{ln}(1+e^x)$, which is a smoothed version of the ReLU function.

Figure 7.5 shows the graphs of the listed functions. A key feature of activation functions is that they are nonlinear, meaning that they are not just straight lines, $f\left(x\right)=mx+b$.

Another activation function that is important in neural networks, softmax, takes a vector of real number values and yields a vector of values scaled into the interval between 0 and 1, which can be interpreted as discrete probability distribution. (Recall from Discrete and Continuous Probability Distributions that discrete probability distributions provide measures of probability or likelihood that each of a finite number of values might occur.) The formula for softmax is:

$$\sigma (x_1,x_2,\dots ,x_n)=(\frac{e^{x_1}}{D},\frac{e^{x_2}}{D},\dots ,\frac{e^{x_n}}{D})$$

where $D=e^{x_1}+e^{x_2}+\dots +e^{x_n}$. We will encounter softmax activation in an example in Backpropagation.

Figure 7.5 Graphs of Activation Functions. Top, from left to right, (a) step, (b) sigmoid, and (c) hyperbolic tangent (tanh) functions. Bottom, from left to right, (d) ReLU, (e) LReLU, and (f) softplus functions.

Example 7.1

Problem

A simple neuron has four inputs, $x_1,x_2,x_3,x_4$, and one output, $y$. The weights of the four inputs are $w_1=0.34$, $w_2=0.07$, $w_3=0.59$, and $w_4=-0.21$. The bias of the neuron is $b=0.19$. Find the output in each of the following cases.

1. $(x_1,x_2,x_3,x_4)=(1,0,0,1)$. Activation function: ReLU.
2. $(x_1,x_2,x_3,x_4)=(1,0,0,1)$. Activation function: sigmoid.
3. $(x_1,x_2,x_3,x_4)=(1,0,0,1)$. Activation function: step, $f\left(x\right)=\{\begin{array}{l}0,\mathrm{if}x<0.5\\ 1,\mathrm{if}x\ge \mathrm{0.\; 5}\end{array}$.
4. $(x_1,x_2,x_3,x_4)=(0,2.3,-1.6,0.8)$. Activation function: softplus.

Solution

Setup for each part:

$$y=f(w·x+b)=f(w_1{x}_1+w_2{x}_2+w_3{x}_3+w_4{x}_4+b)=f(0.34x_1+0.07x_2+0.59x_3-0.21x_4+0.19)$$

1. $\text{ReLU}\left(0.34\right(1)+0.07(0)+0.59(0)-0.21(1)+0.19)=\text{ReLU}(0.32)=0.32$, since $0.32>0$
2. $\sigma \left(0.32\right)=\frac{1}{1+e^{-0.32}}=0.58$ (Note, 0.32 was already computed in part a.)
3. $f\left(0.32\right)=0$, since $0.32<0.5$ (Note, 0.32 was already computed in part a.)
4. $f\left(0.34\right(0)+0.07\left(2.3\right)+0.59(-1.6)-0.21(0.8)+0.19)=f(-0.761)=\text{ln}(1+e^{-0.761})=-0.761$

In the next section, we explore neural networks that use the simplest of the activation functions, the step function.

Perceptrons

Although the idea of using structures modeled after biological neurons had been around prior to the development of what we would now consider neural networks, a significant breakthrough occurred in the middle of the 20th century. In 1957, Frank Rosenblatt developed the perceptron, a type of single-layer neural network designed for binary classification tasks—in other words, a neural network whose output could be “true” or “false” (see Figure 7.7). After initial successes, progress in AI faced challenges due to limited capabilities of AI programs and the lack of computing power, leading to a period of minimal progress, called the “Winter of AI.” Despite these setbacks, new developments occurred in the 1980s leading to the multilayer perceptron (MLP), which serves as the basic paradigm for neural networks having multiple hidden layers.

The single-layer perceptron learns weights through a simple learning rule, known as the perceptron learning rule.

1. Initialize the weights ($w$) and bias ($b$) to small random values.
2. For each input ($x$) in the training set,
   1. Compute predictions $\hat{y}=f(w·x+b)$, where $f$ is the step function,
      $$f\left(x\right)=\{\begin{array}{l}0,\mathrm{if}x<0\\ 1,\mathrm{if}x\ge 0\end{array}$$
   2. Find the error $E=y-\hat{y}$, where $y$ is the true or desired value corresponding to $x$. Note that $E$ can be positive or negative. Positive $E$ implies the weights and/or bias is too low and needs to be raised. Negative $E$ implies the opposite.
   3. Update the weights and biases according to the following formulas, where $h$ is a small positive constant that controls the learning rate. Often the change of weights and biases will be small, not immediately causing a large effect on input. However, with repeated training, the perceptron will learn the appropriate values of $w$ and $b$ to achieve the desired output.
      
      $$w\leftarrow w+hEx$$
      $$b\leftarrow b+hE$$

For example, suppose a perception with three neurons currently has weights $w=(0.5,0.7,0.1)$ and bias $b=-0.9$. On input $x=(0.6,1,0)$, we get:

Suppose that the true output should have been $y=0$. So there’s an error of $E=0-1=-1$. If the learning rate is $h=0.1$, then the weights and bias will be updated as follows:

On the same input, the perceptron now has a value of:

In this simple example, the value $\hat{y}$ changed from 1 to 0, eliminating the error. However, there is no guarantee that the perceptron will classify all input without error, regardless of the number of training steps that are taken.

Building a Simple Neural Network

In What Is Data Science?, Example 1.7, we looked at the Iris Flower dataset to become familiar with dataset formats and structures. In this section, we will use the Iris dataset, iris.csv, to create a neural network with four inputs and one output neuron (essentially, a simple perceptron) to classify irises as either setosa or not setosa, according to four features: sepal length (SL), sepal width (SW), petal length (PL), and petal width (PW), all of which are measured in centimeters. The Iris dataset iris.csv has 150 samples of irises, consisting of 50 of each species, setosa, versicolor, and virginica. For the purposes of this example, the first 100 rows of data were selected for training and testing, using 75% of the data for training and the remaining 25% for testing. (Recall from Decision-Making Using Machine Learning Basics that it is good practice to use about 70%–80% of the data for training, with the rest used for testing.) The network will be trained using the perceptron learning rule, but instead of using the step function, the activation function will be the hyperbolic tangent, $\text{tanh}x=\frac{e^x-e^{-x}}{e^x+e^{-x}}$, which is chosen simply for illustration purposes.

The model for the neural network is shown in Figure 7.6.

Figure 7.6 Simple Perceptron Neural Network Model. Here, $x_1,x_2,x_3,x_4$ are the inputs, $w_1,w_2,w_3,w_4$ represent the weights, and $b$ represents the bias. The output, or response, is $y$.

The inputs $(x_1,x_2,x_3,x_4)$ will take the values of SL, SW, PL, and PW, respectively. There are four corresponding weights $(w_1,w_2,w_3,w_4)$ and one bias value $b$, which are applied to the input layer values, and then the result is fed into the tanh function to obtain the output $y$. Thus, the formula used to produce the predicted $y$ values, denoted by $\hat{y}$, is:

In this very simplistic model, we will interpret the value of $y$ as follows:

• If $\hat{y}>0$, classify the input as setosa.
• If $\hat{y}<0$, classify the input as not setosa.

As for classifying error, we need to pick concrete target values to compare our outputs with. Let’s consider the target value for classifying setosa as $1$, while the target for classifying not setosa is $-1$. This is reasonable as the values of $\text{tanh}\left(x\right)$ approach $1$ as $x$ increases to positive infinity and $-1$ as $x$ decreases to negative infinity.

For example, if after training, the ideal weights were found to be $(w_1,w_2,w_3,w_4)=(-0.2,0.6,-0.9,0.1)$, and the bias is $b=0.5$, then the response on an input $(x_1,x_2,x_3,x_4)=(5.1,3.5,1.4,0.2)$ would be:

Since $\hat{y}>0$, classify the input as setosa. Of course, the natural question would be, how do we obtain those particular weights and bias values in the first place? This complicated task is best suited for a computer. We will use Python to train the perceptron.

The Python library sklearn.datasets has a function, load_iris, that automatically loads the Iris dataset, so you should not need to import the dataset iris.csv. Other modules that are imported from sklearn are Perceptron, used to build the perceptron model, train_test_split, used to randomly split a dataset into a testing set and training set. Here, we will use a split of 75% train and 25% test, which is handled by the parameter test_size=0.25. We will also renormalize the data so that all features contribute equally to the model. The way to do this in Python is by using StandardScalar, which renormalizes the features of the dataset so that the mean is 0 and standard deviation is 1. Finally, accuracy_score is used to compute the accuracy of the classification.

    
    # import the sklearn library (specific modules)
    from sklearn.datasets import load_iris
    from sklearn.linear_model import Perceptron
    from sklearn.model_selection import train_test_split
    from sklearn.preprocessing import StandardScaler
    from sklearn.metrics import accuracy_score
    
    # Load the Iris dataset
    iris = load_iris()
    X = iris.data[:100] # Only taking first 100 samples for binary classification
    y = iris.target[:100] # "0 for setosa", 1 for "not setosa"
    
    # Split the dataset into training and testing sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)
    
    # Standardize features by removing the mean and scaling to unit variance
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)
    
    # Initialize and train the perceptron
    perceptron = Perceptron()
    perceptron.fit(X_train_scaled, y_train)
    
    # Make predictions on the test set
    y_pred = perceptron.predict(X_test_scaled)
    
    # Calculate accuracy
    accuracy = accuracy_score(y_test, y_pred)
    print("Accuracy:", accuracy)
    

Accuracy: 1.0

Because the dataset was relatively small (100 rows) and well-structured, the perceptron model was able to classify the data with 100% accuracy (“Accuracy: 1.0”). In the real world, when datasets are larger and less regular, we do not expect 100% accuracy. There could also be significant overfitting in the model, which would lead to high variance when classifying data not found in the original dataset.

Here is how to use the perceptron model we just created to classify new data (outside of the original dataset):

    
    # Assume you have new inputs stored in a variable called 'new_inputs'
    new_inputs = [[5.1, 3.5, 1.4, 0.2], # Example input 1
          [6.3, 2.9, 5.6, 1.8] # Example input 2
          ]
    
    # Standardize the new inputs using the same scaler used for training
    new_inputs_scaled = scaler.transform(new_inputs)
    
    # Use the trained perceptron model to make predictions on the new inputs
    predictions = perceptron.predict(new_inputs_scaled)
    
    # Display the predictions
    for i, prediction in enumerate(predictions):
     print(f"Input {i+1} prediction: {'Setosa' if prediction == 0 else 'Not Setosa'}")
    

Input 1 prediction: Setosa
Input 2 prediction: Not Setosa