7.2 Backpropagation

Error and Loss

Error generally refers to the difference between the predicted output of the neural network and the actual target values for a single data point. It is a quantitative measure of the mistakes made by the network in its predictions. The loss function is a mathematical function that takes the predicted output and the actual target values as inputs and outputs a single value representing the error. Learning is essentially minimizing the loss function over all training data. Suppose that the neural network outputs ${\hat{y}=(\hat{y}}_1,{\hat{y}}_2,\dots ,{\hat{y}}_n)$ and the target output is $y=(y_1,y_2,\dots ,y_n)$. Common loss functions that would measure the error between $\hat{y}$ and $y$ include:

• Mean squared error (MSE): $\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$. The formula for MSE (introduced in Training and Testing a Model), is important due to its connection with the concept of variance in statistics. It is a commonly used loss function in regression tasks, where the goal is to predict continuous values.
• Binary cross entropy loss: $-\frac{1}{n}\sum _{i=1}^n[y_i\text{ ln }{\hat{y}}_i+(1-y_i\left)\text{ ln }{(1-\hat{y}}_i)\right]$. This loss function is commonly used in binary classification tasks—that is, when the output is either 0 or 1. (Note: The minus sign in front of the formula is there to make the overall value positive, as the values of the logarithms will generally be negative. The term entropy is related to the measures of information defined in Decision Trees.)
• Hinge loss: $\frac{1}{n}\sum _{i=1}^n\text{max}(0,1-y_i{\hat{y}}_i)$. This function is also commonly useful in binary classification.
• Sparse categorical cross entropy: A generalization of binary cross entropy, useful when the target labels are integers.

Average loss is computed as an average (mean) of the loss over all the data points. The closer to zero the average loss is, the smaller the error in predictions. The choice of loss function depends on the specific task, the nature of the inputs and outputs, and many other factors, which will be discussed in more detail in Introduction to Deep Learning.

Gradient Descent and Backpropagation

Since the goal of training a neural network is to reduce the value of error or loss, we essentially need to solve a (very large) optimization problem. Suppose that you are training a neural network. Consider all the weights and biases—of which there may be thousands, millions, or even more!—as parameters currently in the network that may be changed to improve the model. On a forward pass with an arbitrary input vector $x^{\left(0\right)}$, the output $\hat{y}$ is obtained and compared against the actual output $y$ via a loss function, $C$. How would you be able to use this information to change the parameters (weights and biases) so that on the next pass with the same input $x^{\left(0\right)}$, the output $\hat{y}$ provides a more accurate representation of the actual output $y$, in the sense that the value of $C$ reduces? The key is to regard the whole neural network together with the loss calculation as a function $F$ of the weights $W$ and biases $B$ and use then techniques such as gradient descent (which we will briefly discuss in a following section) to find a minimum value. Here, $W$ and $B$ are capitalized because they represent the multitude of weights and biases over the entire network, not just from a single neuron or layer. Figure 7.7 shows a schematic diagram illustrating the function $F$ as a composite process.

As a very simple example, suppose that a neural network has only one neuron with weight $w$ and bias $b$. Input $x$ is fed into the neuron, the weight and bias are applied to $x$, and then an activation function is applied to the result to produce an output $\hat{y}$. We’ll use the sigmoid, $\sigma \left(x\right)=\frac{1}{1+e^{-x}}$ as activation function for simplicity. Now, the result, $\hat{y}=\sigma (wx+b)$, is compared to the true value $y$, using a cost function, which for this example will be MSE. The composite function $F$ looks like this:

If there were more neurons, weights, and biases, then the formula would look much more complicated as we would need to keep track of many more variables and parameters, but the basic concept remains the same. We want to find values of $w$ and $b$ that minimize the value of $F(w,b)$. It takes many repeated cycles of adjusting weights and biases based on the values of the loss function. Each complete training cycle is called an epoch.

Figure 7.7 Neural Network with Loss Function $C$, Regarded as a Function of Weights and Biases. Given a set of weights $W$ and biases $B$, the neural network is run on a set of input data $x^{\left(0\right)}$, the output $\hat{y}$ is obtained and compared against the actual output $y$ via a loss function, $C$, providing a single number: the loss, $z$. The goal of training is to adjust $W$ and $B$ such that $z$ is minimized.

Now suppose that $w=0.5$ and $b=0.25$. Consider a single point from the set of training data—for example, $(x,y)=(1,0)$. The neural network computes a prediction, $\hat{y}$, as follows:

The loss with respect to the given actual output $(y=0)$ is:

Thus, we have the value $F(0.5,0.25)=0.46$, which indicates that there is some nonzero amount of loss on the prediction just made, and so the values of $w$ and $b$ should be adjusted. Instead of trying to adjust $w$ and $b$ individually, let’s find the direction from the point (0.5, 0.25) that would result in the fastest decrease of the error $z$. If we label the direction of steepest decrease $(dx,dy)$, then we could move from the point (0.5, 0.25) to a new point, $(0.5,0.25)+h·(dw,db)=$ $(0.5+h·dw,0.25+h·db)$, where $h$ is a small number that affects the learning rate of the neural network. This is the main idea behind gradient descent. In our example, we could find the direction of steepest descent by simply looking at how the surface $z=F(w,b)$ is slanted near the point (0.5, 0.25). Figure 7.8 indicates the best direction with an arrow. (However, in practice, there will be no way to visualize the graph of loss with respect to weights $\left(W\right)$ and biases ($B$) because the points $(W,B,z)$ exist in very high-dimensional spaces. The details of gradient descent and how it is used in neural networks falls outside the scope of this textbook. (See this IBM article for a great introduction to this topic.)

Figure 7.8 Simple Application of Gradient Descent. The point $(w,b)=(0.5,0.25)$ has loss of $z=0.46$. The arrow indicates the direction of steepest descent from the point $(0.5,0.25,0.46)$ on the surface.

One final point to mention is that gradient descent only works when the function F is differentiable, meaning that there are no corners, cusps, or points of discontinuity on the graph of $z=F\left(W,B\right)$. Since $F$ is a composite of many component functions, each component needs to be differentiable. Activation functions such as step, ReLU, and LReLU cannot be used with gradient descent. There are other methods available to backpropagate and train those kinds of networks. Alternatively, the non-smooth activation functions could be approximated by smooth versions—for example, sigmoid or tanh in place of step, or softplus in place of ReLU.

Training a Neural Network Using Backpropagation in Python using TensorFlow

The Python library TensorFlow, originally developed by the Google Brain team in 2015, can be used to build a neural network with backpropagation. (A tensor is a multidimensional array, generalizing the concept of vector.)

Let’s set up a neural network and train it to recognize the handwritten letters in the MNIST database. This dataset, called mnist_784, can be loaded automatically using the fetch_openml command. The target labels must be converted to integers explicitly. Then the data is split into training (80%) and testing (20%) sets. The features are renormalized using StandardScaler. Next, the neural network model is created using tf.keras.Sequential. The model can be quite complex, but we will stick to the simplest case—one input layer with exactly as many neurons as features (784) and an output layer with 10 neurons, one for each digit. There are no hidden layers in this model. The command tf.keras.layers.Dense builds the output layer of 10 neurons that is fully connected to the input layer of 784 layers (the value of X_train_scaled.shape[1]). The softmax activation function is used, but other functions like ReLU are also available. The parameter optimizer='adam' in the function model.compile refers to an algorithm, Adaptive Moment Estimation, used to update the weights and biases of the model during training, the details of which are beyond the scope of this text.

Note: This code takes a while to execute—up to 5 minutes! Even with no hidden layers, the backpropagation takes some time to complete. When you run the code, you should see progress bars for each epoch (not shown in the Python output in the feature box).

    # import the libraries
    import tensorflow as tf
    from sklearn.datasets import fetch_openml
    from sklearn.model_selection import train_test_split
    from sklearn.preprocessing import StandardScaler
    
    # Load MNIST Digits dataset
    mnist = fetch_openml('mnist_784', version=1)
    
    # Split the dataset into training and testing sets 80/20
    X_train, X_test, y_train, y_test = train_test_split(mnist.data, mnist.target, test_size=0.2, random_state=42)
    
    # Standardize features by removing the mean and scaling to unit variance
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)
    
    # Cast the target variable as an integer (int32)
    y_train = y_train.astype('int32')
    y_test = y_test.astype('int32')
    
    # Define the neural network architecture
    model = tf.keras.Sequential([
     tf.keras.layers.Dense(10, activation='softmax', input_shape=(X_train_scaled.shape[1],))
    ])
    
    # Compile the model
    model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    
    # Train the model
    history = model.fit(X_train_scaled, y_train, epochs=10, batch_size=32, validation_split=0.2)
    
    # Evaluate the model on test data
    test_loss, test_accuracy = model.evaluate(X_test_scaled, y_test)
    print("Test Loss:", test_loss)
    print("Test Accuracy:", test_accuracy)
    

Test Loss: 0.4051963686943054
Test Accuracy: 0.9189285635948181

Unrolling an RNN

The effect of feedback loops may be visualized by “unrolling” the RNN. Consider the simplest case of a single feedback loop from one neuron to itself. The effect of the connecting weight is equivalent to connecting to a copy of the same RNN (with identical weights and biases). Of course, since the second copy of the RNN has the same feedback loop, it can be unrolled to a third, fourth, fifth copy, etc. An “unrolled” RNN with $n$ copies of the neural net has $n$ inputs. The effect of feeding the entire vector $x^{\left(0\right)}$ into the unrolled model is equivalent to feeding the data points $(x_1,x_2,x_3,\dots ,x_n)$ sequentially into the RNN. Figure 7.10 shows how this works.

Figure 7.10 Unrolling an RNN

Vanishing/Exploding Gradient Problem

It is harder to train an RNN because the model can be very sensitive to changes in the connecting weights. During the training phase, gradient descent and a modified version of backpropagation (BPTT, as mentioned earlier) would be used to adjust all the weights and biases. However, because of the feedback loops in RNN, connecting weights can become compounded many times until they become very large. This causes algorithms like gradient descent to perform very poorly because the high weights cause proportionally high changes in parameters, as opposed to the tiny changes required to find minimum points. This is known as the exploding gradient problem. On the other hand, if connecting weights are too small to begin with, then training can cause them to quickly approach zero, which is called the vanishing gradient problem. Both issues are important to be aware of and addressed when working with RNNs; otherwise, the accuracy of your model may be compromised.

Long Short-Term Memory Networks

A long short-term memory (LSTM) network is a type of RNN designed to overcome the problems of exploding or vanishing gradients by incorporating memory cells that can capture long-term dependencies better than simple feedback loops can. LSTMs were introduced in 1997 and have since become widely used in various applications, including natural language processing, speech recognition, and time series forecasting.

LSTMs are generally more stable and easier to train than traditional RNNs, making them particularly well-suited for forecasting time series data with long-range trends and cyclic behaviors as well as working with sequential data that may seem much more unpredictable like natural language modeling, machine translation, and sentiment analysis. RNNs and LSTMs are a steppingstone to very sophisticated AI models, which we will discuss in the next section.

RNNs in Python

Consider the dataset MonthlyCoalConsumption.csv, which we analyzed using basic time series methods in Time Series and Forecasting. The dataset contains observations up to the end of 2022. Suppose you want to predict the monthly coal consumption in each month of the next year using a recurrent neural network. First, we will set up a simple RNN in Python. The library TensorFlow can be used to build an RNN. We will use pandas to load in the data and numpy to put the data in a more usable form (np.array) for the model. The RNN model is defined by the command tf.keras.layers.SimpleRNN, where units specifies the number of neurons in the layer and activation='tanh' is the activation function.

    import numpy as np
    import pandas as pd
    import tensorflow as tf
    from sklearn.preprocessing import MinMaxScaler
    
    # Load and preprocess the data
    data = pd.read_csv('MonthlyCoalConsumption.csv')
    values = np.array(data['Value'])
    scaler = MinMaxScaler(feature_range=(0, 1))
    scaled_values = scaler.fit_transform(values.reshape(-1, 1))
    
    # Prepare input sequences and target values
    window_size = 12 # Number of months in each input sequence
    X = []
    y = []
    for i in range(len(scaled_values) - window_size):
      X.append(scaled_values[i:i+window_size])
      y.append(scaled_values[i+window_size])
    X = np.array(X)
    y = np.array(y)
    
    # Split data into training and testing sets
    split = int(0.8 * len(X))
    X_train, X_test = X[:split], X[split:]
    y_train, y_test = y[:split], y[split:]
    
    # Define the RNN architecture
    model = tf.keras.Sequential([
      tf.keras.layers.SimpleRNN(units=32, activation='tanh',
                    input_shape=(window_size, 1)),
      tf.keras.layers.Dense(1)
    ])
    
    # Compile the model
    model.compile(optimizer='adam', loss='mean_squared_error')
    
    # Train the model
    model.fit(X_train, y_train, epochs=50, batch_size=16)
    
    # Evaluate the model
    loss = model.evaluate(X_test, y_test)
    print("Test Loss:", loss)
    

Test Loss: 0.018054470419883728

The test loss is very low, and so we expect the model to be very accurate. Next, the model can be used to predict the unknown values in the next 12 months. Note: Since the original data was rescaled, the predictions made by the RNN need to be converted back into unnormalized data using the command scaler.inverse_transform.

    # Predict future values
    future_months = 12
    last_window = scaled_values[-window_size:].reshape(1, window_size, 1)
    predicted_values = []
    for _ in range(future_months):
     prediction = model.predict(last_window)
     predicted_values.append(prediction[0, 0])
     last_window = np.append(last_window[:, 1:, :], prediction.reshape(1, 1, 1), axis=1)
    
    # Inverse transform the predicted births to get actual values
    predicted_values = scaler.inverse_transform(np.array(predicted_values).reshape(-1, 1))
    print("Predicted Values for the Next 12 Months:\n", predicted_values.flatten())
    

Predicted Values for the Next 12 Months:
[49227.05 49190.258 41204.16 30520.389 29591.129 34516.586 42183.016
38737.39 32026.324 23482.027 23381.914 30206.467]

Finally, here is the plot containing the original data together with the predicted values. Note: There are a lot of lines of code devoted to formatting the data and axis labels before graphing. This is, unfortunately, the nature of the beast. The full details of what each piece of code does will not be explained thoroughly in this text.

Python Code

    import matplotlib.pyplot as plt
    from matplotlib.ticker import MaxNLocator ## for graph formatting
    from matplotlib.ticker import FuncFormatter ## for formatting y-axis
    
    # Function to format the Y-axis values
    def y_format(value, tick_number):
     return f'{value:,.0f}'
    
    # Plot original time series data and predicted values
    plt.figure(figsize=(10, 6))
    plt.plot(data['Month'], data['Value'], label='Original Data', color='blue')
    plt.plot(range(len(data['Value']), len(data['Value']) + future_months), predicted_values,
        label='Predicted Values', color='red')
    
    # Create a list for the months
    pd_months = pd.date_range(start='2016-01-01', end='2023-07-01', freq='MS')
    
    xticks_positions = range(0, len(pd_months), 6) # Positions to display ticks (e.g., every 12 months)
    xticks_labels = [pd_months[pos].strftime('%Y-%m') for pos in xticks_positions] # Labels corresponding to positions
    
    plt.xlabel('Month')
    plt.ylabel('Value')
    plt.title('Monthly Consumption of Coal from 2016 to 2022 and Predicted Values for 2023')
    plt.legend()
    plt.xticks(ticks=xticks_positions, labels=xticks_labels, rotation=45)
    # Apply the formatter to the Y-axis
    plt.gca().yaxis.set_major_formatter(FuncFormatter(y_format))
    
    plt.show()
    

The resulting output will look like this: