6.5 Other Machine Learning Techniques

Random Forests

In Decision Trees, we constructed decision trees and discussed some methods, such as pruning, that would improve the reliability of predictions of the decision tree. For each classification problem, one tree is created that serves as the model for all subsequent purposes. But no matter how much care is taken in creating and pruning a decision tree, at the end of the day, it is a single model that may have some bias built into its very structure due to variations in the training set. If we used a different training set, then our decision tree may have come out rather differently.

In the real world, obtaining a sufficient amount of data to train a machine learning model may be difficult, time-consuming, and expensive. It may not be practical to go out and find additional training sets just to create and evaluate a variety of decision trees. A random forest, a classification algorithm that uses multiple decision trees, serves as a way to get around this problem. Similar to what we did for linear regressions in Machine Learning in Regression Analysis, the technique of bootstrap aggregating is employed. In this context, bootstrap aggregating (bagging) involves resampling from the same testing dataset multiple times, creating a new decision tree each time. The individual decision trees that make up a random forest model are called weak learners. To increase the diversity of the weak learners, there is random feature selection, meaning that each tree is built using a randomly selected subset of the feature variables. For example, if we are trying to predict the presence of heart disease in a patient using age, weight, and cholesterol levels, then a random forest model will consist of various decision trees, some of which may use only age and weight, while others use age and cholesterol level, and still others may use weight alone. When classifying new data, the final prediction of the random forest is determined by “majority vote” among the individual decision trees. What’s more, the fact that individual decision trees use only a subset of the features means that the importance of each feature can be inferred by the accuracy of decision trees that utilize that feature.

Because of the complexity of this method, random forests will be illustrated by way of the following Python code block.

Suppose you are building a model that predicts daily temperatures based on such factors as the temperature yesterday and the day before as well forecasts from multiple sources, using the dataset temps.csv. Since there are a lot of input features, it will not be possible to visualize the data. Moreover, some features are likely not to contribute much to the predictions. How do we sort everything out? A random forest model may be just the right tool. The Python library sklearn has a module called sklearn.ensemble that is used to create the random forest model.

Python Code

      # Import libraries
      import pandas as pd  ## for dataset management
      from sklearn.model_selection import train_test_split
      import sklearn.ensemble as ens
      
      # Read input file
      features = pd.read_csv('temps.csv').dropna()
      
      # Use 'actual' as the response variable
      labels = features['actual']
      
      # Convert text data into numerical values
      # This is called "one-hot" encoding
      features = pd.get_dummies(features)
      
      # The other columns are used as features
      features = features.drop('actual', axis=1)
      feature_list = list(features.columns)
      
      # Split data into training and testing sets
      train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=0.25)
      
      # Create random forest model
      rf = ens.RandomForestRegressor(n_estimators=1000)
      rf.fit(train_features, train_labels)
    

The resulting output will look like this:

The preceding Python code reads in the dataset temps.csv, identifying the column “actual” (actual temperatures each day) as the variable to be predicted. All the other columns are assumed to be features. Since the “week” column contains categorical (text) data, it needs to be converted to numerical data before any model can be set up. The method of one-hot encoding does the trick here. In one-hot encoding, each category is mapped onto a vector containing a single 1 corresponding to that category while all other categories are set to 0. In our example, each day of the week maps to a seven-dimensional vector as follows:

• Monday: [1, 0, 0, 0, 0, 0, 0]
• Tuesday: [0, 1, 0, 0, 0, 0, 0]
• Wednesday: [0, 0, 1, 0, 0, 0, 0]
• Thursday: [0, 0, 0, 1, 0, 0, 0]
• Friday: [0, 0, 0, 0, 1, 0, 0]
• Saturday: [0, 0, 0, 0, 0, 1, 0]
• Sunday: [0, 0, 0, 0, 0, 0, 1]

The dataset is split into training (75%) and testing (25%) sets, and the random forest model is trained on the training set, as seen in this Python code block:

      import statistics as stats # to compute the mean

      # Get predictions and compute average error
      predictions = rf.predict(test_features)
      errors = abs(predictions - test_labels)
      round(np.mean(errors),2)
    

3.91

With a mean absolute error of only 3.91, predicted temperatures are only off by about 4°F on average, so the random forest seems to do a good job with the test set, given that it is notoriously difficult to predict weather data since it typically shows high variance. Now let’s find out which features were most important in making predictions, which is stored in rf.feature_importances_. Note the number of Python commands used solely for sorting and formatting the results, which can be safely ignored in this snippet of code.

    # Find the importance of each feature
    importances = list(rf.feature_importances_)
    feature_importances = [(feature, round(importance,2)) for feature, importance in zip(feature_list, importances)]
    feature_importances = sorted(feature_importances, key=lambda x: x[1], reverse=True)
    [print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];
    

Variable: temp_1               Importance: 0.62
Variable: average              Importance: 0.19
Variable: forecast_acc         Importance: 0.06
Variable: forecast_noaa        Importance: 0.04
Variable: day                  Importance: 0.02
Variable: temp_2               Importance: 0.02
Variable: forecast_under       Importance: 0.02
Variable: friend               Importance: 0.02
Variable: month                Importance: 0.01
Variable: year                 Importance: 0.0
Variable: week_Fri             Importance: 0.0
Variable: week_Mon             Importance: 0.0
Variable: week_Sat             Importance: 0.0
Variable: week_Sun             Importance: 0.0
Variable: week_Thurs           Importance: 0.0
Variable: week_Tues            Importance: 0.0
Variable: week_Wed             Importance: 0.0

As we can see, the single best predictor of daily temperatures is the temperature on the previous day (“temp_1”), followed by the average temperature on this day in previous years (“average”). The forecasts from NOAA and ACC showed only minor importance, while all other feature variables were relatively useless in predicting current temperatures. It should come as no surprise that the day of the week (Monday, Tuesday, etc.) played no significant part in predicting temperatures.

Multinomial Naïve Bayes Classifiers

One method for classifying data is to look for commonly occurring features that would distinguish one data point from another. For example, imagine you have a vast collection of news articles and you want to automatically categorize them into topics such as sports, politics, business, entertainment, and technology. You may have noticed certain patterns, such as sport articles tend to mention team names and give numerical scores, while articles about politics mention the words “Democrat” and “Republican” fairly often. Business articles might use words such as “outlook” and “downturn” much more often than entertainment or technology articles would. Multinomial naïve Bayes can be applied to classify these articles based on the frequency and distribution of words typically found in each kind of article.

The multinomial naïve Bayes classification algorithm is based on Bayes’ Theorem, making use of prior probabilities and Bayes’ Theorem to predict the class or label of new data. (See Normal Continuous Probability Distributions for more details.) The naïve Bayes classifier algorithm is a powerful tool for working out probabilities of events based on prior knowledge. Recall, the notation $P\left(A\right|B)$ stands for the conditional probability that event $A$ occurs given that event $B$ is known to occur (or has already occurred or is presumed to occur). In the simplest version, Bayes’ Theorem takes the form

Here, we regard $A$ as an event that we would like to predict and $B$ as information that has been given to us (perhaps as a feature of a dataset). The conditional probability $P\left(B\right|A)$ represents the known likelihood of seeing event $B$ in the case that A occurs, which is typically found using the training data. The value of $P\left(A\right)$ would be computed or estimated from training data as well and is known as the prior probability.

In practice, there are often a multitude of features, $B_1,B_2,B_3,\dots B_n$, where each feature could be the event that a particular word occurs in a message, and we would like to classify the message into some type or another based on those events. In other words, we would like to know something about $P\left(A\right|B_1,B_2,B_3,\dots ,B_n)$, the probability of A occurring if we know that events $B_1,B_2,B_3,\dots B_n$ occurred. In fact, we are just interested in distinguishing event $A$ from $A\prime$ rather than precise computations of probabilities, and so we will do simple comparisons to find out this information. The process is best illustrated by example.

Note: Naïve Bayes classification assumes that all features of the data are independent. While this condition may not necessarily be true in real-world datasets, naïve Bayes may still perform fairly well in real-world situations. Moreover, naïve Bayes is a very simple and efficient algorithm, and so it is often the first method used in a classification problem before more sophisticated models are employed.

Gaussian Naïve Bayes for Continuous Probabilities

Gaussian naïve Bayes is a variant of the naïve Bayes classification algorithm that is specifically designed for data with continuous features. Unlike the standard (multinomial) naïve Bayes, which is commonly used to classify text data with discrete features, Gaussian naïve Bayes is suitable for datasets with features that follow a Gaussian (normal) distribution (You may recall the discussion of the normal distribution in Discrete and Continuous Probability Distributions). It's particularly useful for problems involving real-valued, continuous data in which the feature variables are relatively independent.

In this example, we will work with a large dataset, cirrhosis.csv, that has many features, both categorical and numerical. This dataset (along with many more!) is available for free download from Kaggle.com. We will only use four of the numerical columns as features. The response variables is “Status,” which may take the values $D$ for death, $C$ for censored (meaning that the patient did not die during the observation period), and CL for censored due to liver transplantation.

      # Import libraries
      import pandas as pd  ## for dataset management
      from sklearn.model_selection import train_test_split
      from sklearn.naive_bayes import GaussianNB
      from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
      
      # Load the dataset
      data = pd.read_csv('cirrhosis.csv').dropna()
      
      # Choose feature columns and target labels
      X = data[['Bilirubin','Cholesterol','Albumin','Copper']]
      y = data['Status']
      
      # Split the dataset into training and testing sets (80% training, 20% testing)
      X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
      
      # Initialize and train the Gaussian naive Bayes classifier
      gnb_classifier = GaussianNB()
      gnb_classifier.fit(X_train, y_train)
      
      # Predict labels for the test set
      y_pred = gnb_classifier.predict(X_test)
      
      # Evaluate the classifier's performance
      accuracy = accuracy_score(y_test, y_pred)
      confusion = confusion_matrix(y_test, y_pred, labels=['D','C','CL'])
      report = classification_report(y_test, y_pred)
      
      print(f"Accuracy: {accuracy:.2f}")
      print("Confusion matrix with label order D, C, CL:\n", confusion)
    

Accuracy: 0.71
Confusion matrix with label order D, C, CL:
[[10 14  2]
[ 0 30  0]
[ 0  0  0]]

The accuracy score of 0.71 means that correct labels were assigned 71% of the time. We can see that 10 deaths ($D$) and 30 non-deaths ($C$ or $CL$) were corrected predicted. However, 16 predictions of death were ultimately non-deaths. This is a much more desirable error than the reverse (prediction of non-death when in fact the patient will die), so the model seems to be appropriate for the task of predicting cirrhosis fatalities.

Working with Big Data

In the rapidly evolving landscape of machine learning, the concept of big data looms large. As we saw in Handling Large Datasets, large datasets, often termed big data, refers to extremely large and complex datasets that are beyond the capacity of traditional data processing and analysis tools to efficiently handle, manage, and analyze. Big data is characterized by the “three Vs”:

• Volume: Big data involves massive amounts of data, often ranging from terabytes ($2^{40}\approx {10}^{12}$ bytes) to as large as petabytes ($2^{50}\approx {10}^{15}$ bytes) and beyond. Where does such vast amounts of data originate? One example is the Common Crawl dataset, which contains petabytes of data consisting of the raw web page data, metadata, and other information available from the internet. It is likely that there will be datasets containing exabytes ($2^{60}\approx {10}^{18}$ bytes) of information in the near future.
• Velocity: In some important real-world applications, data is generated and collected at an incredibly high speed. This continuous flow of data may be streamed in real time or stored in huge databases for later analysis. One example is in the area of very high-definition video, which can be captured, stored, and analyzed at a rate of gigabits per hour.
• Variety: Big data also encompasses diverse types of data, including structured data (e.g., databases), unstructured data (e.g., text, images, videos), and anything in between. This variety adds considerable complexity to data management and analysis. Consider all the various sources of data and how different each may be in transmission and storage. From social media interactions and e-commerce transactions to sensor data from Internet of Things (IoT) devices, data streams in at a pace, scale, and variety never before seen. Moreover, few will be able to predict the brand-new ways that data will be used and gathered in the future!