8.1 The Business Analytics Process

Challenge: Volume

The collection and use of big data have become increasingly important in today’s business landscape, yet harnessing the very real potential of big data comes with significant challenges. The sheer volume, velocity of production, and variety of data can overwhelm those who cling to traditional data management and analysis methods. Analysts report that by 2025 the global volume of digital information is expected to reach 200 zettabytes.⁴ Organizations must grapple with storing and processing enormous amounts of data. Designers and analysts need to work together to create and maintain scalable infrastructure capable of hosting advanced analytics tools.

Challenge: Quality

In addition to volume, the quality of big data poses challenges, as unstructured and noisy data can hinder accurate analysis and interpretation. This has prompted concern in situations where data analytics is key to success. Reliability issues stem from multiple causes. They can include inaccurate data, redundant entries, and simple human error in data entry.

Duplicated, or redundant, entries are multiple entries placed in the same dataset by mistake. There are various methods to respond to redundant entries. The first and most obvious is to simply remove them. Data engineers may use tools such as basic Python code and spreadsheet functions to filter out corrupt data at prescribed levels to produce a more accurate dataset. Input tools such as QR code scanners can help by automating the process. Another technique to address the issue of redundancy is to assign another value to an outlier (an observation that deviates significantly from the rest of the dataset), potentially indicating anomalies, errors, or unique patterns that require special attention during analysis. In other words, you would choose a value with significantly lower impact on the dataset to replace the outliers, such as an average value.

Challenge: Governance

Have you ever had your identity stolen? If not, you may know someone who has. These concerns relate to privacy and data governance, which is the overall management of the availability, usability, integrity, and security of data used in an enterprise. At the business level, companies do their best to comply with regulations and protect sensitive information. However, enforcement of strict digital privacy laws can vary from state to state, or nation to nation. Companies that do business in Europe must also abide by Europe’s General Data Protection Regulation (GDPR), which as you may recall from 6.1 Key Concepts in Data Privacy and Data Security, is a leading global regulation in terms of enforcing transparency in how data are handled and strictly forbids the purchase and sale of personally identifiable data while allowing individuals the right to be forgotten. The GDPR is built upon several fundamental principles aimed at protecting the personal data of individuals within the European Union (EU). Refer to 6.4 Managing Enterprise Risk and Compliance to review these fundamental principles.

Challenge: Actionable Insights

The process of systematically using statistical and logical techniques to review, sort, and condense data for the purpose of gaining insight on areas of interest is called data analysis. Effective data analysis is critical for ensuring that the information is accurate so that an organization can then extract actionable insights from it. Extracting these insights is, however, a significant challenge. First, it requires special training to increase skills and expertise in this area. Data scientists and analysts must possess a combination of statistical knowledge, programming skills, and domain expertise to navigate the complexities of big data analysis. Additionally, it is important to be able to comprehend the results and to communicate them effectively to a broad audience. Incorrectly linking correlation to causation during data analysis can be an issue with both experts and software, and false positives and false negatives can lead conclusions astray. Additional challenges can arise from the regulations in some regions, such as the EU, that prohibit collecting and storing meaningful data to ensure privacy or decrypting encrypted data.

Step 1: Problem Definition

The first step in the process is problem definition. Here, the company sets out to name the problem or challenge that needs a solution that involves using analytics. To illustrate this process, consider an organization studying botany that sets out to classify varieties of iris. By clearly defining the problem, organizational leaders can focus their data analytics efforts and ensure alignment with the organization’s goals. They can then begin to gather information to form the classifications.

Step 2: Data Preparation

Next comes collecting, cleaning, and transforming the data for analysis. This step includes gathering data from various sources, integrating separate datasets, and ensuring data quality. That can be a cumbersome task, since the organization will attempt to address issues such as missing values, outliers, or inconsistencies. Techniques for data cleaning and data transformation, such as data normalization or feature engineering, may be applied to ensure the data are suitable for analysis.

Data normalization involves adjusting data so that they have a standard scale, making it easier to compare different types of values. It ensures that one feature does not dominate others due to its scale. Dividing irises into categories is a relatively simple analysis and does not require data normalization. Other examples that would benefit from data normalization include comparing salary in thousands of dollars to years of experience, or comparing house prices and sizes. In the latter example, normalizing the size (by dividing all sizes by the largest size) can put that variable on a comparable scale to prices.

Feature engineering is transforming raw data into useful inputs for a model by creating, modifying, or selecting features (data points). It helps models understand patterns better by making relevant information more accessible. As an example, for predicting house prices, creating a new feature like “price per square foot” combines raw price and size into something more insightful.

As a simple use case, imagine predicting student test scores using hours studied and study material pages read. These features can be normalized so that the number of pages read does not overpower the number of hours studied, and a feature like efficiency (pages read per hour) can be engineered to capture how productive a student is. Effective data preparation is crucial for accurate and reliable results in subsequent stages of the analytics process.

Built-in Libraries

Many programming languages, like Python, can use built-in libraries for the purpose of testing models. If you use Python for data analytics, you’ll find it equipped with powerful libraries of built-in code segments and datasets tailored for various tasks, such as NumPy and Pandas. NumPy is useful for numerical calculations, while Pandas excels in handling data analytics tasks.

With these available libraries, Python becomes an ideal choice for scientific analysis and data-driven applications. Let’s use the classic public domain dataset for iris classification from R. A. Fisher⁵ for this example. The following snippet shows the code for importing the library and creating a pie chart. The example output is shown in Figure 8.3. Note that the line from sklearn import datasets instructs Python to use the library sklearn, which allows access to data on the iris species. You are also importing the matplotlib to create the pie chart.

# Iris Species Data Study
# Import libraries
from sklearn import datasets
import pandas as pd
import matplotlib.pyplot as plt

# Load the Iris dataset
iris_data = datasets.load_iris()

# Create a DataFrame using the data and feature names
iris = pd.DataFrame(iris_data.data, columns=iris_data.feature_names)

# Add the species column by mapping the target integer values to species names
iris['species'] = iris_data.target_names[iris_data.target]

# Plot a pie chart showing the distribution of species in the dataset
species_count = iris['species'].value_counts() # Count occurrences of each species
plt.figure(figsize=(7, 7))
plt.pie(species_count, labels=species_count.index, autopct='%1.1f%%', startangle=90)
plt.title('Distribution of Iris Species')
plt.show()

Figure 8.3 Python code was written to load and create a pie chart of various species of iris plants. (data source: R. A. Fisher, “The Use of Multiple Measurements in Taxonomic Problems,” Annals of Eugenics 7, no. 2 (September 1936): 179–88. https://doi.org/10.1111/j.1469-1809.1936.tb02137.x; attribution: Copyright Rice University, OpenStax, under CC BY 4.0 license)

You can observe how the code created a simple pie chart output to show the proportion of species of irises.

# Pull Data from an Excel Spreadsheet
import pandas as pd
import matplotlib.pyplot as plt

# Load the Excel file
df = pd.read_excel('C:/Users/daxbr/iris.xlsx')

# Count the occurrences of each species
x = df['Species'].value_counts()
# Get the labels (species names) from the value counts theLabels
= x.index.tolist()

# Plot the pie chart of species distribution
plt.pie(x, labels=theLabels, autopct='%1.1f%%', startangle=90)
plt.title('Distribution of Iris Species')
plt.show()

# Enter Your Own Data
theSlices = [33.3, 33.3, 33.3]
theLabels = ["Virginica", "Versicolor", "Setosa"]
theColors = ['#96f97b','#ff81c0','r']

# Plot the pie chart
import matplotlib.pyplot as plt

plt.pie(theSlices, labels=theLabels, colors=theColors, autopct='%1.1f%%', startangle=90)
plt.title('Iris Species Distribution')
plt.show()

Step 3: Statistical Analysis

Once the data are prepared, data analysts apply statistical analysis to uncover patterns, relationships, and insights. This stage involves using statistical methods, machine learning algorithms, or data mining techniques, or a combination of methods, to explore and analyze the data (Figure 8.4). Depending on the nature of the data, analysts may employ descriptive statistics, regression analysis, clustering, classification, or predictive modeling. Analysts use these tools to run simulations. This provides opportunities to observe potential costs, predict return on investment (ROI), and identify metrics. Descriptive statistics help to give a “snapshot” of data and provide a jumping-off point for analysis.

Figure 8.4 Some analysis methods can be categorized as statistical methods, machine learning algorithms, or data mining techniques, but many can fit into more than one category. (attribution: Copyright Rice University, OpenStax, under CC BY 4.0 license)

Causality or Correlation?

Correlation does not imply causation, but does one attribute affect another? Returning to the iris data, the following simple command can explore the correlation of sepal length and petal length (Figure 8.5).:

df.corr()

Figure 8.5 The simple Python command generates a comparison of the lengths and widths of petals and sepals and shows a positive correlation between the sepal length and petal length. (attribution: Copyright Rice University, OpenStax, under CC BY 4.0 license)

We can recognize that the petal length and the sepal length are strongly correlated; however, that correlation does not prove causality. It is a common pitfall for an analyst to believe that they have proven causation because of a strong correlation. The challenge in statistics is to remain objective and be cautious about using the word “proof.” For example, there is 100 percent correlation between eating chocolate and being born. Every person who eats chocolate has been born, but being born does not cause one to eat chocolate. In the iris case study, we showed correlation between sepal length and petal length only. There is no evidence from this data that either sepal length causes petal length or petal length causes sepal length.

Step 4: Results Interpretation

The next step is results interpretation, where the insights are translated into actionable information. Analysts evaluate the findings in the context of the problem at hand, interpret the statistical results, and draw conclusions. They often create visualizations, charts, or reports from the data to effectively communicate insights to stakeholders.

Step 5: Implementation

The best data in the world are functionally useless without action. In the final phase, implementation involves applying the obtained insights and recommendations to practice. This may involve strategic decision-making, process improvements, or operational changes based on the findings. Implementation may also require collaboration across departments or the integration of analytical models into existing systems or workflows.