4.3 Correlation and Linear Regression Analysis

Interpret a Correlation Coefficient

Once the value of $r$ is calculated, this measurement provides two indicators for the correlation:

1. the strength of the correlation based on the value of $r$
2. the direction of the correlation based on the sign of $r$

The value of $r$ gives us this information:

• The value of $r$ is always between $-1$ and $+1$: $-1\le r\le 1$.
• The size of the correlation $r$ indicates the strength of the linear relationship between the two variables. Values of $r$ close to $-1$ or to $+1$ indicate a stronger linear relationship.
• If $r=0$, there is no linear relationship between the two variables (no linear correlation).
• If $r=1$, there is perfect positive correlation. If $r=-1$, there is perfect negative correlation. In both of these cases, all the original data points lie on a straight line.

The sign of $r$ gives us this information:

• A positive value of $r$ means that when $x$ increases, $y$ tends to increase, and when $x$ decreases, $y$ tends to decrease (positive correlation).
• A negative value of $r$ means that when $x$ increases, $y$ tends to decrease, and when $x$ decreases, $y$ tends to increase (negative correlation).

From the statistical results shown in the Python output, the correlation coefficient $r$ is 0.923, which indicates that the relationship between Nike stock and the value of the S&P 500 over this time period represents a strong, positive correlation.

Test a Correlation Coefficient for Significance

The correlation coefficient, $r$, tells us about the strength and direction of the linear relationship between $x$ and $y$. The sample data are used to compute $r$, the correlation coefficient for the sample. If we had data for the entire population (that is, all measurements of interest), we could find the population correlation coefficient, which is labeled as the Greek letter $\rho$ (pronounced “rho”). But because we have only sample data, we cannot calculate the population correlation coefficient. The sample correlation coefficient, $r$, is our estimate of the unknown population correlation coefficient.

• $\rho =\text{population correlation coefficient (unknown)}$
• $r=\text{sample correlation coefficient (known; calculated from sample data)}$

An important step in the correlation analysis is to determine if the correlation is significant. By this, we are asking if the correlation is strong enough to allow meaningful predictions for $y$ based on values of $x$. One method to test the significance of the correlation is to employ a hypothesis test. The hypothesis test lets us decide whether the value of the population correlation coefficient $\rho$ is close to zero or significantly different from zero. We decide this based on the sample correlation coefficient $r$ and the sample size n.

If the test concludes that the correlation coefficient is significantly different from zero, we say that the correlation coefficient is significant.

• Conclusion: There is sufficient evidence to conclude that there is a significant linear relationship between the two variables because the correlation coefficient is significantly different from zero.
• What the conclusion means: There is a significant linear relationship between the two variables. If the test concludes that the correlation coefficient is not significantly different from zero (it is close to zero), we say that correlation coefficient is not significant.

A hypothesis test can be performed to test if the correlation is significant. A hypothesis test is a statistical method that uses sample data to test a claim regarding the value of a population parameter. In this case, the hypothesis test will be used to test the claim that the population correlation coefficient $\rho$ is equal to zero.

Use these hypotheses when performing the hypothesis test:

• Null hypothesis: $H_0:\rho =0$
• Alternate hypothesis: $H_a:\rho \ne 0$

The hypotheses can be stated in words as follows:

• Null hypothesis $H_0$: The population correlation coefficient is not significantly different from zero. There is not a significant linear relationship (correlation) between $x$ and $y$ in the population.
• Alternate hypothesis $H_a$: The population correlation coefficient is significantly different from zero. There is a significant linear relationship (correlation) between $x$ and $y$ in the population.

A quick shorthand way to test correlations is the relationship between the sample size and the correlation. If $|r|\ge \frac{2}{\sqrt{n}}$, then this implies that the correlation between the two variables demonstrates that a linear relationship exists and is statistically significant at approximately the 0.05 level of significance. As the formula indicates, there is an inverse relationship between the sample size and the required correlation for significance of a linear relationship. With only 10 observations, the required correlation for significance is 0.6325; for 30 observations, the required correlation for significance decreases to 0.3651; and at 100 observations, the required level is only 0.2000.

Method of Least Squares and Residuals

To create a linear model that best fits the $(x,y)$ points on a scatterplot, researchers generally use the least squares method based on a mathematical algorithm designed to minimize the squared distances from the $(x,y)$ data points to the fitted line.

Recall from algebra that the equation of a straight line is given by:

where $m$ is the slope of the line and $b$ is the y-intercept of the line.

The slope measures the steepness of the line, and the y-intercept is that point on the $y$-axis where the graph crosses, or intercepts, the $y$-axis.

In linear regression analysis, the equation of the straight line is written in a slightly different way using the model

In this format, $b$ is the slope of the line, and $a$ is the y-intercept of the line. The notation $\hat{y}$ is called $y$-hat and is used to indicate a predicted value of the dependent variable $y$ for a certain value of the independent variable $x$.

If a line extends uphill from left to right, the slope is a positive value (see Figure 4.6; if the line extends downhill from left to right, the slope is a negative value).

Figure 4.6 Three examples of graphs of $\hat{y}=a+bx$. (a) If $b>0$, the line slopes upward to the right. (b) If $b=0$, the line is horizontal. (c) If $b<0$, the line slopes downward to the right.

When generating the equation of a line in algebra using $y=mx+b$, two $(x,y)$ points were required to generate the equation. However, in regression analysis, all the $(x,y)$ points in the dataset will be utilized to develop the linear regression model.

The first step in any regression analysis is to create the scatterplot. Then proceed to calculate the correlation coefficient $r$ and check this value for significance. If we think that the points show a linear relationship, we would like to draw a line on the scatterplot. This line can be calculated through a process called linear regression. However, we only calculate a regression line if one of the variables helps to explain or predict the other variable. If $x$ is the independent variable and $y$ the dependent variable, then we can use a regression line to predict $y$ for a given value of $x$.

As an example of a regression equation, assume that a correlation exists between the monthly amount spent on advertising and the monthly revenue for a Fortune 500 company. After collecting $(x,y)$ data for a certain time period, the company determines the regression equation is of the form

where $x$ represents the monthly amount spent on advertising (in thousands of dollars), $\hat{y}$ represents the monthly revenues for the company (in thousands of dollars), the slope is 61.8, and the y-intercept is 9376.7.

A scatterplot of the $(x,y)$ data is shown in Figure 4.7.

Figure 4.7 Scatterplot of Revenue versus Advertising for a Fortune 500 Company ($000s)

The company would like to predict the monthly revenue if its executives decide to spend $150,000 on advertising next month. To determine the estimate of monthly revenue, let $x=150$ in the regression equation and calculate a corresponding value for $\hat{y}$:

This predicted value of $y$ indicates that the anticipated revenue would be $18,646,700, given the advertising spend of $150,000.

Notice that from past data, there may have been a month where the company actually did spend $150,000 on advertising, and thus the company may have an actual result for the monthly revenue. This actual, or observed, amount can be compared to the prediction from the linear regression model to calculate what is called a residual.

A residual is the difference between an observed y-value and the predicted y-value obtained from the linear regression equation. As an example, assume that in a previous month, the actual monthly revenue for an advertising spend of $150,000 was $19,200,000, and thus $y=\mathrm{19,200}$. The residual for this data point can be calculated as follows:

Notice that residuals can be positive, negative, or zero. If the observed y-value exactly matches the predicted y-value, then the residual will be zero. If the observed y-value is greater than the predicted y-value, then the residual will be a positive value. If the observed y-value is less than the predicted y-value, then the residual will be a negative value.

When formulating the best-fit linear regression line to the points on the scatterplot, the mathematical analysis generates a linear equation where the sum of the squared residuals is minimized. This analysis is referred to as the method of least squares. The result is that the analysis generates a linear equation that is the “best fit” to the points on the scatterplot, in the sense that the line minimizes the differences between the predicted values and observed values for $y$; this is generally referred to as the best-fit linear equation.

The goal in regression analysis is to determine the coefficients $a$ and $b$ in the following regression equation:

Once the $(x,y)$ has been collected, the slope ($b$) and y-intercept ($a$) can be calculated using the following formulas:

where $n$ refers to the number of data pairs and $\Sigma x$ indicates sum of the x-values.

Notice that the formula for the y-intercept requires the use of the slope result (b), and thus the slope should be calculated first, and the y-intercept should be calculated second.

When making predictions for $y$, it is always important to plot a scatter diagram first. If the scatterplot indicates that there is a linear relationship between the variables, then it is reasonable to use a best-fit line to make predictions for $y$, given $x$ is within the domain of x-values in the sample data, but not necessarily for x-values outside that domain.

Assumptions for Linear Regression

Testing the significance of the correlation coefficient requires that certain assumptions about the data are satisfied. The premise of this test is that the data are a sample of observed points taken from a larger population. We have not examined the entire population because it is not possible or feasible to do so. We are examining the sample to draw a conclusion about whether the linear relationship that we see between $x$ and $y$ data in the sample data provides strong enough evidence that we can conclude that there actually is a linear relationship between $x$ and $y$ data in the population.

The regression line equation that we calculate from the sample data gives the best-fit line for our particular sample. We want to use this best-fit line for the sample as an estimate of the best-fit line for the population (Figure 4.8). Examining the scatterplot and testing the significance of the correlation coefficient helps us determine if it is appropriate to do this.

These are the assumptions underlying the test of significance:

1. There is a linear relationship in the population that models the average value of $y$ for varying values of $x$. In other words, the expected value of $y$ for each particular value lies on a straight line in the population. (We do not know the equation for the line for the population. Our regression line from the sample is our best estimate of this line in the population.)
2. The y-values for any particular x-value are normally distributed about the line. This implies that there are more y-values scattered closer to the line than are scattered farther away. Assumption (1) implies that these normal distributions are centered on the line: the means of these normal distributions of y-values lie on the line.
3. The standard deviations of the population y-values about the line are equal for each value of $x$. In other words, each of these normal distributions of y-values has the same shape and spread about the line.
4. The residual errors are mutually independent (no pattern).
5. The data are produced from a well-designed random sample or randomized experiment.

Figure 4.8 Best-Fit Line. Note that the y-values for each x-value are normally distributed about the line with the same standard deviation. For each x-value, the mean of the y-values lies on the regression line. More y-values lie near the line than are scattered farther away from the line

Develop a Linear Regression Model

In the following example, Python will be used for the following analysis based on a given dataset of $(x,y)$ data.

• Create a scatterplot.
• Calculate the correlation coefficient.
• Construct the best-fit linear equation.
• Predict values of the dependent variable for given values of the independent variable.

There are various ways to accomplish these tasks in Python, but we will use several functions available within the scipy library, such as:

Pearsonr() for correlation coefficient

linregress() for linear regression model

plt.scatter() for scatterplot generation

Example 4.24

Problem

A marketing manager is interested in studying the correlation between the amount spent on advertising and revenue at a certain company. Twelve months of data were collected as shown in Table 4.16. (Dollar amounts are in thousands of dollars.)

Use Python to perform the following analysis:

1. Generate a scatterplot.
2. Calculate the correlation coefficient $r$.
3. Determine if the correlation is significant (use level of significance of 0.05).
4. Construct a linear regression model.
5. Use the model to predict the revenue for a certain month where the amount spent on advertising is $100.

Month	Advertising Spend	Revenue
Jan	49	12210
Feb	145	17590
Mar	57	13215
Apr	153	19200
May	92	14600
Jun	83	14100
Jul	117	17100
Aug	142	18400
Sep	69	14100
Oct	106	15500
Nov	109	16300
Dec	121	17020

Table 4.16 Revenue versus Advertising for a Company (in $000s)

Solution

1. The first step in the analysis is to generate a scatterplot. (Recall that scatterplots were also discussed in Scatterplots and Correlation.)
   The Python function plt.scatter() can be used to generate the scatterplot for the $(x,y)$ data. Note that we consider advertising to be the independent variable ($x$-data) and revenue to be the dependent variable ($y$-data) since revenue depends on the amount of advertising.

Here is the Python code to generate the scatterplot:

Python Code

        # import the matplotlib library
        import matplotlib.pyplot as plt
        
        # define the x-data, which is amount spent on advertising
        x = [49, 145, 57, 153, 92, 83, 117, 142, 69, 106, 109, 121]
        
        # define the y-data, which is revenue
        y = [12210, 17590, 13215, 19200, 14600, 14100, 17100, 18400, 14100, 15500, 16300, 17020]
        
        # use the scatter function to generate a time series graph
        plt.scatter(x, y)
        
        # Add a title using the title function
        plt.title("Revenue versus Advertising for a Company")
        
        # Add labels to the x and y axes by using xlabel and ylabel functions
        plt.xlabel("Advertising $000")
        plt.ylabel ("Revenue $000")
        

The resulting output will look like this:

2. The Python function pearsonr() can be used to generate the correlation coefficient, $r$.

Here is the Python code to calculate the correlation coefficient:

Python Code

        # import libraries
        import numpy as np
        from scipy.stats  import pearsonr
        
        # define the x-data, which is amount spent on advertising
        x = [49, 145, 57, 153, 92, 83, 117, 142, 69, 106, 109, 121]
        
        # define the y-data, which is revenue
        y = [12210, 17590, 13215, 19200, 14600, 14100, 17100, 18400, 14100, 15500, 16300, 17020]
        
        #pearson r returns both value of r and corresponding p-value
        r, p = pearsonr(x, y)
        
        #print value of r, rounded to 3 decimal places
        
        print("Correlation coefficient: ", round(r, 3))
        

 

The resulting output will look like this:

Correlation coefficient: 0.981
This value of $r=0.981$ indicates a strong, positive correlation between advertising spend and revenue.

3. When using a level of significance of 0.05, if $|r|\ge \frac{2}{\sqrt{n}}$, then this implies that the correlation between the two variables demonstrates a linear relationship that is statistically significant at approximately the 0.05 level of significance. In this example, we check if $\left|0.981\right|$ is greater than or equal to $\frac{2}{\sqrt{n}}$ where $n=12$. Since $\frac{2}{\sqrt{12}}$ is approximately 0.577, this indicates that the r-value of 0.981 is greater than $\frac{2}{\sqrt{n}}$, and thus the correlation is deemed significant.
4. The linear regression model will be of the form:
   
   $$\hat{y}=a+bx$$
   To determine the slope and y-intercept of the regression model, the Python function linregress() can be used.
   This function takes the $x$ and $y$ data as inputs and provides the slope and intercept as well as other regression-related output.

Here is the Python code to generate the slope and y-intercept of the best-fit line:

Python Code

        # import libraries
        import numpy as np
        from scipy.stats  import linregress
        
        # define the x-data, which is amount spent on advertising
        x = [49, 145, 57, 153, 92, 83, 117, 142, 69, 106, 109, 121]
        
        # define the y-data, which is revenue
        y = [12210, 17590, 13215, 19200, 14600, 14100, 17100, 18400, 14100, 15500, 16300, 17020]
        
        #pearson r returns both value of r and corresponding p-value
        slope, intercept, r_value, p_value, std_err  = linregress(x, y)
        
        print ("slope  =", slope)
        print("intercept = ", intercept)
        

 

The resulting output will look like this:

slope = 61.79775818816664
intercept = 9376.698881009072

Based on this Python equation, the regression equation can be written as

$$\begin{array}{rcl}\hat{y}& =& a+bx\\ \hat{y}& =& 9376.7+61.8x\end{array}$$

where $x$ represents the amount spent on advertising (in thousands of dollars) and $y$ represents the amount of revenue (in thousands of dollars).

It is also useful to plot this best-fit line superimposed on the scatterplot to show the strong correlation between advertising spend and revenue, as follows (see Figure 4.9):

Figure 4.9 Best-Fit Line Superimposed on the Scatterplot

5. The regression equation is: $\hat{y}=\mathrm{9,376.7}+61.8x$.
   The given amount of advertising spend of $100 corresponds to a given x-value, so replace “$x$” with 100 in this regression equation to generate a predicted revenue:
   
   $$\begin{array}{rcl}\hat{y}& =& 9376.7+61.8x\\ \hat{y}& =& 9376.7+61.8\left(100\right)\\ \hat{y}& =& 15556.7\end{array}$$
   Since both advertising and revenue are expressed in thousands of dollars, the conclusion is that if advertising spend is $100,000, the predicted revenue for the company will be $15,556,700.

Interpret and Apply the Slope and y-Intercept

The slope of the line, $b$, describes how changes in the variables are related. It is important to interpret the slope of the line in the context of the situation represented by the data. You should be able to write a sentence interpreting the slope in plain English.