Hypothesis Testing and Confidence Intervals

We have been using t-statistics and p-values to for testing hypotheses such as whether a correlation coefficient is significantly different from zero or if a regression coefficient is significantly different from zero.

In the following section, we will introduce the procedure of hypothesis testing formally.

Note that we will mainly focus on two-sided hypothesis tests when testing for the significance of a single coefficient as they are most commonly used in practice.

We will use one-sided tests when introducing F-tests for the overall significance of a regression model and for model comparison.

Study objectives for hypothesis testing

The goal is be able to conduct hypothesis test manually given necessary information.

For example, in case of testing if a regression coefficient is significantly different from zero, you should be able to

Establish null and alternative hypotheses.
Calculate the test statistic given the coefficient estimate, standard error.
Find the critical value of your test statistic given the significance level, 5% is commonly used.

You need to be familiar with the distribution table to find the critical value. The tables can be found HERE.
Decision rule.

Compare the test statistic with the critical value to make decision on rejecting or not rejecting the null hypothesis.
Interpret the result in context. What it means in reality?

1 t-test for Single Coefficient

Using the simple linear regression model of California School Test Scores in the previous session as an example, test whether expenditure has a significant effect on test scores at the 5% significance level.

Call:
lm(formula = TestScore ~ expenditure, data = cas)

Residuals:
    Min      1Q  Median      3Q     Max 
-50.146 -14.206   0.689  13.513  50.127 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 6.236e+02  7.720e+00  80.783  < 2e-16 ***
expenditure 5.749e-03  1.443e-03   3.984 7.99e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 18.72 on 418 degrees of freedom
Multiple R-squared:  0.03659,   Adjusted R-squared:  0.03428 
F-statistic: 15.87 on 1 and 418 DF,  p-value: 7.989e-05

Here is the hypothesis test step by step for the expenditure coefficient in the simple linear regression of California school test scores.

State Hypotheses
- $H_0:\ \beta_{\text{exp}} = 0$ (expenditure has no effect on test scores)
- $H_1:\ \beta_{\text{exp}} \neq 0$ (two-sided)
Calculate test statistic

\[ t=\frac{\hat\beta - \beta}{\text{SE}(\hat\beta)} \sim t_{df} \]

where the test statistic follows a $t$-distribution with degrees of freedom (df) $= n - k - 1.$ $k$ is the number of predictors (excluding the intercept).

Coming back to the regression output, we have estimate $= 0.005749,$ value to test against with $= 0,$ standard error $= 0.001443.$

That is, $\hat\beta = 0.005749,$ $\beta = 0,$ and $\text{SE}(\hat\beta) = 0.001443,$

\[ t=\frac{\hat\beta - \beta}{\text{SE}(\hat\beta)}=\frac{0.005749 - 0}{0.001443}=3.984 \]

where $n=420$ (number of observations) and $k=1$ (number of predictors), so $df=420-1-1=418$.
Find critical value, $C_{\alpha/2}.$

For significance level at 5%, i.e., $\alpha=0.05$, the critical value is given by

\[ C_{\alpha/2} = t_{0.975, df} \]

Referring to the $t$-distribution table, $t_{0.975,418}\approx 1.96$.

Finding critical value

Notice that when df is large, the critical value approaches that of the standard normal distribution, $z_{0.975}=1.96$.

The rule-of-thumb is that when $df > 30$, you can use the standard normal critical values.
Decision rule
- Reject $H_0$ if $|t|>1.96$.
- Fail to reject $H_0$ if $|t|\leq 1.96$.
Since $3.984>1.96$, we reject $H_0$ and conclude that $\beta_{\text{exp}}$ is statistically significantly different from zero.
Interpretation in context

Expenditure has a statistically significant positive association with test scores at the 5% level. The point estimate implies that a $1,000 increase in per-student expenditure is associated with about $0.005749\times 1000=5.75$ points higher test scores.

1.1 P-value Approach

Statistical software often reports the p-value, which is the smallest significance level at which you would reject the null hypothesis.

If the p-value $< \alpha$, reject $H_0$.
If the p-value $\geq \alpha$, fail to reject $H_0$.
Common significance levels are 1%, 5%, and 10%.
Interpretation: The probability of rejecting the null hypothesis when the null hypothesis is true.

Q: How to find the p-value given test statistic?
A: For two-sided test, the p-value is given by

\[ p\text{-value} = 2\;\P(T > |t|) \]

Example 1 A marketing manager wants to understand whether the number of social media posts influences monthly customer engagement for an online store. She runs a regression analysis and finds a t-statistic of $t = 2.457$ for the coefficient on the number of posts, with $df = 30$.

Compute the two-sided p-value.
Based on a 5% significance level, would you reject the null hypothesis that the number of posts has no effect on customer engagement?

The p-value is obtained by: \[p\text{-value} = 2\;\P(T > |t|) = 2 \times 0.01 = 0.02\] The p-value is 0.02, which is less than the common significance level of 0.05. Therefore, we reject the null hypothesis and conclude that the regression coefficient is statistically significant at the 5% level.

Interpretation: There is statistically significant evidence that the number of social media posts affects customer engagement for the online store.

Example 2 A financial analyst is studying the relationship between advertising expenditure and sales revenue for a chain of retail stores. She runs a regression and obtains an estimated coefficient for advertising spend. To test whether advertising has a statistically significant effect on sales, she computes a t-statistic of $t = 2.15$ with $df = 25$.

At the 5% significance level, should she reject the null hypothesis that advertising has no effect on sales?

The critical value at 5% significance level is $t_{0.975,25}=2.060$. Since $2.15 > 2.060$, we reject the null hypothesis at the 5% significance level.

Interpretation: There is statistically significant evidence that advertising expenditure affects sales revenue for the retail chain.

Example 3 A store manager wants to investigate whether the amount spent on in-store promotions affects weekly sales. She runs a regression and obtains a t-statistic of $t = 0.82$ for the coefficient on promotion spending, with $df = 28$.

At the 5% significance level, would you reject the null hypothesis that promotion spending has no effect on weekly sales?

The critical value at 5% significance level is $t_{0.975,28}=2.048$. Since $0.82 < 2.048$, we fail to reject the null hypothesis at the 5% significance level.

Interpretation:

There is no statistically significant evidence that the amount spent on in-store promotions affects weekly sales.

2 Confidence Interval for Single Coefficient

A 95 percent confidence interval for the slope is given by

\[ \left(\hat{\beta}-C_{\alpha/2}\times SE(\hat{\beta}),\; \hat{\beta}+C_{\alpha/2}\times SE(\hat{\beta})\right) \]

where $C_{\alpha/2}$ is the critical value for a two-sided test at significance level $\alpha$.

For the expenditure coefficient in the simple linear regression of California school test scores, $\hat{\beta}=0.005749$, $SE(\hat{\beta})=0.001443$, and $C_{0.025}=1.96$.

A 95% confidence interval is therefore

\[ 0.005749 \pm 1.96\times 0.001443 \approx [0.0029,\ 0.0086], \]

which corresponds to roughly 2.9 to 8.6 points per $1,000.

The effect is statistically significant but modest in size, and the low $R^2$ indicates that many other factors also influence test scores.