Module 03 — Linear Regression

A model is fit as $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \varepsilon_i$ by ordinary least squares. Which statement best describes this model?

• A It is nonlinear regression because the response curves with $x$.
• B It is a generalised additive model because two basis terms enter additively.
• C It can only be fit by gradient descent because of the quadratic term.
• D It is linear regression because it is linear in the parameters $\boldsymbol\beta$.

Starting salary (in $\$1{,}000$) is modelled as $$\hat y = 50 + 20\,\text{GPA} + 0.07\,\text{IQ} + 35\,\text{Level} + 0.01\,(\text{GPA}\cdot\text{IQ}) - 10\,(\text{GPA}\cdot\text{Level})$$ with $\text{Level} = 1$ for college and $0$ for high school. Which statement is correct, holding $\text{IQ}$ and $\text{GPA}$ fixed?

• A College graduates earn more on average than high-school graduates regardless of GPA.
• B High-school graduates earn more on average than college graduates regardless of GPA.
• C College graduates earn more on average, provided that GPA is high enough.
• D High-school graduates earn more on average, provided that GPA is high enough.

Adult weight (kg) is modelled on age (years) and sex (male = 1, female = 0) with an $\text{age}\times\text{sex}$ interaction. The fitted equation is $$\hat{\text{weight}} = 60.2 + 0.11\,\text{age} + 5.03\,\text{sex} + 0.68\,(\text{age}\cdot\text{sex}).$$ Mark each statement as true or false.

• a) Female weight increases on average by $0.79$ kg per year of age. True False
• b) Males weigh on average $5.03$ kg more than females across the entire age range in the data. True False
• c) The age slope for males is $0.79$ kg per year. True False
• d) The dependence of weight on age is allowed to differ between males and females in this model. True False

With $n=100$ observations of a single predictor $X$ and quantitative response $Y$, you fit Model A: $\hat y = \hat\beta_0 + \hat\beta_1 x$, and Model B: $\hat y = \hat\beta_0 + \hat\beta_1 x + \hat\beta_2 x^2 + \hat\beta_3 x^3$. Suppose the *true* relationship is linear. What can you say about the **training** RSS of the two models?

• A Training RSS is the same, since the truth is linear and OLS is unbiased.
• B Training RSS is lower for Model B, because adding parameters cannot raise training RSS.
• C Training RSS is lower for Model A, because the cubic terms add irreducible noise to the residuals.
• D Cannot be determined without seeing the test set, since RSS depends on the held-out split.

Same setup as Q4 (true relationship is linear, $n=100$). What do you expect for the **test** RSS of Model A versus Model B?

• A Lower for Model A on average; Model B overfits noise.
• B Lower for Model B on average; more flexibility always helps prediction.
• C Identical, because both models contain the truth.
• D The cubic and linear models give exactly the same predictions, so test RSS is identical.

A Boston-housing-style regression of median home value (medv, in $\$1000$) on nine predictors plus an $\text{rm}^2$ quadratic term gives the following partial output for the $\text{chas}$ indicator (1 if tract bounds the Charles River, 0 otherwise):

chas  Estimate 3.36  Std. Error 0.86  t value 3.91  Pr(>|t|) 0.0001

Using the working approximation $t_{0.975, n-p-1}\approx 2$, the approximate 95% confidence interval for the chas effect is closest to:

• A $[3.36 \pm 0.86]$, i.e. $[2.50,\ 4.22]$ thousand dollars.
• B $[3.36 \pm 1.72]$, i.e. $[1.64,\ 5.08]$ thousand dollars.
• C $[3.36 \pm 7.82]$, i.e. $[-4.46,\ 11.18]$ thousand dollars.
• D $[0.0001,\ 3.36]$ thousand dollars.

A linear model is fit with an intercept, 7 continuous predictors, a 9-level factor (entered as dummies with one reference level), and one quadratic term I(rm^2). How many regression parameters does the fitted model consume (counting the intercept)?

• A $7 + 9 + 1 = 17$.
• B $1 + 7 + 8 + 1 = 17$.
• C $1 + 7 + 9 + 1 = 18$.
• D $1 + 7 + 8 + 2 = 18$.

A multiple regression is fit with $n = 200$ observations and $p = 9$ slope parameters. For one predictor, the output gives $\hat\beta = 0.40$ and $\mathrm{SE}(\hat\beta) = 0.20$. Using the working cutoff $t_{0.975, 190}\approx 1.97$, is this coefficient significant at the 5% level, and what is its t-statistic?

• A $t = 2.0$; significant at the 5% level.
• B $t = 0.5$; not significant at the 5% level.
• C $t = 0.08$; not significant at the 5% level.
• D $t = 2.0$; not significant because $|t|$ has to exceed $t_{0.99}\approx 2.6$.

For a t-test with null $H_0:\beta_j = 0$ and observed two-sided p-value $p$, mark each statement as true or false.

• a) $1 - p$ is the probability that $H_0$ is true. True False
• b) If $p > 0.05$, we conclude that $H_1$ is not true. True False
• c) $p$ is the probability of observing a test statistic at least as extreme as the one we got, computed under $H_0$. True False
• d) $p$ tells us the probability that our results happened by random chance. True False

A regression on $n = 50{,}000$ observations gives $\hat\beta = 0.003$ kg/year for a body-fat predictor, with $p < 10^{-6}$. The investigator concludes that the effect is large because the p-value is tiny. The best critique is:

• A A p-value below $10^{-6}$ on a sample this large proves that the linear model is correctly specified, so the conclusion about effect size is logically sound.
• B The effect must be large because the p-value is so small; at sample sizes in the tens of thousands, effect size and statistical significance carry essentially the same information.
• C The p-value is invalid because it falls below the conventional reporting threshold of $0.001$ and should be re-computed with a more conservative correction.
• D A small p-value is evidence the effect is real, but the effect size is the slope, not the p-value; $0.003$ kg/year is practically negligible.

Recall the simple-LR formula $\mathrm{SE}(\hat\beta_1)^2 = \sigma^2 / \sum_i (x_i - \bar x)^2$. Mark each statement about $\mathrm{SE}(\hat\beta_1)$ as true or false.

• a) Doubling $n$ while keeping the spread of $x$ fixed approximately divides $\mathrm{SE}(\hat\beta_1)$ by $2$. True False
• b) Spreading the $x$ values further from $\bar x$ (without changing $n$ or $\sigma^2$) decreases $\mathrm{SE}(\hat\beta_1)$. True False
• c) Adding a second predictor that is highly correlated with $x_1$ generally decreases $\mathrm{SE}(\hat\beta_1)$. True False
• d) Larger $\sigma^2$ (more noise in the response) increases $\mathrm{SE}(\hat\beta_1)$. True False

At a fixed test point $\mathbf{x}_0$, you can construct (a) a 95% confidence interval for $\mathbf{x}_0^\top\boldsymbol\beta$ and (b) a 95% prediction interval for a future $Y$ at $\mathbf{x}_0$. Which of the following correctly contrasts them?

• A The CI captures uncertainty in $\hat{\boldsymbol\beta}$ only; the PI adds the irreducible noise variance $\sigma^2$, so PI is always wider.
• B The CI captures the true individual response; the PI captures the population mean, so PI is wider only when $\mathbf{x}_0$ is far from the data mean.
• C Both intervals carry the same uncertainty; the only difference is the nominal coverage level.
• D The PI is always narrower because it conditions on the observed $\mathbf{x}_0$ rather than averaging over data.

Mark each statement about $R^2$ and $R^2_{\text{adj}}$ in OLS as true or false.

• a) Adding any predictor to an OLS model with intercept never decreases the training $R^2$. True False
• b) Adjusted $R^2$ is monotone non-decreasing in the number of predictors, just like ordinary $R^2$. True False
• c) For simple linear regression with intercept, $R^2$ equals the squared sample correlation $\mathrm{Cor}(\mathbf{x}, \mathbf{y})^2$. True False
• d) An $R^2$ of $0.7$ means the model's predictions are correct $70\%$ of the time. True False

A linear model is fit to $n = 8$ observations with response variance summary $\mathrm{TSS} = 800$. The fitted residuals satisfy $\mathrm{RSS} = 200$. What is $R^2$?

• A $0.20$.
• B $0.25$.
• C $0.75$.
• D $4.00$.

After fitting an OLS regression you produce a QQ plot of standardised residuals against theoretical normal quantiles. The middle of the plot lies on the reference line, but both tails curl strongly upward at the high end and downward at the low end (heavy "S-shape"). Which assumption is most directly called into question?

• A Independence of the errors $\varepsilon_i$ across observations in time or space.
• B Normality of the errors — the residuals look heavy-tailed relative to a Gaussian.
• C Linearity of the conditional mean $E[Y\mid X]$ as a function of the predictors.
• D Constancy (homoscedasticity) of $\sigma^2$ across the full range of fitted values.

In simple linear regression with $n = 5$ predictor values $\mathbf{x} = (1, 2, 3, 4, 10)$, what is the leverage $h_{55}$ of the fifth observation ($x_5 = 10$)? Recall $h_{ii} = \tfrac{1}{n} + (x_i - \bar x)^2 / \sum_j (x_j - \bar x)^2$.

• A $0.20$.
• B $0.72$.
• C $0.92$.
• D $1.20$.

On a residuals-vs-leverage plot, four points are highlighted. Which one is *most* dangerous for the OLS fit (i.e. exerts the largest influence on $\hat{\boldsymbol\beta}$)?

• A A point with low leverage and a large residual.
• B A point with high leverage and a residual close to zero.
• C A point with high leverage and a large residual.
• D A point with low leverage and a residual close to zero.

Two predictors $x_1$ and $x_2$ are highly correlated (sample correlation $\approx 0.99$). Mark each statement about the OLS fit as true or false.

• a) The standard errors of $\hat\beta_1$ and $\hat\beta_2$ are inflated relative to a fit with only one of the two predictors. True False
• b) The point estimates $\hat\beta_1, \hat\beta_2$ become biased. True False
• c) Individual t-tests on $\beta_1$ and $\beta_2$ may both fail to reject even when the joint contribution is highly significant. True False
• d) Predictions $\hat y$ on data similar to the training set are unreliable, because high collinearity propagates the inflated coefficient variance into prediction MSE. True False

You want to test whether *any* of the predictors in a multiple regression with $p = 5$ slopes contributes to predicting $Y$. Which is the appropriate procedure?

• A Pick the smallest individual p-value and report it as the overall test.
• B Compute the residual standard error and reject if it is below $\sigma$.
• C Run an F-test on $H_0:\beta_1 = \cdots = \beta_5 = 0$ before drilling into individual t-tests.
• D Compare $R^2$ between the full model and the intercept-only model and reject if $R^2 > 0.5$.

Earthworm weight $Y$ is modelled on stomach circumference $X$ (continuous) and genus $G \in \{L, N, Oc\}$ (factor; L as reference) via $\hat Y = \hat\beta_0 + \hat\beta_1 X + \hat\beta_2 \mathbf{1}_{G=N} + \hat\beta_3 \mathbf{1}_{G=Oc}$. The fitted equation for genus Oc is:

• A $\hat Y = \hat\beta_0 + \hat\beta_1 X$.
• B $\hat Y = (\hat\beta_0 + \hat\beta_3) + \hat\beta_1 X$.
• C $\hat Y = (\hat\beta_0 + \hat\beta_2) + \hat\beta_1 X$.
• D $\hat Y = (\hat\beta_0 + \hat\beta_2 + \hat\beta_3) + \hat\beta_1 X$.

A categorical predictor takes $K = 4$ levels. Why does R encode it as $3$ dummy columns rather than $4$?

• A Because the column of ones plus $4$ dummies is linearly dependent, so $\mathbf{X}^\top\mathbf{X}$ is singular and the fit is unidentifiable.
• B Because using all $4$ dummies would force the four levels onto an artificial $0$-$1$-$2$-$3$ ordering, biasing the slope estimates of the unordered categories.
• C Because R automatically reserves one design-matrix column for the residual-variance estimate $\hat\sigma^2$ used in standard-error calculations.
• D Because the F-test for the joint significance of this factor uses exactly $K-1$ degrees of freedom by definition, so the design matrix must match.

Mark each statement about interaction terms in linear regression as true or false.

• a) If you fit $y = \beta_0 + \beta_3 (X\cdot Z) + \varepsilon$ without including $X$ and $Z$ as main effects, the model violates the hierarchical (main-effects) principle. True False
• b) In a model with $X$, $Z$ binary, and the $X\cdot Z$ term, the coefficient on $Z$ alone is the $Z=1$-vs-$Z=0$ difference at $X=0$. True False
• c) Adding an interaction lets two groups have different intercepts but forces the slopes to be equal. True False
• d) A small p-value on the interaction term is evidence that the *main effect* of $X$ is non-zero in the population. True False

Hourly temperature is regressed on time of day. Sampling resolution is then refined from hourly to every $5$ minutes (so $n$ grows by $\sim 12\times$). Reported t-statistics for the time slope shoot up; the p-value drops to $<10^{-9}$. The most accurate critique is:

• A Once $n$ exceeds the number of distinct hours in the original data, the OLS point estimate $\hat\beta$ becomes biased and the slope estimate cannot be trusted.
• B An $R^2$ that exceeds $0.95$ on this many points indicates overfitting; the appropriate fix is to reduce model flexibility (drop predictors or use ridge).
• C The Gaussian-error assumption is broken because the response (temperature in Celsius) is bounded both below and above, so the t-distribution does not apply.
• D Adjacent time bins are highly correlated; effective sample size is much smaller than $n$, so the SEs are underestimated and the reported significance is unreliable.

Mark each statement about the residuals-vs-fitted plot as true or false.

• a) A clear cone (fanning) shape indicates the homoscedasticity assumption may be violated. True False
• b) Curvature (a clear U or inverted-U pattern) suggests the linear-mean assumption may be missing a non-linear term. True False
• c) A flat, structureless band of residuals around zero proves that the errors are independent and Gaussian. True False

Which statement best captures the difference between an *error* $\varepsilon_i$ and a *residual* $e_i = y_i - \hat y_i$ in the linear-regression model?

• A Errors are computed from the data after the fit, while residuals are the parameters that define the noise model.
• B Errors and residuals coincide whenever the fitted model contains an intercept term, so the distinction is bookkeeping only.
• C Residuals are unbiased for $\sigma^2$ but errors are biased, which is why the textbook variance estimator divides by $n-p-1$ rather than $n$.
• D Errors are unobservable random variables; residuals are observed predictions of those errors.

Under the linear model $y_i = \mathbf{x}_i^\top\boldsymbol\beta + \varepsilon_i$ with $\varepsilon_i \overset{\text{iid}}\sim N(0, \sigma^2)$, what is the *correct* short justification that the least-squares estimator equals the maximum-likelihood estimator?

• A Because $\hat{\boldsymbol\beta}$ is unbiased under the Gaussian model, and any unbiased estimator of a parameter automatically maximises the joint likelihood.
• B Because $\log L \propto -\sum_i (y_i - \mathbf{x}_i^\top\boldsymbol\beta)^2$ in $\boldsymbol\beta$, so maximising $\log L$ is equivalent to minimising the SSE — the LS objective.
• C Because the OLS closed form $(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}$ exists in one matrix multiplication, and closed-form estimators coincide with MLEs by definition.
• D Because the Gaussian density is symmetric, and any symmetric-loss minimiser is automatically an MLE under the symmetric likelihood.

Differentiating the matrix-form residual sum of squares $\mathrm{RSS}(\boldsymbol\beta) = (\mathbf{y} - \mathbf{X}\boldsymbol\beta)^\top(\mathbf{y} - \mathbf{X}\boldsymbol\beta)$ with respect to $\boldsymbol\beta$ and setting the result to zero gives:

• A $\mathbf{X}^\top\mathbf{X}\,\hat{\boldsymbol\beta} = \mathbf{X}^\top\mathbf{y}$ (the normal equations).
• B $\mathbf{X}\,\hat{\boldsymbol\beta} = \mathbf{y}$, i.e. the fitted values reproduce the observed response exactly with no residual.
• C $\hat{\boldsymbol\beta} = \mathbf{y}^\top\mathbf{X} / (\mathbf{X}^\top\mathbf{X})$, scalar division of the cross-product by the Gram matrix.
• D $\hat{\boldsymbol\beta} = \mathbf{X}\,(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{y}$, with the design matrix premultiplying the inverse.

Let $\mathbf{H} = \mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top$ be the hat matrix in OLS. Mark each statement as true or false.

• a) $\mathbf{H}$ depends only on the design matrix $\mathbf{X}$, not on the response $\mathbf{y}$. True False
• b) The diagonal entries $h_{ii}$ measure leverage; they sum to $n$ (the sample size). True False
• c) $\mathbf{H}$ is symmetric and idempotent ($\mathbf{H}^\top = \mathbf{H}$ and $\mathbf{H}^2 = \mathbf{H}$). True False

A regression of life expectancy on years of education estimates $\hat\beta = 0.6$ years of life per year of schooling, with a small p-value. Which of the following claims is best supported by this OLS fit alone?

• A Holding all other factors fixed, increasing schooling by one year causes the average person's life expectancy to rise by $0.6$ years.
• B Among the predictors that influence life expectancy in this population, years of education is the single dominant driver.
• C In the data, more-educated individuals have on average $0.6$ more years of life expectancy per year of schooling, controlling for whatever else is in the model.
• D Years of schooling and life expectancy are statistically independent at the population level, despite the fitted slope.