Module 06 — PCA, PCR & PLS

In the PCR pipeline, the first principal component $Z_1 = \sum_j \phi_{j1} X_j$ of the standardized predictors is the linear combination that:

• A minimizes the within-cluster sum of squared distances from each $x_i$ to its nearest centroid.
• B maximizes $\text{Cov}(Z_1, Y)$ with $\sum_j \phi_{j1}^2 = 1$.
• C equals the single $X_j$ with the largest sample variance, the other loadings set to $0$.
• D maximizes $\text{Var}(Z_1)$ subject to $\sum_j \phi_{j1}^2 = 1$.

Suppose someone proposed dropping the unit-norm constraint $\sum_j \phi_{j1}^2 = 1$ from the PCA optimization. Why does the constraint exist — what specifically goes wrong without it?

• A Without the constraint, orthogonality between successive PCs breaks, so $Z_1$ and $Z_2$ carry overlapping information and the decorrelation guarantee that PCR relies on for stable downstream OLS is lost.
• B Without the constraint the optimization is unbounded — scaling $\phi_1$ by any positive constant inflates $\text{Var}(\phi_1^\top x)$ without bound, so "maximize variance" has no finite solution.
• C Without the constraint, PCA loses invariance to rotations of the coordinate system, so different column orderings of $X$ would yield genuinely different first principal components on the same dataset.
• D Without the constraint, the loadings $\phi_{j1}$ stop being interpretable as the correlations between $Z_1$ and the original $X_j$, which breaks the standard loading-as-correlation reading used in PCA biplots.

Mark each statement about principal component analysis as true or false.

• a) PCA is scale-invariant: rescaling one column by a factor of 1000 does not change the loadings. True False
• b) The squared loadings of a single principal component sum to 1: $\sum_j \phi_{jm}^2 = 1$. True False
• c) Principal components are unique only up to a sign flip; opposite-sign loadings from two software packages describe the same component. True False
• d) PCA constructs components using the response variable $Y$. True False
• e) Computing all $p$ principal components and keeping every one of them is itself a form of dimensionality reduction. True False

PCA is sometimes called a "rotation." In what precise sense is this language exact, and where does it become merely useful shorthand?

• A PCA is exactly a rotation only on standardized data; on unstandardized data it becomes a stretching operation along the standard-deviation axis of each column, so the rotation language fails whenever variances differ across predictors.
• B The "rotation" is around the data centroid, applied to each observation in turn — after PCA the points sit at new coordinates while the original axes stay fixed in physical space.
• C Keeping all $p$ PCs is an exact orthonormal change of basis; truncating to $M < p$ adds the projection step. "Rotation" is shorthand for the whole rotate-then-truncate pipeline.
• D The rotation is exact only when no two eigenvalues are tied; ties between eigenvalues collapse the orthonormal basis into a degenerate form and break the rotation interpretation.

PCA is performed on a dataset with 6 standardized predictors. The eigenvalues of the sample covariance are:

$\lambda_1 = 3.0,\ \lambda_2 = 1.5,\ \lambda_3 = 0.8,\ \lambda_4 = 0.4,\ \lambda_5 = 0.2,\ \lambda_6 = 0.1.$

How many principal components must you keep to retain at least 90% of the total variance?

• A 4 components
• B 3 components
• C 5 components
• D 6 components

Continuing the setup: the loading vector for PC1 is $\phi_1 = (0.7,\ 0.5,\ 0.4,\ 0.3,\ 0.1,\ 0.0)^\top$. For an observation with standardized values $x = (1,\ -1,\ 0.5,\ 0,\ 2,\ -2)$, what is the score $z_1 = \phi_1^\top x$?

• A $1.6$
• B $0.6$
• C $2.0$
• D $0.7$

PCA on a centered dataset gives PC1 with variance $\lambda_1 = 7.2$, while the total variance of the data is $9.6$. What is the proportion of variance explained by PC1?

• A $0.50$
• B $0.25$
• C $0.75$
• D $0.87$

Suppose your $n$ observations lie almost exactly on a unit circle in $\mathbb{R}^2$, so the underlying structure is one-dimensional and curved. What does PCA report — and what does this tell you about its scope?

• A A single dominant PC capturing the circular structure exactly, since the data lie on an intrinsically 1-D manifold and PCA finds the manifold.
• B A first PC that traces along the circle non-linearly, with the second eigenvalue essentially zero — so the scree plot peaks sharply at PC1.
• C A flat scree plot with both eigenvalues equal to zero, since the data have no linear variance and PCA returns the null direction.
• D Two PCs of roughly equal variance — PCA's linear projections can't follow a curve, so the algorithm reports the data as effectively 2-D.

Mark each statement as true or false.

• a) Standardizing predictors before PCA / PCR is recommended whenever the variables are measured in different units. True False
• b) PCR is scale-invariant: rescaling one predictor does not change the fitted PCR model. True False
• c) After back-transforming PCR coefficients to original-$X$ space, all $\hat\beta_j$ are typically nonzero — even after $M < p$ components are kept. True False
• d) Without standardization, the variable with the largest unit-magnitude tends to dominate PC1 purely because its raw numbers are bigger. True False

You run PCA on four standardized macroeconomic predictors ($X_1$ = GDP growth, $X_2$ = interest rate, $X_3$ = inflation, $X_4$ = unemployment) and obtain the first PC loadings $\phi_1 = (0.7,\ -0.6,\ 0.3,\ -0.2)^\top$. Which is the most accurate reading of PC1?

• A An "economic-activity" axis: high PC1 corresponds to high GDP growth and low interest rates, with smaller contributions from inflation and unemployment.
• B A weighted-average axis of all four variables, since the loadings are similar in absolute value and the unit-norm constraint forces near-equal weighting.
• C The response that PCR will regress $Y$ onto, since the loadings have been chosen to maximize $\text{Cov}(\phi_1^\top X, Y)$.
• D A three-variable axis: the negative loading on $X_2$ means interest rate has been excluded from the PCA, leaving GDP, inflation, and unemployment.

Why is standardization mandatory for PCA / PCR / PLS but only cosmetic for OLS? Pick the explanation that most directly captures the asymmetry.

• A PCA's variance objective is unit-dependent — rescaling $X_j$ by $c$ multiplies its variance by $c^2$ and changes the PC ranking. OLS's $\hat\beta_j$ rescales by $1/c$ in compensation, so the fitted $\hat y$ is unchanged.
• B OLS uses an internal weighted-least-squares pass that adjusts for scale differences across columns of $X$, while PCA's eigendecomposition of the sample covariance has no such compensation step and so inherits raw-unit sensitivity.
• C OLS is fit by maximum likelihood under a Gaussian model, and Gaussian likelihoods are unit-invariant under linear rescaling by construction; PCA uses a quadratic-form covariance metric whose value depends on the absolute scale of each column, so it is not.
• D PCA's eigendecomposition only converges when all predictors share common units, otherwise the iteration on the covariance matrix fails to a numerical limit; OLS solves the closed-form normal equations directly and so handles mixed units natively.

Which of the following describes the canonical PCR procedure with $M$ chosen by cross-validation?

• A Run PCA on the raw unstandardized $X$, then standardize the resulting components to unit variance, and fit OLS of $Y$ on the first $M$ standardized components.
• B Standardize $Y$, run PCA on the standardized response, then regress each $X_j$ on the first $M$ PCs of $Y$ and average the coefficients.
• C Fit OLS of $Y$ on $X$, run PCA on the residuals, and keep the first $M$ residual components as the new predictors in a refit.
• D Standardize $X$, run PCA, then for each $M = 1, \ldots, p$ fit OLS of $Y$ on the first $M$ PCs and pick the $M$ minimizing CV-MSE.

What is the relationship between PCA and PCR?

• A PCR is the supervised version of PCA: PCR maximizes $\text{Var}(Z \mid Y)$ to align the rotation with the response, whereas PCA uses unconditional $\text{Var}(Z)$.
• B PCA is unsupervised dimension reduction; PCR follows PCA with an OLS regression of $Y$ on the first $M$ components.
• C PCA produces orthogonal components but cannot regress on them; PCR is the modification adding an $L_2$ penalty on the PC scores to absorb collinearity.
• D PCR uses $\text{Cov}(X, Y)$ to choose its components, whereas PCA uses $\text{Var}(X)$ alone with no reference to $Y$.

The prof framed PCR as "a discretized version of ridge regression." Which sentence best captures the analogy?

• A Both methods pressure the small-eigenvalue PC directions; PCR drops them past the cutoff $M$, while ridge shrinks them smoothly via $\lambda_j^2 / (\lambda_j^2 + \lambda)$, heaviest on the smallest eigenvalues.
• B Both methods solve the same penalized least-squares problem on the original predictors; ridge writes it in its standard Lagrangian form, while PCR is the equivalent budget-constraint form $\sum_j \beta_j^2 \leq M$ with $M$ playing the role of the budget.
• C PCR equals ridge regression exactly in the limit $\lambda \to 0$; outside that limit the two methods are mathematically unrelated and their fitted coefficients behave differently as functions of the tuning parameter.
• D Both methods perform exact variable selection on the original $X$ by zeroing out the smallest original-$X$ coefficients $\hat\beta_j$ once their estimated effect drops below a threshold set by the tuning parameter.

Both ridge and PCR put pressure on the small-eigenvalue PC directions. Why does this happen — what is the deeper mathematical reason these two methods end up doing similar things?

• A Both methods are penalized least-squares estimators sharing a common $L_2$ regularizer applied directly to the original-$X$ coefficients $\hat\beta_j$, with the tuning parameter ($\lambda$ for ridge, $M$ for PCR) controlling the strength of the same shared penalty term.
• B Small-eigenvalue PC directions are where OLS estimates are unstable — their variance scales as $\sigma^2 / \lambda_j$. Both methods damp this: ridge smoothly via $\lambda_j^2 / (\lambda_j^2 + \lambda)$, PCR by hard truncation past $M$.
• C Both methods preprocess $X$ via singular-value decomposition and then solve the downstream regression on the resulting orthogonal columns, which gives them mathematically identical bias profiles for any choice of their tuning parameters.
• D Both methods are limiting cases of lasso along the elastic-net family: ridge as $\alpha \to 0$ in the standard elastic-net mixing parameter, and PCR as $\alpha \to 1$ but with an additional hard-thresholding step applied to the principal components.

Mark each statement about principal-components regression (PCR) as true or false.

• a) PCR performs variable selection: choosing $M < p$ zeroes out the original-$X$ coefficient $\hat\beta_j$ for each dropped predictor. True False
• b) PCR can be seen as a discretized version of ridge regression. True False
• c) After back-transforming, the original-$X$ coefficient is $\hat\beta_j = \sum_{m=1}^{M} \hat\theta_m \phi_{jm}$, where $\hat\theta_m$ are the regression coefficients on the components. True False
• d) PCR uses the response $Y$ when constructing the principal components. True False
• e) When $M = p$, PCR coincides with OLS on the original standardized predictors. True False

On the Credit dataset, the prof reported that PCR needed $M \approx 10$ components (out of $\sim 11$) to perform well, while ridge regression performed comparably keeping all directions but shrinking. Why doesn't ridge suffer the same failure?

• A Ridge places its hyperparameter $\lambda$ on the $L_1$ norm of $\hat\beta$, which automatically picks the right small set of components for the response.
• B Ridge runs an internal cross-validation pass for each predictor separately, so it can adapt to which raw $X_j$ drives $Y$ regardless of the predictor's $X$-variance and re-fit its shrinkage strength column-by-column on the Credit data.
• C Ridge keeps every PC direction, just shrinking the small-eigenvalue ones — a low-$X$-variance but $Y$-predictive direction is retained at smaller magnitude, while PCR's hard truncation drops it.
• D Ridge applies the same shrinkage factor to every direction, equalizing their importance across the eigen-spectrum and so picking up small-$X$-variance directions automatically alongside the dominant high-variance ones in the Credit fit.

On the Credit dataset, the prof reported that PCR's CV-MSE was minimized at $M = 10$ out of $\sim 11$ available components — almost no dimension reduction. What does this tell you?

• A PCR overfit the Credit data because $M$ was chosen too close to $p$, so the cross-validated test error climbed at large $M$ — the textbook overfitting signature of a model that retained too many components for its sample size.
• B The high-variance directions of $X$ were not the directions driving $Y$; income was a low-$X$-variance but predictive feature, so PCR needed many components before that signal entered.
• C The Credit predictors are so strongly collinear that each PC captures only a fragment of the $Y$-relevant direction, so $M$ has to be large to reassemble the signal.
• D PCA was run without standardization, so one large-unit predictor dominated the leading PCs and the rest of the components were needed to recover the signal.

You run PCR with $M = 2$ on three standardized predictors. The first two loading vectors are $\phi_1 = (0.7,\ 0.5,\ 0.5)^\top$ and $\phi_2 = (-0.3,\ 0.6,\ 0.7)^\top$. The OLS regression of $Y$ on the components gives $\hat\theta_1 = 4$ and $\hat\theta_2 = 2$. What is the back-transformed coefficient $\hat\beta_2$ on the second original predictor?

• A $1.2$
• B $2.0$
• C $1.1$
• D $3.2$

PCR with $M = 1$ keeps only one component out of $p = 20$, producing a model with effectively one feature. Why is this still not a form of variable selection?

• A Variable selection requires explicit hypothesis testing or $p$-value thresholding on each individual $X_j$, machinery PCR omits entirely in favour of unsupervised variance-driven thresholds at the component level.
• B $M = 1$ produces a model too simple to count as having identified any predictors, regardless of which $X_j$ end up in the back-transform — no method with only one degree of freedom can be doing real selection.
• C PCR uses an $L_2$ penalty internally on the principal-component coefficients, and $L_2$ regularization cannot drive coefficients exactly to zero in finite samples — so no predictor is ever eliminated from the fitted model.
• D The single component is a linear combination of all $p$ predictors, so the back-transform $\hat\beta_j = \hat\theta_1 \phi_{j1}$ is nonzero on every $X_j$ — no predictor has been excluded.

What is the principled way to choose the number of components $M$ in PCR?

• A Pick the $M$ at the elbow of the scree plot of $X$, since PCA's diagnostic identifies the natural variance cutoff.
• B Set $M = p$ to retain 100% of the variance in $X$ and let OLS handle the rest of the variance budget.
• C Cross-validate the downstream regression on $Y$ and pick the $M$ at the CV-MSE minimum, optionally via the 1-SE rule.
• D Pick the smallest $M$ for which the cumulative PVE in $X$ reaches a threshold like 95%, since that captures most of the input variance.

Why does PCR with $M = p$ produce exactly the same fitted values $\hat y$ as OLS on the standardized predictors?

• A A rotation to all $p$ orthogonal PC directions preserves $\text{col}(X)$, so fitting $Y$ on the rotated columns gives the same projection of $Y$ onto $\text{col}(X)$ that OLS does.
• B With $M = p$, PCR's tuning parameter effectively equals zero, so the discretized-ridge analogy means it coincides with ridge at $\lambda = 0$, which itself equals OLS.
• C When $M = p$, the back-transform formula $\hat\beta_j = \sum_m \hat\theta_m \phi_{jm}$ collapses to the closed-form lasso solution because the loadings span all of $\mathbb{R}^p$.
• D When $M = p$, the back-transform sums over all $p$ components and so multiplies the OLS fit by an extra $\Phi^\top \Phi$ rotation factor, but the orthogonality of $\Phi$ cancels it.

Mark each statement as true or false.

• a) Lasso can produce exact zeros in $\hat\beta$; PCR generally cannot, even after back-transform. True False
• b) Ridge regression and PCR both place heavier pressure on small-eigenvalue PC directions; ridge shrinks smoothly while PCR truncates. True False
• c) Faced with two nearly perfectly correlated predictors, lasso typically keeps both with shared weight; ridge typically drops one and zeroes the other. True False
• d) Ridge, PCR, and PLS are good choices when predictive accuracy matters and you do not need to identify which raw variables drive $Y$. True False

In partial least squares (PLS) regression, the first component $Z_1 = \sum_j \phi_{j1} X_j$ is constructed to maximize:

• A $\text{Var}(Z_1)$ subject to $\sum_j \phi_{j1}^2 = 1$.
• B $\sum_j |\phi_{j1}|$ subject to a fixed $\text{Var}(Z_1)$.
• C the joint log-likelihood of $(X, Y)$ under a Gaussian model.
• D $\text{Cov}(Z_1, Y)$ subject to $\sum_j \phi_{j1}^2 = 1$.

In the PLS algorithm, the loading $\phi_{j1}$ for the first component is set to:

• A the $j$-th component of the eigenvector of $X^\top X$ corresponding to the largest eigenvalue, normalized to unit length.
• B the OLS coefficient from the simple regression of $Y$ on $X_j$ alone, which is proportional to $\text{Corr}(Y, X_j)$.
• C $1/p$ for every $j$, so PLS averages all $p$ predictors with equal weight when forming its first component $Z_1$.
• D zero for every $j$ except the predictor with the largest sample variance, with that single loading set to $1$.

PLS sets $\phi_{j1}$ proportional to the simple-regression coefficient of $Y$ on $X_j$. What is the intuition — why is this rule the right one for the supervised criterion PLS optimizes?

• A Simple regression of $Y$ on each $X_j$ is the closed-form solution to PCA's variance-maximization problem when the unit-norm constraint $\sum_j \phi_{j1}^2 = 1$ is dropped, and PLS adopts that closed-form because it inherits PCA's optimization directly with no further modification.
• B The rule is required for the unit-norm constraint $\sum_j \phi_{j1}^2 = 1$ to hold automatically without an explicit normalization step in the algorithm — simple-regression coefficients on standardized predictors already form a unit vector by construction.
• C To maximize $\text{Cov}(Z_1, Y)$ you weight each $X_j$ by how strongly it alone predicts $Y$, and the simple-regression coefficient of $Y$ on $X_j$ is exactly that relevance metric ($\propto \text{Corr}(Y, X_j)$).
• D Simple-regression coefficients are unit-invariant under linear rescaling of each column, so weighting by them makes PLS fully scale-invariant — unlike PCA, this removes any need to standardize the predictors before fitting the algorithm.

Mark each statement about partial least squares (PLS) as true or false.

• a) PLS uses the response $Y$ when constructing components. True False
• b) Like PCR, PLS yields a model whose back-transformed $\hat\beta_j$ are typically nonzero on every original predictor. True False
• c) PLS performs explicit variable selection by zeroing out small loadings. True False
• d) Successive PLS components are orthogonal by construction; the algorithm deflates $X$ before computing the next component. True False

You are predicting $Y$ from standardized predictors. Through prior knowledge you know that $Y$ depends strongly on a particular predictor $X_3$ that happens to have a small variance relative to the other predictors. Compared with PCR, PLS will generally:

• A throw $X_3$ away earlier than PCR, since PLS prioritizes high-variance directions of $X$ and drops low-variance ones outright at each step.
• B behave identically to PCR on $X_3$, since both methods are unsupervised dimension-reduction tools that only see the design matrix $X$ when constructing components and so weight each column by exactly the same variance-driven criterion.
• C weight $X_3$ identically to PCR, since both methods rank predictors by sample variance after standardization brings every $X_j$ to unit variance.
• D weight $X_3$ heavily in $Z_1$ via its large simple-regression coefficient on $Y$, while PCR pushes $X_3$ into a late component because its $X$-variance is small.

Mark each statement about PLS's empirical behavior as true or false.

• a) PLS generally outperforms ridge and PCR in test MSE because it uses the response $Y$. True False
• b) Supervised dimension reduction can reduce bias relative to PCR but may also increase variance, so PLS does not strictly dominate. True False
• c) PLS, PCR, and ridge tend to behave similarly in practice; ridge is often preferred because it shrinks smoothly rather than in discrete steps. True False
• d) Like PCR, PLS produces back-transformed $\hat\beta_j$ that are linear combinations of the original predictors, with no exact zeros. True False

Mark each statement about PCA, PCR, PLS, ridge, and lasso as true or false.

• a) Ridge regression and PCR both struggle with collinearity; only lasso handles it. True False
• b) Lasso typically produces exact zeros in $\hat\beta$; ridge typically does not; PCR (after back-transform) typically does not either. True False
• c) PLS uses $Y$ to choose its components; PCA does not. True False
• d) When the truth is sparse (only a few raw $X_j$'s drive $Y$), lasso is generally preferred to PCR / PLS / ridge. True False
• e) PCA is a special case of PLS with $Y = X$, since both methods maximize the covariance with the response. True False

Two analyses fit the same regression: one with PCR ($M = 4$), one with ridge (with $\lambda$ chosen by CV). They report essentially identical CV-MSE. The prof would lean toward reporting which, and why?

• A PCR — keeping $M = 4$ components means a more parsimonious model than ridge's full-rank fit on every direction, and Occam's razor favours the smaller model when two candidates tie on cross-validated test error.
• B Ridge — its $\lambda$ moves continuously, so a small CV reshuffle doesn't flip the model; PCR's $M$ jumps in unit steps and can swing $\hat\beta$ across resamples.
• C PCR — the $M = 4$ representation is a discrete, easily reportable choice and so is more interpretable than a continuous $\lambda$.
• D PCR — its hard truncation produces a sparser back-transformed $\hat\beta$ in original-$X$ space than ridge does, giving a cleaner variable-importance story.

You are predicting credit-card spend from $p = 50$ candidate predictors, many of which are nearly collinear. A stated goal is interpretability: you want to report which raw variables drive $Y$. Which method best fits the goals?

• A Ridge regression: handles collinearity by shrinking smoothly and sharing weight across correlated predictors.
• B PCR: rotates correlated predictors into orthogonal PCs and drops the small-variance ones via the $M$ cutoff.
• C Lasso: handles collinearity by zeroing out non-driving predictors, leaving an interpretable subset of raw variables.
• D PLS: same skeleton as PCR but supervised, using $\text{Cov}(X, Y)$ to pick directions while still rotating predictors.

Mark each statement about choosing $M$ and reading PCA diagnostics as true or false.

• a) When PCA is the front end of a supervised regression (PCR), $M$ should be chosen by cross-validating the downstream regression rather than by the scree elbow. True False
• b) The "elbow" of a scree plot is determined by a unique well-accepted procedure that always produces the same $M$. True False
• c) Cumulative PVE is monotonically non-decreasing in $M$. True False
• d) If the scree plot is roughly flat — every PC has similar PVE — PCA has revealed strong low-dimensional structure in the data. True False

Which of the following is best classified as variable selection rather than dimensionality reduction?

• A Lasso with $\lambda > 0$.
• B PLS with $M = 2$.
• C An autoencoder mapping $\mathbb{R}^{100} \to \mathbb{R}^{10}$.
• D LDA as a $K-1$ projection.

When the true response depends on a small subset of the original predictors, which method's signature failure is to scatter that signal across many components, leaving no clean variable interpretation in the back-transformed coefficients?

• A Lasso, because the $L_1$ penalty enforces sparsity in $\hat\beta$ that mirrors the sparse-truth assumption directly.
• B Ridge regression, because its $L_2$ penalty shrinks every original-$X$ coefficient toward zero by the same proportional factor, blurring the boundary between the truly active subset and the truly null variables across the whole coefficient vector.
• C PCR, because every PC is a linear combination of all original predictors, so the back-transformed $\hat\beta_j$ are generically all nonzero.
• D Best-subset selection, because enumerating every subset spreads the variance across many candidate models and obscures the sparse truth.