Module 04 — Classification

In the classification setup, what is the Bayes error rate?

• A The training error of a fitted Bayes-network classifier on the data at hand, evaluated under the $0/1$ loss.
• B The classification analogue of the residual sum of squares: $\sum_i \mathbf 1(y_i \neq \hat y_i)$ counted on the training set.
• C The expected $0/1$ loss of the classifier $\hat y(x) = \arg\max_k \Pr(Y=k\mid X=x)$ that uses the true posterior, i.e. $1 - E[\max_k \Pr(Y=k\mid X)]$.
• D The misclassification rate of the naive-Bayes classifier when its conditional-independence assumption is satisfied.

On average, what fraction of credit-card holders with odds $0.37$ of defaulting will in fact default?

• A $0.37$
• B $0.63$
• C $0.59$
• D $0.27$

An individual has a $16\%$ chance of defaulting on her credit-card payment. What are the odds that she will default?

• A $0.16$
• B $0.84$
• C $0.19$
• D $5.25$

A logistic model for $Y = I(\text{student gets an A})$ has $\hat\beta_0 = -6$, $\hat\beta_1 = 0.05$ (study hours), $\hat\beta_2 = 1$ (GPA). For a student with $40$ hours and GPA $3.5$, what is $\hat p$?

• A $\hat p \approx 0.38$
• B $\hat p \approx 0.5$
• C $\hat p \approx 0.62$
• D $\hat p \approx 0.73$

Same model as Question 4 ($\hat\beta_0 = -6$, $\hat\beta_1 = 0.05$, $\hat\beta_2 = 1$). At GPA $3.5$, how many study hours give $\hat p = 0.5$?

• A $40$ hours
• B $90$ hours
• C $55$ hours
• D $50$ hours

A logistic model for default uses SEX (1 = male, 2 = female), PAY_0, and the interaction SEX:PAY_0. Output gives $\hat\beta_{\text{PAY}_0} = 0.80$ and $\hat\beta_{\text{SEX:PAY}_0} = 0.15$. Holding other covariates fixed, by what factor do the odds of default change for one additional month of delayed payment, separately for males and females?

• A Both groups: $e^{0.80} \approx 2.23$.
• B Males: $e^{0.80} \approx 2.23$; females: $e^{0.80 + 0.15} \approx 2.59$.
• C Males: $e^{0.80} \approx 2.23$; females: $e^{0.15} \approx 1.16$.
• D Both groups: $e^{0.80 + 0.15} \approx 2.59$.

Mark each statement about logistic regression as true or false.

• a) For a one-unit increase in $x_j$ (no interaction), $\beta_j$ gives the additive change in the probability of $Y=1$. True False
• b) The maximum-likelihood estimate $\hat{\boldsymbol\beta}$ has a closed-form solution analogous to OLS's $(X^\top X)^{-1}X^\top y$. True False
• c) Flipping the reference class (encoding the other class as 1) flips the sign of every coefficient. True False
• d) Severe collinearity among the predictors can leave the MLE non-unique, the same failure mode as in OLS. True False

A KNN classifier with $K=35$ on credit-default data gives the test confusion matrix below (rows = true, columns = predicted):

$\begin{array}{c|cc} & \hat y = 0 & \hat y = 1 \\\hline y = 0 & 1820 & 60 \\ y = 1 & 380 & 240 \end{array}$

What are the sensitivity and specificity (positive class = default = 1)?

• A Sensitivity $\approx 0.39$, specificity $\approx 0.97$.
• B Sensitivity $\approx 0.97$, specificity $\approx 0.39$.
• C Sensitivity $\approx 0.80$, specificity $\approx 0.83$.
• D Sensitivity $\approx 0.39$, specificity $\approx 0.86$.

Mark each statement about sensitivity and specificity as true or false.

• a) Sensitivity $= TP/(TP+FN)$ is the fraction of true positives that the classifier successfully catches. True False
• b) Lowering the classification threshold from $0.5$ to $0.2$ generally increases sensitivity and decreases specificity. True False
• c) The quantity $TP/(TP+FP)$ equals the sensitivity. True False
• d) Under heavy class imbalance with $95\%$ negatives, a "predict 0 always" rule has accuracy $\approx 0.95$, sensitivity $0$, specificity $1$. True False

Which statement most accurately describes a ROC curve and AUC for a binary classifier producing scores $\hat p(x)$?

• A The $y$-axis is sensitivity, the $x$-axis is specificity, and the curve is traced by sweeping the classification threshold; AUC near $1$ indicates a good classifier.
• B The $y$-axis is sensitivity, the $x$-axis is $1-$specificity; the curve sweeps the threshold; AUC equals the probability that a random positive scores higher than a random negative.
• C The curve plots overall accuracy against the threshold value; AUC is the area under that curve and equals exactly $1$ when the classifier is perfect.
• D The curve plots precision against recall as the threshold sweeps; an AUC near $0$ means the classifier carries essentially no useful information about class.

Mark each statement about ROC curves and AUC as true or false.

• a) A classifier with AUC $= 0.5$ is no better than random guessing. True False
• b) A classifier with AUC $= 0.2$ is worse than random and contains no useful information. True False
• c) For a fixed dataset, comparing two classifiers by AUC is independent of any specific threshold. True False

In LDA, why is the discriminant $\delta_k(x)$ linear in $x$, while in QDA it is quadratic?

• A In LDA the priors $\pi_k$ are required to be equal across classes, while in QDA the priors differ, and this difference contributes the quadratic term in the discriminant.
• B LDA assumes the class-conditionals are Gaussian, while QDA drops this assumption and replaces $f_k(x)$ with a kernel density estimate, which makes the discriminant locally quadratic in $x$.
• C LDA writes the discriminant in terms of $\Sigma$ while QDA writes it in terms of $\Sigma^{-1}$, and the squared inverse turns the linear cross-term into a quadratic form in $x$.
• D Pooling $\Sigma$ in LDA makes the term $-\tfrac12 x^\top\Sigma^{-1}x$ independent of $k$ and it cancels in $\arg\max_k$; QDA keeps a class-specific $\Sigma_k$, so $-\tfrac12 x^\top\Sigma_k^{-1}x$ depends on $k$ and survives.

Two-class LDA with $\mu_A = (1,1)^\top$, $\mu_B = (3,3)^\top$, shared covariance $\Sigma = 2I$, equal priors $\pi_A = \pi_B = 0.5$. The decision boundary is:

• A $x_1 + x_2 = 2$
• B $x_1 + x_2 = 4$
• C $x_1 - x_2 = 0$
• D $x_1^2 + x_2^2 = 8$

In 1D LDA with two classes, $\mu_1 = 2$, $\mu_2 = 8$, shared variance $\sigma^2 = 9$, and equal priors. The decision threshold $x^\star$ where $\delta_1(x) = \delta_2(x)$ is:

• A $x^\star = 5$
• B $x^\star = (\mu_1 + \mu_2)/2 - \tfrac{\sigma^2}{\mu_2 - \mu_1} \approx 3.5$
• C $x^\star = \tfrac{8 \cdot 2 + 2 \cdot 8}{9} \approx 3.6$
• D $x^\star = \sqrt{2 \cdot 8} \approx 4$

In LDA, you raise the prior $\pi_A$ from $0.5$ to $0.7$ (and lower $\pi_B$ to $0.3$) without changing the means or the shared covariance. What happens to the decision boundary between classes $A$ and $B$?

• A It shifts in space away from $\mu_A$ and toward $\mu_B$, so more of the input space is classified as class $A$.
• B Its intercept moves by $\log(\pi_B/\pi_A)$ but its orientation is unchanged, so equal volumes of the input space are still classified as $A$ and $B$ — only the location of the divide drifts symmetrically.
• C It is unchanged, since the LDA boundary depends only on the class means $\mu_k$ and the pooled covariance $\Sigma$.
• D It rotates around the midpoint $(\mu_A + \mu_B)/2$ by an angle proportional to $\log(\pi_A/\pi_B)$.

Mark each statement comparing LDA and QDA as true or false.

• a) When the true Bayes boundary is linear, QDA tends to give lower test error than LDA on small samples. True False
• b) On the training set, QDA's misclassification rate is generally no higher than LDA's, because QDA is strictly more flexible. True False
• c) As the sample size $n$ grows, QDA's relative test performance versus LDA tends to improve when the true class covariances differ. True False
• d) For fixed $p$, both LDA and QDA estimate the same number of covariance parameters per class. True False

For $p = 10$ predictors and $K = 3$ classes, how many free covariance parameters does QDA estimate (counting only the class covariance matrices, not means or priors)?

• A $55$
• B $100$
• C $300$
• D $165$

What is the defining modeling assumption of (Gaussian) naive Bayes?

• A The predictors $X_1, \dots, X_p$ are unconditionally independent of each other in the population, regardless of the class label.
• B The class prior $\pi_k$ is assumed uniform across the $K$ classes, so that all classification information comes from the class-conditional density.
• C The predictors $X_1, \dots, X_p$ are conditionally independent given the class; in the Gaussian case, this is equivalent to QDA with each $\Sigma_k$ restricted to be diagonal.
• D The Bayes posterior is computed using a directed-graphical Bayesian network rather than via the Bayes-classifier $\arg\max_k$ abstraction.

Mark each statement about the diagnostic vs sampling (generative) paradigm as true or false.

• a) Logistic regression and KNN sit in the diagnostic paradigm; LDA, QDA, and naive Bayes sit in the sampling/generative paradigm. True False
• b) The sampling paradigm models $f_k(x) = \Pr(X \mid Y = k)$ and $\pi_k$, then recovers $\Pr(Y \mid X)$ via Bayes' theorem. True False
• c) Logistic regression and LDA both produce a log-odds that is linear in $x$, but estimate the parameters differently (direct MLE vs Gaussian-plug-in). True False
• d) Naive Bayes belongs to the diagnostic paradigm because it directly estimates the class probabilities by counting. True False

A 7-point training set with two predictors and labels $\{A, A, A, B, B, B, B\}$. We want to predict the class at $x_0 = (1,2)$ using KNN. The prof's exercise asks why $K = 7$ is a poor choice. The best explanation is:

• A With $K = 7$ on a $7$-point training set, every training point is a neighbor; the prediction equals the global majority class and entirely ignores the location of $x_0$.
• B With $K = 7$ the classifier overfits drastically because it uses too many neighbors and chases noise in the training labels.
• C A choice of $K = 7$ on a $7$-point training set produces ties and leaves the majority-vote prediction mathematically undefined here.
• D $K = 7$ is fine in principle; the actual issue is that with only $7$ training points the local-neighbourhood estimate of $\Pr(Y\mid X)$ has too high variance to use any KNN at all here.

Mark each statement about KNN classification as true or false.

• a) Small $K$ (e.g. $K = 1$) gives high variance and low bias; large $K$ gives the reverse. True False
• b) If the true Bayes boundary is highly nonlinear, the optimal $K$ tends to be large (smoother fit). True False
• c) Standardizing predictors before KNN is recommended because Euclidean distance is sensitive to scale differences across features. True False
• d) Choosing $K$ by cross-validation on the test set is a valid procedure for honest test-error estimation. True False

Observations are uniformly distributed on the unit hypercube $[0,1]^p$. To predict at a new point we use only training observations whose coordinates each lie within a $10\%$ window centred on the test point's coordinate. As $p$ grows from $1$ to $100$, the average fraction of training points actually used shrinks roughly as:

• A $0.1$ for any $p$, since each coordinate independently keeps the same $10\%$ window around the test point.
• B $0.1^p$ — exponentially toward zero, so at $p = 100$ essentially no training points qualify as "near".
• C $0.1 \cdot p$ — linearly growing, so the curse of dimensionality only kicks in at very large $p$.
• D $1 - 0.1^p$ — most points are nearby in high dimensions because the unit-cube volume concentrates near its centre.

Mark each statement about high-dimensional classification as true or false.

• a) Of the methods covered in module 4, KNN is the most strongly affected by the curse of dimensionality. True False
• b) Standardizing the predictors removes the curse of dimensionality. True False
• c) Naive Bayes is often preferred to QDA when $p$ is large because its parameter count is $O(p)$ rather than $O(p^2)$. True False

In the credit-default data, only about $5\%$ of clients default. A trivial classifier that predicts "no default" for every client achieves around $95\%$ accuracy. The most useful conclusion to draw from this is:

• A Accuracy is misleading under class imbalance; sensitivity is $0$ and specificity is $1$, so the classifier is useless for catching defaulters.
• B Accuracy is the right summary metric here; any candidate classifier whose accuracy is below this $95\%$ baseline should simply be discarded.
• C The trivial classifier has good sensitivity but poor specificity, since it never raises false alarms but also catches every actual default.
• D The trivial classifier's $95\%$ accuracy reflects a genuinely high specificity at the chosen threshold; on this dataset the operating point is simply too aggressive on negatives, and dialling the threshold down to $0.2$ would recover useful sensitivity from the same model.

For an iris-like 2-predictor / 3-class dataset, you observe the following error rates:

$\begin{array}{lcc} \text{Method} & \text{Train error} & \text{Test error} \\\hline \text{LDA} & 0.19 & 0.17 \\ \text{QDA} & 0.17 & 0.32 \end{array}$

Which method should you prefer, and why?

• A QDA: it has the lowest training error, which is the most reliable indicator of generalisation.
• B QDA: it is strictly more flexible, so any apparent training-error advantage will translate to test-error advantage with enough regularisation.
• C Either; the difference between $0.17$ and $0.32$ is too small to matter without a formal hypothesis test.
• D LDA: its test error is lower, suggesting the extra parameters in QDA hurt more (variance) than they help (bias) on this sample size.

A 1D LDA setup: companies that issued a dividend ($Y = 1$) had $\bar X = 10$, those that did not had $\bar X = 0$, with shared $\hat\sigma^2 = 36$. The prior is $\pi_1 = 0.8$. Assuming Gaussian class-conditionals, what is $\Pr(Y = 1 \mid X = 4)$?

• A $\approx 0.20$
• B $\approx 0.50$
• C $\approx 0.75$
• D $\approx 0.94$

Mark each statement about LDA's class-conditional density assumption as true or false.

• a) LDA assumes $X \mid Y = k$ is multivariate normal with class-specific mean $\mu_k$ and a covariance $\Sigma$ that is the same across classes. True False
• b) The pooled covariance estimator is $\hat\Sigma = \sum_k \frac{n_k - 1}{n - K}\hat\Sigma_k$. True False
• c) The LDA decision rule depends on $\mu_0 - \mu_1$ alone; the covariance $\Sigma$ does not affect the boundary's orientation. True False
• d) When the Gaussian-with-shared-$\Sigma$ assumption is correct, LDA is the maximum-likelihood Bayes-optimal classifier. True False

Two competing classifiers $p(x)$ and $q(x)$ produce probabilities of disease for the same population. On independent validation sets, $p(x)$ has AUC $0.6$ and $q(x)$ has AUC $0.7$. Which method would you prefer, and why?

• A $q(x)$, but only after the operating threshold is fixed: an AUC gap of $0.1$ is meaningless until you read off the corresponding sensitivities at one common threshold.
• B $q(x)$: higher AUC means a larger probability that a random positive scores higher than a random negative, summarising better trade-offs across all thresholds.
• C Cannot tell: AUCs computed on independent validation sets are not comparable across classifiers in any meaningful way.
• D Cannot tell: AUC is unreliable when the validation set has class imbalance, so without the prevalence we cannot know which classifier discriminates better.