Naive Bayes — Problem Set with Answers
20 problems covering probability, Naive Bayes, Gaussian NB, MSE, evaluation metrics, and the bridge to logistic regression. Work each one before reading the answer.
Practice problems for the concepts in Naive Bayes, From the Ground Up. Work each problem before reading the answer key at the bottom.
Problems 1–4 cover probability and setup. 5–8 cover the full Naive Bayes algorithm. 9–10 cover Gaussian NB. 11–12 cover MSE. 13–15 cover evaluation metrics. 16–20 bridge into logistic regression (odds, sigmoid, cross-entropy). The last few are marked [Challenge].
Problems
Problem 1 — Basic probability from a table
120 customer reviews, each either positive or negative, each either containing “great” or not.
| contains “great” | no “great” | row total | |
|---|---|---|---|
| positive | 45 | 15 | 60 |
| negative | 15 | 45 | 60 |
| column total | 60 | 60 | 120 |
a) P(positive)
b) P(“great”)
c) P(positive ∩ “great”)
d) P(positive | “great”)
e) P(“great” | positive)
f) Are (d) and (e) the same number? Why or why not?
Problem 2 — Conditional probability, two directions
200 medical test results:
| test positive | test negative | row total | |
|---|---|---|---|
| has condition | 38 | 2 | 40 |
| no condition | 32 | 128 | 160 |
| column total | 70 | 130 | 200 |
a) P(condition | test positive)
b) P(test positive | condition)
c) Which would a doctor want to know after a patient tests positive?
d) Which is easier to measure from training data, and why?
Problem 3 — Bayes’ theorem from scratch
- P(flu) = 0.05
- P(fever | flu) = 0.80
- P(fever) = 0.15
Compute P(flu | fever). Then explain: even though 80% of flu patients have a fever, why is P(flu | fever) much lower than 0.80?
Problem 4 — Multinomial Naive Bayes: classify one document
160 product reviews: 100 positive, 60 negative.
| word | count in positive | count in negative |
|---|---|---|
| “amazing” | 60 | 6 |
| “broken” | 10 | 42 |
| total words | 400 | 240 |
New review contains “amazing” and “broken”. Compute a Naive Bayes score for each class and classify. Show every step.
Problem 5 — When the prior flips the answer
Same likelihoods as Problem 4:
- P(“amazing” | positive) = 0.150, P(“broken” | positive) = 0.025
- P(“amazing” | negative) = 0.025, P(“broken” | negative) = 0.175
New platform where 95% of reviews are positive: P(positive) = 0.95, P(negative) = 0.05.
New review contains only “broken”.
a) Classify using the new priors.
b) Classify using the original priors (100/160 and 60/160).
c) Do the two priors give the same answer? What does this tell you about priors?
Problem 6 — Laplace smoothing
300 spam emails, 5000 total words in spam, vocabulary size 800 words.
“invoice” appears 0 times in spam training. “click” appears 50 times.
a) Without smoothing: what is P(“invoice” | spam)? What happens to the entire spam score?
b) With Laplace smoothing (α = 1): compute P(“invoice” | spam).
c) With Laplace smoothing (α = 1): compute P(“click” | spam).
d) Smoothing slightly lowers the probability of common words like “click”. Why, and is it a problem?
Problem 7 — Log-space arithmetic
| word | P(word | spam) | P(word | ham) |
|---|---|---|
| “win” | 0.40 | 0.02 |
| “free” | 0.35 | 0.04 |
| “now” | 0.30 | 0.12 |
| “link” | 0.20 | 0.10 |
Priors: P(spam) = 0.30, P(ham) = 0.70
a) Compute raw scores (prior × all four likelihoods). Which class wins?
b) Compute log scores (log(prior) + sum of log(likelihoods)). Which class wins?
c) Do both methods give the same winner?
d) If the email had 80 words all with likelihoods ~0.10, what would the raw product look like, and why does log-space become essential?
Reference: ln(0.30) ≈ −1.204, ln(0.70) ≈ −0.357, ln(0.40) ≈ −0.916, ln(0.35) ≈ −1.050, ln(0.20) ≈ −1.609, ln(0.02) ≈ −3.912, ln(0.04) ≈ −3.219, ln(0.12) ≈ −2.120, ln(0.10) ≈ −2.303
Problem 8 — Gaussian NB: classify by height
| species | mean height (μ) | std dev (σ) |
|---|---|---|
| species A | 5.0 cm | 0.30 cm |
| species B | 6.2 cm | 0.80 cm |
| species C | 7.5 cm | 0.40 cm |
Equal priors (1/3 each). New plant measures 6.0 cm.
a) Compute the z-score for each species: z = |x − μ| / σ
b) Based only on z-scores, which species is the new plant closest to?
c) Species A is 1 cm away, species C is 1.5 cm away. Without computing, why might species C still beat species A?
d) Compute the Gaussian density for species A and B at x = 6.0:
Species A: (x−μ)²/2σ² ≈ 5.56 → e^(−5.56) ≈ 0.0038
Species B: (x−μ)²/2σ² ≈ 0.031 → e^(−0.031) ≈ 0.969
Use √(2π) ≈ 2.507.
Problem 9 — Same distance, different spread
Two groups share mean 50.0. New value is 55.0 — 5 units from both centers.
- Group X: σ = 2.0
- Group Y: σ = 10.0
a) Compute the z-score for each group.
b) Which group considers 55.0 a more typical value?
c) If priors are equal, which group would the Gaussian classifier predict?
d) Why does raw distance alone not tell you which class a value belongs to?
Problem 10 — Linear regression: computing MSE
| x (hours studied) | y (exam score) |
|---|---|
| 1 | 3 |
| 2 | 5 |
| 3 | 6 |
| 4 | 10 |
Line A: ŷ = 2.0x + 0
Line B: ŷ = 2.0x + 1
a) For each line, compute predicted value, error, and squared error at every point.
b) Compute MSE for each.
c) Which line fits better?
d) What does a non-zero intercept mean in plain English for this problem?
Problem 11 — Confusion matrix: all four metrics
250 emails:
| Predicted spam | Predicted ham | |
|---|---|---|
| Actually spam | 80 | 20 |
| Actually ham | 10 | 140 |
Compute each, showing the formula and numbers:
a) Accuracy
b) Precision
c) Recall
d) F1
e) The 20 missed spam emails — what’s the technical term, and which metric captures this?
Problem 12 — [Challenge] The 99% accuracy trap
Fraud detection on 10,000 transactions. Only 80 are fraudulent (0.8%). Model always predicts “not fraud.”
a) Compute the accuracy.
b) Compute the recall for the “fraud” class.
c) Would you deploy this model?
d) What single metric immediately reveals this model is useless?
Problem 13 — [Challenge] Choosing the right mode
| mode | precision | recall |
|---|---|---|
| A (conservative) | 0.92 | 0.45 |
| B (aggressive) | 0.48 | 0.91 |
a) Compute F1 for both modes.
b) Which has the higher F1?
c) A hospital wants this for early cancer screening. Which mode, and why — even if its F1 is lower?
d) A legal team with 3 lawyers can only review 20 contracts a week. Which mode fits, and why?
Part 2 — Bridge to Logistic Regression
Concept: Odds and Log-Odds
Probability: out of all possible outcomes, what fraction is the one I care about?
Odds: how many favourable outcomes are there for every one unfavourable?
\[\text{odds} = \frac{P}{1 - P}\]P = 0.75 → odds = 3. P = 0.5 → odds = 1. P = 0.1 → odds = 0.111.
Log-odds (logit):
\[\text{log-odds} = \ln\!\left(\frac{P}{1-P}\right)\]Range is −∞ to +∞. P = 0.5 → log-odds = 0. Above 0.5 → positive. Below 0.5 → negative. This unbounded range is why logistic regression can model any straight-line relationship using its linear part, then map back to probability.
Problem 14 — Odds: converting between probability and odds
a) P(spam) = 0.80. Compute the odds.
b) P(rain) = 0.25. Compute the odds.
c) Odds of winning are 4:1. What is the probability?
d) Odds of positive test (given sick) are 19:1. What is the probability?
e) Model 1 predicts P = 0.50 for two emails. Model 2 predicts P = 0.90 and P = 0.10. Both average P = 0.50. Do they have the same average odds? Compute and check.
Problem 15 — Log-odds: the number line from −∞ to +∞
Fill in the table. Use: log-odds = ln(P / (1−P)).
| P | odds = P/(1−P) | log-odds = ln(odds) |
|---|---|---|
| 0.10 | ||
| 0.25 | ||
| 0.50 | ||
| 0.75 | ||
| 0.90 |
Reference: ln(0.111) ≈ −2.197, ln(0.333) ≈ −1.099, ln(1) = 0, ln(3) ≈ 1.099, ln(9) ≈ 2.197
a) Log-odds when P = 0.5?
b) What pattern do you notice comparing P = 0.10 and P = 0.90?
c) Why is log-odds a natural thing to model with a linear part, rather than probability directly?
Concept: The Sigmoid Function
The sigmoid is the inverse of log-odds — takes a real number and outputs a probability:
\[\sigma(z) = \frac{1}{1 + e^{-z}}\]At z = 0: σ = 0.5. Large positive z → approaches 1. Large negative z → approaches 0.
Reference: e^(−1) ≈ 0.368, e^(−2) ≈ 0.135, e^(−3) ≈ 0.050, e^1 ≈ 2.718, e^2 ≈ 7.389, e^(−0.5) ≈ 0.607
Problem 16 — Sigmoid: computing probabilities from linear scores
Compute σ(z) = 1 / (1 + e^(−z)) for each value:
a) z = 0
b) z = 2
c) z = −2
d) z = 0.5
e) z = −4 (reason from e^4 ≈ 54.6)
f) At what z does the model predict exactly 50/50?
g) z = 0.1 and z = 3.0 both predict “spam” at threshold 0.5. Are they equally confident? What does this mean in practice?
Problem 17 — Full logistic regression prediction
Weights learned:
| feature | weight |
|---|---|
| contains “free” | +2.4 |
| contains “unsubscribe” | −1.8 |
| subject line length (normalized) | −0.6 |
| bias | −0.5 |
Email A: “free” = 1, “unsubscribe” = 0, length = 0.5
Email B: “free” = 0, “unsubscribe” = 1, length = 1.0
For each email:
a) Compute z = (weight · feature) summed + bias
b) Compute P(spam) = σ(z)
c) Classify at threshold 0.5
Reference: e^(−1.1) ≈ 0.333, e^(1.1) ≈ 3.004, e^(−0.8) ≈ 0.449, e^(2.9) ≈ 18.17
Concept: Cross-Entropy Loss
MSE measures gaps between predicted and actual numbers. Logistic regression output is a probability, so MSE is the wrong tool. Instead: binary cross-entropy:
\[L = -\left[y \cdot \ln(p) + (1 - y) \cdot \ln(1 - p)\right]\]Simplified:
- True label 1: loss = −ln(p). Zero when p = 1 (perfect), climbs to +∞ as p → 0.
- True label 0: loss = −ln(1−p). Zero when p = 0 (perfect), climbs to +∞ as p → 1.
Confident correct predictions cost almost nothing. Confident wrong predictions cost a lot.
Reference: ln(0.9) ≈ −0.105, ln(0.7) ≈ −0.357, ln(0.5) ≈ −0.693, ln(0.3) ≈ −1.204, ln(0.1) ≈ −2.303
Problem 18 — Cross-entropy for single predictions
Compute L = −[y·ln(p) + (1−y)·ln(1−p)] for each:
| case | true label (y) | predicted P(positive) | loss |
|---|---|---|---|
| a | 1 | 0.90 | |
| b | 1 | 0.70 | |
| c | 1 | 0.30 | |
| d | 1 | 0.10 | |
| e | 0 | 0.10 | |
| f | 0 | 0.90 |
g) Cases (a) and (f) are both “confident and correct.” Are their losses the same?
h) Cases (d) and (f) are both maximally wrong. Are their losses the same? What does this tell you?
Problem 19 — Average cross-entropy over a batch
| actual y | predicted P(spam) |
|---|---|
| 1 | 0.90 |
| 1 | 0.30 |
| 0 | 0.10 |
| 0 | 0.70 |
a) Compute cross-entropy loss for each.
b) Compute the average loss — what the model minimizes during training.
c) Which prediction contributes most to the loss?
d) Which prediction will drive the largest weight update during gradient descent?
Problem 20 — [Challenge] Why not MSE for logistic regression?
True label y = 1, model predicts p = 0.02 (very confident, very wrong).
a) Compute MSE loss = (y − p)² and CE loss = −ln(p).
b) Try p = 0.98 (very confident, very right). Compute both losses.
c) CE grows much faster than MSE when wrong. Why does this matter for gradient descent?
d) The sigmoid is flat at its extremes (gradient ≈ 0). Chaining MSE with sigmoid:
When p = 0.02, y = 1: (p − y) = −0.98, σ′(z) ≈ 0.0196. What is the combined gradient?
Cross-entropy gradient simplifies to just (p − y). Compute and compare.
Answer Key
Don’t scroll here until you’ve worked through the problems.
Answer 1
a) P(positive) = 60/120 = 0.50
b) P(“great”) = 60/120 = 0.50
c) P(positive ∩ “great”) = 45/120 = 0.375
d) P(positive | “great”) = (45/120) / (60/120) = 45/60 = 0.75
Stand inside the “great” circle (60 reviews). 45 of those are positive.
e) P(“great” | positive) = (45/120) / (60/120) = 45/60 = 0.75
Stand inside the “positive” circle (60 reviews). 45 of those contain “great”.
f) Equal here (both 0.75) because the table is symmetric — both circles are the same size (60 each). In general they are not equal; the denominator is whichever circle you’re standing in. Change any cell and they diverge immediately.
Answer 2
a) P(condition | test positive) = 38/70 ≈ 0.543
b) P(test positive | condition) = 38/40 = 0.95
c) P(condition | test positive) — the doctor wants to know: now that this patient tested positive, what’s the probability they’re actually sick?
d) P(test positive | condition) — measured by running the test on confirmed sick patients and counting how often it returns positive. P(condition | test positive) requires knowing the base rate of the disease in the population, which is harder to measure.
Answer 3
\[P(\text{flu} \mid \text{fever}) = \frac{0.80 \times 0.05}{0.15} = \frac{0.04}{0.15} \approx \mathbf{0.267}\]Only ~27% chance of flu, even with a fever. Because P(flu) = 0.05 is small — flu is rare. Most people with fevers have something else. A fever is weak evidence because it’s common across many conditions. The prior pulls the answer back toward “probably not flu.”
Answer 4
Priors: P(positive) = 100/160 = 0.625, P(negative) = 60/160 = 0.375
Likelihoods:
P(“amazing” | positive) = 60/400 = 0.150, P(“broken” | positive) = 10/400 = 0.025
P(“amazing” | negative) = 6/240 = 0.025, P(“broken” | negative) = 42/240 = 0.175
Scores:
positive = 0.625 × 0.150 × 0.025 = 0.002344
negative = 0.375 × 0.025 × 0.175 = 0.001641
Prediction: positive. “Amazing” outweighs “broken,” helped by the positive prior.
Answer 5
a) New priors (95% positive), word “broken” only:
positive = 0.95 × 0.025 = 0.02375
negative = 0.05 × 0.175 = 0.00875
Prediction: positive — the prior overwhelms the word evidence.
b) Original priors:
positive = 0.625 × 0.025 = 0.015625
negative = 0.375 × 0.175 = 0.065625
Prediction: negative — the word evidence wins.
c) Different answers. “Broken” clearly points negative, but when 95% of reviews are positive, the prior is so dominant it overrides unfavorable evidence. Priors encode how common something is before you see any evidence — when one class is rare enough, you need very strong evidence to overcome it. Not a flaw: if 95 of 100 reviews really are positive, assuming positive before looking at words is a reasonable starting point.
Answer 6
a) P(“invoice” | spam) = 0. The entire spam score becomes 0 — one unseen word wipes out all other evidence.
b) P(“invoice” | spam) = (0 + 1) / (5000 + 800) = 1/5800 ≈ 0.000172
c) P(“click” | spam) = (50 + 1) / (5000 + 800) = 51/5800 ≈ 0.00879
Without smoothing: 50/5000 = 0.01.
d) Smoothing adds α to every count including common words, growing the denominator (by α × vocab size), which slightly dilutes all probabilities. Not a problem — relative ranking is preserved, common words still score much higher than rare ones, and the benefit (no zero-probability words) far outweighs the small distortion.
Answer 7
a) Raw scores:
spam: 0.30 × 0.40 × 0.35 × 0.30 × 0.20 = 0.00252
ham: 0.70 × 0.02 × 0.04 × 0.12 × 0.10 = 0.00000672
Spam wins.
b) Log scores:
spam: −1.204 + (−0.916) + (−1.050) + (−1.204) + (−1.609) = −5.983
ham: −0.357 + (−3.912) + (−3.219) + (−2.120) + (−2.303) = −11.911
−5.983 > −11.911 → Spam wins.
c) Same winner. Log is monotone: preserves ranking, changes scale.
d) 0.10^80 = 10^(−80) — far below what a 64-bit float can represent. The computer rounds to exactly 0, making comparison impossible. In log-space: 80 × ln(0.10) = −184.2 — a perfectly representable number.
Answer 8
a) Z-scores:
Species A: |6.0 − 5.0| / 0.30 = 3.33 spread-widths away
Species B: |6.0 − 6.2| / 0.80 = 0.25 spread-widths away
Species C: |6.0 − 7.5| / 0.40 = 3.75 spread-widths away
b) Species B — 0.25 spread-widths from center, essentially right in the middle of its distribution.
c) Species A’s spread is very tight (σ = 0.30). Being 1 cm away means 3.33 standard deviations out — deep in the tail. Species C’s σ = 0.40; being 1.5 cm away means 3.75 standard deviations. Both are far from their centers in relative terms.
d) Species A: 1/(2.507 × 0.30) × 0.0038 ≈ 1.330 × 0.0038 ≈ 0.00505
Species B: 1/(2.507 × 0.80) × 0.969 ≈ 0.499 × 0.969 ≈ 0.483
Species B wins by a huge margin (0.483 vs 0.005).
Answer 9
a) Group X: 5.0/2.0 = 2.5 spread-widths. Group Y: 5.0/10.0 = 0.5 spread-widths.
b) Group Y — 0.5 spread-widths is well within normal range. Group X is 2.5 spread-widths out.
c) Group Y — pile is taller at 55.0 because it’s a typical value for Group Y.
d) Raw distance only says how far in the same units. Without knowing how spread out the group normally is, you don’t know if that distance is large or small. The spread is the only ruler that makes the gap meaningful.
Answer 10
Line A (ŷ = 2x):
| x | actual | predicted | error | error² |
|---|---|---|---|---|
| 1 | 3 | 2 | 1 | 1 |
| 2 | 5 | 4 | 1 | 1 |
| 3 | 6 | 6 | 0 | 0 |
| 4 | 10 | 8 | 2 | 4 |
MSE = 6/4 = 1.50
Line B (ŷ = 2x + 1):
| x | actual | predicted | error | error² |
|---|---|---|---|---|
| 1 | 3 | 3 | 0 | 0 |
| 2 | 5 | 5 | 0 | 0 |
| 3 | 6 | 7 | −1 | 1 |
| 4 | 10 | 9 | 1 | 1 |
MSE = 2/4 = 0.50
Line B wins.
d) The data doesn’t start at 0 — when x = 1, y = 3, not 2. The intercept allows the line to shift up or down to fit the overall level, not just the slope. The +1 means “even a student who studied 0 hours is predicted to score 1 point” — it absorbs baseline factors not captured by study hours alone.
Answer 11
TP = 80, FN = 20, FP = 10, TN = 140
a) Accuracy = (80 + 140) / 250 = 0.88
b) Precision = 80 / (80 + 10) ≈ 0.889
c) Recall = 80 / (80 + 20) = 0.80
d) F1 = 2 × (0.889 × 0.80) / (0.889 + 0.80) ≈ 0.842
e) The 20 slipped-through emails are False Negatives. The metric capturing this is Recall.
Answer 12
TP = 0, FP = 0, FN = 80, TN = 9,920.
a) Accuracy = 9920/10000 = 99.2%
b) Recall = 0/(0 + 80) = 0%
c) No. The model does nothing — it’s a rubber stamp. 99.2% accuracy is meaningless because 99.2% of transactions are genuinely clean.
d) Recall (or F1) would immediately expose this. Recall = 0% says the model catches none of the actual fraud. Accuracy hides the problem entirely because the imbalance dominates.
Answer 13
a) Mode A: F1 = 2 × (0.92 × 0.45) / (0.92 + 0.45) ≈ 0.604
Mode B: F1 = 2 × (0.48 × 0.91) / (0.48 + 0.91) ≈ 0.629
b) Mode B has the higher F1.
c) Mode B for cancer screening — missing a real cancer means a patient doesn’t get treated in time. That’s potentially fatal. A false alarm means a follow-up scan — inconvenient but not catastrophic. When the cost of a miss vastly exceeds the cost of a false alarm, maximize recall, even at the cost of F1.
d) Mode A for the legal team — they have constrained capacity. Mode A’s precision of 0.92 means 92% of the 20 contracts they review are actually suspicious. Mode B’s precision of 0.48 means roughly half of everything they review is a false alarm, burning half their capacity. When review resource is scarce, precision matters more than catching everything.
Answer 14
a) odds = 0.80/0.20 = 4
b) odds = 0.25/0.75 = 0.333
c) P = 4/(4+1) = 0.80. General formula: P = odds / (1 + odds).
d) P = 19/(19+1) = 0.95
e) Model 1: both at P = 0.50 → odds = 1 each → average odds = 1.
Model 2: P = 0.90 → odds = 9; P = 0.10 → odds = 0.111 → average odds ≈ (9 + 0.111)/2 ≈ 4.56.
They do not have the same average odds. Averaging probabilities and averaging odds give different results because the odds function is non-linear.
Answer 15
| P | odds | log-odds |
|---|---|---|
| 0.10 | 0.111 | −2.197 |
| 0.25 | 0.333 | −1.099 |
| 0.50 | 1.000 | 0 |
| 0.75 | 3.000 | +1.099 |
| 0.90 | 9.000 | +2.197 |
a) 0 — the model is exactly on the fence.
b) P = 0.10 gives −2.197; P = 0.90 gives +2.197. Equal magnitude, opposite sign. Holds for any P and (1−P) pair: ln(p/(1-p)) = −ln((1-p)/p). The log-odds scale is symmetric around 0.5.
c) Probability is bounded 0–1, so a straight line will eventually violate those bounds as inputs grow. Log-odds ranges −∞ to +∞ — exactly the same range as any linear function. Logistic regression fits a straight line to the log-odds, then converts back to probability via sigmoid. No bounds violated.
Answer 16
a) σ(0) = 1/(1+1) = 0.500
b) σ(2) = 1/(1+0.135) ≈ 0.880
c) σ(−2) = 1/(1+7.389) ≈ 0.119
d) σ(0.5) = 1/(1+0.607) ≈ 0.622
e) σ(−4) = 1/(1+54.6) ≈ 0.018
f) z = 0 gives σ = 0.5. Below z = 0: probability < 0.5 (predict negative). Above: probability > 0.5 (predict positive). The threshold of 0.5 corresponds to log-odds = 0 — the model considers both classes equally likely.
g) z = 0.1 → σ ≈ 0.525 (barely above threshold). z = 3.0 → σ ≈ 0.953 (highly confident). Both predict spam but the first is almost a coin flip. In practice this matters when routing borderline predictions for human review.
Answer 17
Email A: z = (2.4)(1) + (−1.8)(0) + (−0.6)(0.5) + (−0.5) = 1.6
σ(1.6): e^(−1.6) ≈ 0.202, σ = 1/1.202 ≈ 0.832 → spam
Email B: z = (2.4)(0) + (−1.8)(1) + (−0.6)(1.0) + (−0.5) = −2.9
σ(−2.9): e^(2.9) ≈ 18.17, σ = 1/19.17 ≈ 0.052 → ham
Answer 18
For y = 1: L = −ln(p). For y = 0: L = −ln(1−p).
| case | y | p | loss |
|---|---|---|---|
| a | 1 | 0.90 | −ln(0.90) ≈ 0.105 |
| b | 1 | 0.70 | −ln(0.70) ≈ 0.357 |
| c | 1 | 0.30 | −ln(0.30) ≈ 1.204 |
| d | 1 | 0.10 | −ln(0.10) ≈ 2.303 |
| e | 0 | 0.10 | −ln(0.90) ≈ 0.105 |
| f | 0 | 0.90 | −ln(0.10) ≈ 2.303 |
g) Yes — cases (a) and (e) both score ≈ 0.105. Being 90% confident and correct costs the same regardless of which class is correct.
h) Yes — both ≈ 2.303. The loss treats overconfident mistakes symmetrically across both classes. It only cares how wrong you are and how confident you were.
Answer 19
| actual y | predicted p | loss |
|---|---|---|
| 1 | 0.90 | 0.105 |
| 1 | 0.30 | 1.204 |
| 0 | 0.10 | 0.105 |
| 0 | 0.70 | 1.204 |
b) Average = (0.105 + 1.204 + 0.105 + 1.204) / 4 ≈ 0.655
c) Predictions 2 and 4 tie at 1.204.
d) Predictions 2 and 4 drive the largest weight updates — they contribute most to total loss. Gradient descent is computing the gradient of the average loss; the examples with the largest individual losses pull hardest.
Answer 20
a) p = 0.02, y = 1:
MSE = (1 − 0.02)² = 0.9604
CE = −ln(0.02) ≈ 3.912
b) p = 0.98, y = 1:
MSE = (1 − 0.98)² = 0.0004
CE = −ln(0.98) ≈ 0.020
c) Gradient descent adjusts weights by a fraction of the gradient of the loss. A larger gradient = larger update = faster correction. When the model is confidently wrong, CE sends a gradient signal ~4× stronger than MSE. The most egregious mistakes drive the biggest corrections.
d) MSE + sigmoid gradient:
(p − y) × σ′(z) = (−0.98) × (0.02 × 0.98) = (−0.98) × 0.0196 ≈ −0.019
CE gradient:
(p − y) = −0.98
The MSE gradient is −0.019 — the sigmoid’s near-zero derivative almost completely cancels the error signal. CE gives −0.98 — fifty times larger. This is the vanishing gradient problem in miniature. CE cancels out the sigmoid’s flat gradient, leaving a clean (p − y) signal that’s large when wrong and small when right.