Naive Bayes, From the Ground Up

I’ve been going through machine learning fundamentals properly — not just using the scikit-learn API, but actually understanding what’s happening underneath. This is my working study guide for Naive Bayes, built up from the probability basics it actually requires. Each section ends with a calculation I worked through myself.

---

Part 1 — Probability is just area

Forget formulas for a moment. Picture a box that contains everything that could possibly happen — every email you might ever receive, every dice roll, every outcome. That box is the sample space.

Draw a circle inside the box for some event — say, “the email contains the word lottery.” The circle covers whatever fraction of the box satisfies that condition.

P(event) = area of the circle ÷ area of the whole box

That’s the entire definition. Probability is a fraction of a region.

When you have two events — “contains lottery” and “is spam” — you draw two overlapping circles. The overlap is written A ∩ B (“A and B”), and it represents everything that satisfies both conditions at once.

---

Part 2 — Conditional probability: shrinking your world

P(A | B) is read “the probability of A, given B.” It doesn’t ask a question about the whole box. It asks:

If I already know I’m somewhere inside circle B, what fraction of B is also inside circle A?

You mentally throw away everything outside B. Your entire universe shrinks down to just B. Then you ask what fraction of that smaller universe is shaded by A.

The one true formula

\[P(A \mid B) = \frac{P(A \cap B)}{P(B)}\]

Overlap ÷ the circle you’re standing in. Everything else in this guide is a variation or consequence of this one line.

Key trap: P(A | B) and P(B | A) are almost never the same number, because the denominator — the circle you’re standing in — is different in each case. Mixing these up is one of the most common reasoning errors in probability (the prosecutor’s fallacy). Always ask: which circle am I standing inside?

---

Part 3 — Worked example: spam and “lottery”

You have 100 emails total.

	contains “lottery”	does not contain “lottery”	row total
spam	24	16	40
ham	6	54	60
column total	30	70	100

P(spam | lottery)

Stand inside the “lottery” circle (30 emails). What fraction are spam?

\[P(\text{spam} \mid \text{lottery}) = \frac{24}{30} = 0.8\]

80% of emails containing “lottery” are spam.

P(lottery | spam)

Stand inside the “spam” circle (40 emails). What fraction contain “lottery”?

\[P(\text{lottery} \mid \text{spam}) = \frac{24}{40} = 0.6\]

Same overlap (24) on top both times — different denominator, different answer (0.8 vs 0.6). This is the proof, in real numbers, that P(A|B) ≠ P(B|A) in general.

---

Part 4 — Where Bayes’ theorem comes from

Sometimes you don’t have P(A ∩ B) handed to you directly — but you do know P(B | A). There’s an algebraic rearrangement:

\[P(A \cap B) = P(B \mid A) \times P(A)\]

Substitute that into the Part 2 formula:

\[P(A \mid B) = \frac{P(B \mid A) \times P(A)}{P(B)}\]

This is not a second formula — it’s the same definition, with one substitution. You only need it when you have to reconstruct the overlap from the other direction’s conditional probability.

In Naive Bayes, this is exactly the situation: it’s easy to count P(word | class) from training data, but what you actually want is P(class | words). Bayes’ theorem is the bridge between the direction you can measure and the direction you need.

---

Part 5 — Naive Bayes: the full picture

A new email arrives containing “lottery” and “free.” Naive Bayes asks one question per candidate class:

If this email were spam, how likely would I be to see exactly these words? And if it were ham, how likely?

Then it picks whichever class makes the evidence most plausible.

Why it’s called “naive”

Naive Bayes assumes every word’s presence is independent of every other word, given the class. It never checks whether “free” and “lottery” tend to show up together — it just multiplies their individual probabilities as if they don’t interact. This assumption is technically wrong (words obviously correlate) but works well in practice anyway.

The full formula

\[\text{score}(c) = \underbrace{P(c)}_{\text{prior}} \times \underbrace{P(w_1 \mid c) \times P(w_2 \mid c) \times \cdots \times P(w_n \mid c)}_{\text{likelihood — one factor per word}}\]

Compute this score for every candidate class. Predict whichever class scores highest.

---

Part 6 — Worked example: the full calculation

Extending the dataset with a second word, “free”:

	contains “lottery”	contains “free”	class total
spam	24	32	40
ham	6	9	60

New email contains both “lottery” and “free.”

Step 1 — Priors:

\[P(\text{spam}) = \frac{40}{100} = 0.4 \qquad P(\text{ham}) = \frac{60}{100} = 0.6\]

Step 2 — Likelihoods:

\[P(\text{lottery} \mid \text{spam}) = \frac{24}{40} = 0.6 \qquad P(\text{free} \mid \text{spam}) = \frac{32}{40} = 0.8\] \[P(\text{lottery} \mid \text{ham}) = \frac{6}{60} = 0.1 \qquad P(\text{free} \mid \text{ham}) = \frac{9}{60} = 0.15\]

Step 3 — Multiply likelihoods:

\(\text{spam likelihood} = 0.6 \times 0.8 = 0.48\) \(\text{ham likelihood} = 0.1 \times 0.15 = 0.015\)

Step 4 — Multiply in the prior:

\(\text{spam score} = 0.4 \times 0.48 = 0.192\) \(\text{ham score} = 0.6 \times 0.015 = 0.009\)

Prediction: spam (0.192 ≫ 0.009)

---

Part 7 — Why priors actually matter

When the word-evidence is close and the classes are imbalanced, priors can flip the answer.

Word evidence slightly favors spam:

\[\text{likelihood(spam)} = 0.10 \qquad \text{likelihood(ham)} = 0.08\]

But spam is rare in this inbox — only 5% of all mail:

\(\text{spam score} = 0.05 \times 0.10 = 0.005\) \(\text{ham score} = 0.95 \times 0.08 = 0.076\)

Ham wins — even though the raw word evidence slightly favored spam. Same logic as why a positive result on a rare-disease test still often means you probably don’t have the disease: if the disease is rare enough, the prior can dominate a single piece of imperfect evidence.

Never evaluate evidence in a vacuum. How common something is to begin with always matters.

---

Part 8 — Log-sum instead of raw multiplication

In a real spam filter, a message might have 50+ words. Multiplying fifty probabilities, each less than 1, shrinks the result toward zero fast enough that computers round it to exactly 0 — called underflow.

Fix: take the logarithm of each probability and add instead of multiply.

\[\log(P(c) \times P(w_1|c) \times \cdots) = \log P(c) + \log P(w_1|c) + \cdots\]

Logs turn multiplication into addition, and addition doesn’t underflow. The class with the highest log-sum is still the class with the highest original product — the ranking never changes, only the scale does. This is why real implementations (including scikit-learn’s MultinomialNB) work in log-space internally.

---

Part 9 — Laplace smoothing

What happens if an email contains a word that never appeared in spam during training? Its count is 0, making P(word | spam) = 0 — and since everything is multiplied, one zero wipes out the entire score, even if every other word screams “spam.”

Laplace smoothing adds a small constant (alpha, usually 1) to every word count so no probability is ever truly zero:

\[P(w \mid c) = \frac{\text{count}(w, c) + \alpha}{\text{total words in } c + \alpha \times \text{vocabulary size}}\]

One unseen word can no longer single-handedly zero out the whole prediction.

---

Part 10 — The whole algorithm, summarized

TRAINING:
  for each class c:
    prior(c) = (# documents in class c) / (total # documents)
    for each word w:
      likelihood(w, c) = (count of w in c + α) / (total words in c + α × vocab size)

PREDICTION for a new document:
  for each class c:
    score(c) = log(prior(c)) + sum of log(likelihood(w, c)) for every word w
  predict the class with the highest score

That’s the complete algorithm. CountVectorizer automates the word-counting; MultinomialNB automates the scoring, in log-space, with smoothing built in.

---

Part 11 — Gaussian Naive Bayes: continuous features

Everything above worked because words can be counted. But what if a feature is a continuous number — height, temperature, transaction amount? You can’t count “how many times height = 172.3 appeared” — almost every exact value is unique. Counting breaks down.

Gaussian Naive Bayes swaps out one piece. The prior, multiplying likelihoods, comparing scores, picking the winner — all identical to Part 10. Only how you compute P(feature | class) changes.

The picture: piles of dots

Plot everyone’s height as a dot on a number line. Stack the dots: wherever multiple people share a similar height, pile the dots on top of each other. Where the pile is tallest, that height is common for that group.

To classify a new height: at this exact point, which group’s pile is taller? Taller pile = more likely that class. That’s Gaussian Naive Bayes.

What controls a pile’s shape

Two numbers fully describe any pile:

• Mean (μ) — the center. The average value for that group.
• Standard deviation (σ) — the spread. Small spread = tall narrow pile. Large spread = short wide pile.

The formula

\[P(x \mid \text{class}) = \frac{1}{\sqrt{2\pi\sigma^2}} \, e^{-\frac{(x-\mu)^2}{2\sigma^2}}\]

What each piece does:

• (x − μ) — how far the new value is from this class’s center
• ÷ σ — scales that distance by how spread out the class normally is. 9cm from center is unremarkable for a loosely spread group, unusual for a tightly clustered one. Dividing by σ converts “far away in cm” into “far away relative to what’s normal for this group” — a z-score
• e^(−…) — turns that scaled distance into a pile height
• 1/√(2πσ²) — scaling constant so total area under the curve equals 1

Worked example

Training data:

class	mean (μ)	std (σ)
female	162.62 cm	5.01 cm
male	179.25 cm	5.78 cm

New person: 172 cm.

• Distance from female: 172 − 162.62 ≈ 9.4 cm → scaled: 9.4 / 5.01 ≈ 1.88 spread-widths away
• Distance from male: 179.25 − 172 ≈ 7.3 cm → scaled: 7.3 / 5.78 ≈ 1.26 spread-widths away

The male pile is taller at 172 → predicts male. Full density calculation confirms: female score ≈ 0.0069, male score ≈ 0.0157.

Same distance, different spread

Two groups, same center (170cm):

• Group A: σ = 2cm → 4cm away = 2.0 spread-widths (unusual)
• Group B: σ = 10cm → 4cm away = 0.4 spread-widths (normal)

Same raw distance, opposite conclusions. Raw distance alone is meaningless without knowing how spread out the group normally is.

Multinomial vs Gaussian — same skeleton, one swapped part

	Multinomial NB	Gaussian NB
Feature type	word/event counts	continuous measurements
P(feature | class)	counting frequency	bell-curve formula using μ, σ
Training learns	word counts per class	mean & std per class
Prior, multiply, pick winner	identical	identical

---

Part 12 — Linear Regression: finding the best line

A line through data is a prediction machine: one number in, one number out.

\[\text{predicted} = \text{slope} \times x + \text{intercept}\]

• Slope — how much the prediction changes per unit of input
• Intercept — baseline prediction when input is zero; lets the line float up or down

MSE: choosing the best line

\[\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2\]

1. Error: gap between actual value and prediction at each point
2. Squared: makes all errors positive and punishes large misses harder
3. Mean: average across all points

The better line has the smaller MSE.

Gradient descent

Instead of trying random lines, gradient descent finds the best line systematically:

1. Start with any line (slope=0, intercept=0). MSE is terrible.
2. Measure the gradient — which direction increases MSE.
3. Step downhill — adjust slope and intercept slightly to reduce MSE.
4. Repeat until MSE stops improving.

Learning rate controls step size. Too large → overshoot. Too small → takes forever. Just right → fast at first, slows as you approach the bottom.

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

scikit-learn uses a direct algebraic formula (the “normal equation”) for small datasets — same answer, found instantly.

---

Part 13 — Evaluation metrics

Why accuracy alone isn’t enough

A disease affecting 1% of people: a classifier that always says “not sick” is 99% accurate — and catches zero sick people. Accuracy hid a broken model.

The confusion matrix

	Predicted: spam	Predicted: ham
Actually spam	True Positive (TP)	False Negative (FN)
Actually ham	False Positive (FP)	True Negative (TN)

• TP — correctly called spam
• TN — correctly called ham
• FP — called spam, was ham (false alarm / Type I error)
• FN — called ham, was spam (miss / Type II error)

Precision

Of all emails predicted spam, what fraction actually were?

\[\text{Precision} = \frac{TP}{TP + FP}\]

High precision: when you flag something, you’re usually right.

Recall

Of all emails that actually were spam, what fraction did you catch?

\[\text{Recall} = \frac{TP}{TP + FN}\]

High recall: you’re catching most of the real spam.

The precision-recall tradeoff

More aggressive classifier → recall goes up, precision drops. More conservative → precision goes up, recall drops.

Which matters more:

• Spam filter — balance both
• Cancer screening — low recall is catastrophic (missed diagnoses); prefer false alarms over misses
• Legal evidence — low precision is catastrophic (wrongful flagging)

F1 score

\[\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}\]

High only when both are high. Harder to game than accuracy. Uses harmonic mean because the regular average of 0.99 and 0.01 is 0.50 — which sounds okay but masks a broken metric. Harmonic mean of 0.99 and 0.01 is 0.02 — honestly bad.

When to use which

Situation	Prioritise
False alarms are expensive	Precision
Missing a case is expensive	Recall
Both matter roughly equally	F1
Classes balanced, equal error costs	Accuracy
Classes imbalanced	F1 or Precision/Recall — never just Accuracy

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred))

classification_report prints precision, recall, and F1 for each class separately — the most useful single output for evaluating any classifier.

---

Formula sheet

Concept	Formula
Basic probability	P(A) = area(A) / area(box)
Conditional probability	P(A|B) = P(A∩B) / P(B)
Bayes’ theorem	P(A|B) = P(B|A)·P(A) / P(B)
Naive Bayes score	score(c) = P(c) · ∏ P(wᵢ|c)
Log-space version	log P(c) + Σ log P(wᵢ|c)
Laplace smoothing	(count + α) / (total + α·vocab)
Gaussian density	1/√(2πσ²) · e^(−(x−μ)²/2σ²)
MSE	(1/n) Σ(yᵢ − ŷᵢ)²
Accuracy	(TP + TN) / total
Precision	TP / (TP + FP)
Recall	TP / (TP + FN)
F1	2 × (P × R) / (P + R)

The companion post has a full problem set with worked answers to practice all of this.