Naive Bayes, From the Ground Up

I’ve been going through machine learning fundamentals properly — not just using the scikit-learn API, but actually understanding what’s happening underneath. This is my working study guide for Naive Bayes, built up from the probability basics it actually requires. Each section ends with a calculation I worked through myself.

---

Part 1 — Probability is just area

Forget formulas for a moment. Picture a box that contains everything that could possibly happen — every email you might ever receive, every dice roll, every outcome. That box is the sample space.

Draw a circle inside the box for some event — say, “the email contains the word lottery.” The circle covers whatever fraction of the box satisfies that condition.

P(event) = area of the circle ÷ area of the whole box

That’s the entire definition. Probability is a fraction of a region.

When you have two events — “contains lottery” and “is spam” — you draw two overlapping circles. The overlap is written A ∩ B (“A and B”), and it represents everything that satisfies both conditions at once.

---

Part 2 — Conditional probability: shrinking your world

P(A | B) is read “the probability of A, given B.” It doesn’t ask a question about the whole box. It asks:

If I already know I’m somewhere inside circle B, what fraction of B is also inside circle A?

You mentally throw away everything outside B. Your entire universe shrinks down to just B. Then you ask what fraction of that smaller universe is shaded by A.

The one true formula

\[P(A \mid B) = \frac{P(A \cap B)}{P(B)}\]

Overlap ÷ the circle you’re standing in. Everything else in this guide is a variation or consequence of this one line.

Key trap: P(A | B) and P(B | A) are almost never the same number, because the denominator — the circle you’re standing in — is different in each case. Mixing these up is one of the most common reasoning errors in probability (the prosecutor’s fallacy). Always ask: which circle am I standing inside?

---

Part 3 — Worked example: spam and “lottery”

You have 100 emails total.

	contains “lottery”	does not contain “lottery”	row total
spam	24	16	40
ham	6	54	60
column total	30	70	100

P(spam | lottery)

Stand inside the “lottery” circle (30 emails). What fraction are spam?

\[P(\text{spam} \mid \text{lottery}) = \frac{24}{30} = 0.8\]

80% of emails containing “lottery” are spam.

P(lottery | spam)

Stand inside the “spam” circle (40 emails). What fraction contain “lottery”?

\[P(\text{lottery} \mid \text{spam}) = \frac{24}{40} = 0.6\]

Same overlap (24) on top both times — different denominator, different answer (0.8 vs 0.6). This is the proof, in real numbers, that P(A|B) ≠ P(B|A) in general.

---

Part 4 — Where Bayes’ theorem comes from

Sometimes you don’t have P(A ∩ B) handed to you directly — but you do know P(B | A). There’s an algebraic rearrangement:

\[P(A \cap B) = P(B \mid A) \times P(A)\]

Substitute that into the Part 2 formula:

\[P(A \mid B) = \frac{P(B \mid A) \times P(A)}{P(B)}\]

This is not a second formula — it’s the same definition, with one substitution. You only need it when you have to reconstruct the overlap from the other direction’s conditional probability.

In Naive Bayes, this is exactly the situation: it’s easy to count P(word | class) from training data, but what you actually want is P(class | words). Bayes’ theorem is the bridge between the direction you can measure and the direction you need.

---

Part 5 — Naive Bayes: the full picture

A new email arrives containing “lottery” and “free.” Naive Bayes asks one question per candidate class:

If this email were spam, how likely would I be to see exactly these words? And if it were ham, how likely?

Then it picks whichever class makes the evidence most plausible.

Why it’s called “naive”

Naive Bayes assumes every word’s presence is independent of every other word, given the class. It never checks whether “free” and “lottery” tend to show up together — it just multiplies their individual probabilities as if they don’t interact. This assumption is technically wrong (words obviously correlate) but works well in practice anyway.

The full formula

\[\text{score}(c) = \underbrace{P(c)}_{\text{prior}} \times \underbrace{P(w_1 \mid c) \times P(w_2 \mid c) \times \cdots \times P(w_n \mid c)}_{\text{likelihood — one factor per word}}\]

Compute this score for every candidate class. Predict whichever class scores highest.

---

Part 6 — Worked example: the full calculation

Extending the dataset with a second word, “free”:

	contains “lottery”	contains “free”	class total
spam	24	32	40
ham	6	9	60

New email contains both “lottery” and “free.”

Step 1 — Priors:

\[P(\text{spam}) = \frac{40}{100} = 0.4 \qquad P(\text{ham}) = \frac{60}{100} = 0.6\]

Step 2 — Likelihoods:

\[P(\text{lottery} \mid \text{spam}) = \frac{24}{40} = 0.6 \qquad P(\text{free} \mid \text{spam}) = \frac{32}{40} = 0.8\] \[P(\text{lottery} \mid \text{ham}) = \frac{6}{60} = 0.1 \qquad P(\text{free} \mid \text{ham}) = \frac{9}{60} = 0.15\]

Step 3 — Multiply likelihoods:

\(\text{spam likelihood} = 0.6 \times 0.8 = 0.48\) \(\text{ham likelihood} = 0.1 \times 0.15 = 0.015\)

Step 4 — Multiply in the prior:

\(\text{spam score} = 0.4 \times 0.48 = 0.192\) \(\text{ham score} = 0.6 \times 0.015 = 0.009\)

Prediction: spam (0.192 ≫ 0.009)

---

Part 7 — Why priors actually matter

When the word-evidence is close and the classes are imbalanced, priors can flip the answer.

Word evidence slightly favors spam:

\[\text{likelihood(spam)} = 0.10 \qquad \text{likelihood(ham)} = 0.08\]

But spam is rare in this inbox — only 5% of all mail:

\(\text{spam score} = 0.05 \times 0.10 = 0.005\) \(\text{ham score} = 0.95 \times 0.08 = 0.076\)

Ham wins — even though the raw word evidence slightly favored spam. Same logic as why a positive result on a rare-disease test still often means you probably don’t have the disease: if the disease is rare enough, the prior can dominate a single piece of imperfect evidence.

Never evaluate evidence in a vacuum. How common something is to begin with always matters.

---

Part 8 — Log-sum instead of raw multiplication

In a real spam filter, a message might have 50+ words. Multiplying fifty probabilities, each less than 1, shrinks the result toward zero fast enough that computers round it to exactly 0 — called underflow.

Fix: take the logarithm of each probability and add instead of multiply.

\[\log(P(c) \times P(w_1|c) \times \cdots) = \log P(c) + \log P(w_1|c) + \cdots\]

Logs turn multiplication into addition, and addition doesn’t underflow. The class with the highest log-sum is still the class with the highest original product — the ranking never changes, only the scale does. This is why real implementations (including scikit-learn’s MultinomialNB) work in log-space internally.

---

Part 9 — Laplace smoothing

What happens if an email contains a word that never appeared in spam during training? Its count is 0, making P(word | spam) = 0 — and since everything is multiplied, one zero wipes out the entire score, even if every other word screams “spam.”

Laplace smoothing adds a small constant (alpha, usually 1) to every word count so no probability is ever truly zero:

\[P(w \mid c) = \frac{\text{count}(w, c) + \alpha}{\text{total words in } c + \alpha \times \text{vocabulary size}}\]

One unseen word can no longer single-handedly zero out the whole prediction.

---

Part 10 — The whole algorithm, summarized

TRAINING:
  for each class c:
    prior(c) = (# documents in class c) / (total # documents)
    for each word w:
      likelihood(w, c) = (count of w in c + α) / (total words in c + α × vocab size)

PREDICTION for a new document:
  for each class c:
    score(c) = log(prior(c)) + sum of log(likelihood(w, c)) for every word w
  predict the class with the highest score

That’s the complete algorithm. CountVectorizer automates the word-counting; MultinomialNB automates the scoring, in log-space, with smoothing built in.

---

Part 11 — Gaussian Naive Bayes: continuous features

Everything above worked because words can be counted. But what if a feature is a continuous number — height, temperature, transaction amount? You can’t count “how many times height = 172.3 appeared” — almost every exact value is unique. Counting breaks down.

Gaussian Naive Bayes swaps out one piece. The prior, multiplying likelihoods, comparing scores, picking the winner — all identical to Part 10. Only how you compute P(feature | class) changes.

The picture: piles of dots

Plot everyone’s height as a dot on a number line. Stack the dots: wherever multiple people share a similar height, pile the dots on top of each other. Where the pile is tallest, that height is common for that group.

To classify a new height: at this exact point, which group’s pile is taller? Taller pile = more likely that class. That’s Gaussian Naive Bayes.

What controls a pile’s shape

Two numbers fully describe any pile:

• Mean (μ) — the center. The average value for that group.
• Standard deviation (σ) — the spread. Small spread = tall narrow pile. Large spread = short wide pile.

The formula

\[P(x \mid \text{class}) = \frac{1}{\sqrt{2\pi\sigma^2}} \, e^{-\frac{(x-\mu)^2}{2\sigma^2}}\]

What each piece does:

• (x − μ) — how far the new value is from this class’s center
• ÷ σ — scales that distance by how spread out the class normally is. 9cm from center is unremarkable for a loosely spread group, unusual for a tightly clustered one. Dividing by σ converts “far away in cm” into “far away relative to what’s normal for this group” — a z-score
• e^(−…) — turns that scaled distance into a pile height
• 1/√(2πσ²) — scaling constant so total area under the curve equals 1

Worked example

Training data:

class	mean (μ)	std (σ)
female	162.62 cm	5.01 cm
male	179.25 cm	5.78 cm

New person: 172 cm.

• Distance from female: 172 − 162.62 ≈ 9.4 cm → scaled: 9.4 / 5.01 ≈ 1.88 spread-widths away
• Distance from male: 179.25 − 172 ≈ 7.3 cm → scaled: 7.3 / 5.78 ≈ 1.26 spread-widths away

The male pile is taller at 172 → predicts male. Full density calculation confirms: female score ≈ 0.0069, male score ≈ 0.0157.

Same distance, different spread

Two groups, same center (170cm):

• Group A: σ = 2cm → 4cm away = 2.0 spread-widths (unusual)
• Group B: σ = 10cm → 4cm away = 0.4 spread-widths (normal)

Same raw distance, opposite conclusions. Raw distance alone is meaningless without knowing how spread out the group normally is.

Multinomial vs Gaussian — same skeleton, one swapped part

	Multinomial NB	Gaussian NB
Feature type	word/event counts	continuous measurements
P(feature | class)	counting frequency	bell-curve formula using μ, σ
Training learns	word counts per class	mean & std per class
Prior, multiply, pick winner	identical	identical

---

Part 12 — Linear Regression: finding the best line

A line through data is a prediction machine: one number in, one number out.

\[\text{predicted} = \text{slope} \times x + \text{intercept}\]

• Slope — how much the prediction changes per unit of input
• Intercept — baseline prediction when input is zero; lets the line float up or down

MSE: choosing the best line

\[\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2\]

1. Error: gap between actual value and prediction at each point
2. Squared: makes all errors positive and punishes large misses harder
3. Mean: average across all points

The better line has the smaller MSE.

Gradient descent

Instead of trying random lines, gradient descent finds the best line systematically:

1. Start with any line (slope=0, intercept=0). MSE is terrible.
2. Measure the gradient — which direction increases MSE.
3. Step downhill — adjust slope and intercept slightly to reduce MSE.
4. Repeat until MSE stops improving.

Learning rate controls step size. Too large → overshoot. Too small → takes forever. Just right → fast at first, slows as you approach the bottom.

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

scikit-learn uses a direct algebraic formula (the “normal equation”) for small datasets — same answer, found instantly.

---

Part 13 — Evaluation metrics

Why accuracy alone isn’t enough

A disease affecting 1% of people: a classifier that always says “not sick” is 99% accurate — and catches zero sick people. Accuracy hid a broken model.

The confusion matrix — alarm mental model

Before touching formulas, rename the four outcomes into something you can actually remember:

• P (Positive) = the model raises an alarm
• N (Negative) = the model stays silent

Label	What happened
TP — True Positive	Correct alarm — it rang, and it should have
FP — False Positive	Incorrect alarm — it rang, but it shouldn’t have
TN — True Negative	Correct silence — it stayed quiet, rightly so
FN — False Negative	Incorrect silence — it stayed quiet, but should have rung

Every prediction your model makes falls into one of these four cells. All other metrics are arithmetic on these four numbers.

Precision — correctness of the alarm

When the alarm rings, is it right?

\[\text{Precision} = \frac{TP}{TP + FP} = \frac{\text{correct alarms}}{\text{all alarms raised}}\]

The denominator is everything the model called positive. Precision asks: of those, how many were actually correct?

Low precision means the model cries wolf — alarms constantly, mostly wrong, people stop trusting it. High precision means when it rings, you can act on it.

When precision matters most: spam filters (you don’t want real emails deleted), fraud detection (you don’t want innocent accounts frozen).

Recall — completeness of the alarm

When it should ring, does it?

\[\text{Recall} = \frac{TP}{TP + FN} = \frac{\text{correct alarms}}{\text{correct alarms} + \text{incorrect silences}}\]

The denominator is all the real positives in the world — cases that warranted an alarm whether or not the model noticed them. The key: FN (incorrect silence) lives in the Recall formula because a false negative is a missed alarm.

Low recall means the model misses things. There’s a fire and the alarm stays silent. High recall means the model finds nearly everything that matters.

When recall matters most: cancer screening (you must not miss a sick patient), fraud investigation (missing one real case is catastrophic).

The precision-recall tradeoff

They pull in opposite directions. To boost precision, be more conservative — only alarm when very sure, which means more misses. To boost recall, be more aggressive — alarm at the slightest hint, which means more false alarms.

A fire alarm makes this concrete:

Scenario	Precision	Recall
Alarm goes off for every faint smell of smoke	Low	High
Alarm only triggers for a confirmed inferno	High	Low
Alarm triggers at the right times, and only then	High	High

The third row is what you want — and that’s what F1 tries to measure.

F1 score

Does it ring at the right times, and only then?

\[\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}\]

High only when both are high. Uses harmonic mean, not regular average, because a regular average lets you cheat: precision 100%, recall 0% averages to 50% — suggesting a decent model when it’s actually useless (never rings). Harmonic mean of 1.0 and 0.0 is 0.0 — honestly broken.

F1 = 1.0 → perfect precision and perfect recall. F1 = 0.0 → either one is zero.

Reconstructing the formulas from memory

If you can remember which error each metric punishes, you can rebuild the formula without memorising it:

1. Precision punishes FP → FP must be in the denominator → TP / (TP + FP)
2. Recall punishes FN → FN must be in the denominator → TP / (TP + FN)
3. F1 balances both → harmonic mean of the two → 2 × P × R / (P + R)

When to use which

Metric	Question	Penalised by	Use when
Precision	When the alarm rings, is it right?	False alarms (FP)	False alarms are expensive
Recall	When it should ring, does it?	Missed alarms (FN)	Missing a case is expensive
F1	Both at once?	Either extreme	Both matter, classes imbalanced
Accuracy	—	—	Classes balanced, equal error costs

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred))

classification_report prints precision, recall, and F1 for each class separately — the most useful single output for evaluating any classifier.

---

Formula sheet

Concept	Formula
Basic probability	P(A) = area(A) / area(box)
Conditional probability	P(A|B) = P(A∩B) / P(B)
Bayes’ theorem	P(A|B) = P(B|A)·P(A) / P(B)
Naive Bayes score	score(c) = P(c) · ∏ P(wᵢ|c)
Log-space version	log P(c) + Σ log P(wᵢ|c)
Laplace smoothing	(count + α) / (total + α·vocab)
Gaussian density	1/√(2πσ²) · e^(−(x−μ)²/2σ²)
MSE	(1/n) Σ(yᵢ − ŷᵢ)²
Accuracy	(TP + TN) / total
Precision	TP / (TP + FP)
Recall	TP / (TP + FN)
F1	2 × (P × R) / (P + R)

The companion post has a full problem set with worked answers to practice all of this.