Perceptron — Exercises 1 & 2

Author: Felipe Maluli de Carvalho Dias Random seed: 42 Reproducibility: All figures are generated by solution_exercises.py and saved under assets/.

Math Rendering Issue

If mathematical equations are not displaying properly, try refreshing the page (Cmd+R or Alt+F5) or reloading the browser tab.

Installation

Clone or navigate to the project directory:

cd perceptron-exercise

Create and activate a virtual environment:

python -m venv env
source env/bin/activate  # On Windows: env\Scripts\activate

Install dependencies:

pip install -r ../../requirements.txt

Run exercise calculation:

python solution_exercises.py

1) Objective

Implement a single-layer perceptron (NumPy-only, from scratch) and evaluate it on two synthetic 2D datasets:

Exercise 1 (linearly separable): $\mu_0=[1.5, 1.5]$, $\mu_1=[5, 5]$, $\Sigma=\operatorname{diag}(0.5, 0.5)$
Exercise 2 (overlapping): $\mu_0=[3, 3]$, $\mu_1=[4, 4]$, $\Sigma=\operatorname{diag}(1.5, 1.5)$

Each dataset has $N=2000$ samples (1000 per class). Labels use ${-1,+1}$ internally.

2) Model & Update Rule

Decision function. For $x\in\mathbb{R}^2$, $$ s(x) = w^\top x + b,\qquad \hat{y} = \operatorname{sign}\big(s(x)\big)\in{-1,+1}. $$

Perceptron update (on mistakes only). If $y_i \cdot s(x_i) \le 0$, $$ \begin{aligned} w &\leftarrow w + \eta \cdot y_i \cdot x_i \ b &\leftarrow b + \eta \cdot y_i \end{aligned} $$ with learning rate $\eta=0.01$.

Stopping. Stop when an epoch completes with 0 updates (converged) or after 100 epochs.

Why shuffling helps. On overlapping data, fixed sample order can cause update cycling and poor separators. Shuffling each epoch yields a more stable linear boundary.

3) Exercise 1 — Linearly separable Gaussians

Results (this run): epochs = 37, final accuracy = 100.00%, last-epoch updates = 0.

Scatter (data): Open image

Decision boundary after training:

Accuracy vs. epoch:

What to observe / say

The two Gaussian blobs are well separated (means far, low variance).
Perceptron converges quickly (an epoch with 0 updates appears early).
Final training accuracy ≈ 100%, consistent with the convergence theorem.

4) Exercise 2 — Overlapping Gaussians

Results (this run): epochs = 100 (max), final accuracy = 50.35%, last-epoch updates = 4.

Scatter (data):

Decision boundary after training:

Accuracy vs. epoch:

Points to consider

With closer means and larger variance, the classes overlap.
The perceptron is a linear classifier; it can’t perfectly separate overlapping distributions.
Accuracy does not reach 100%; the accuracy curve can plateau and the last epoch may still have some updates (no strict convergence).
Results vary slightly with different random seeds.

5) Implementation notes (from scratch)

Only NumPy is used for math; no ML library is used to implement the perceptron itself.
We use online updates within each epoch (scan samples one by one and update immediately on mistakes).
We plot: data scatter, decision boundary (w\cdot x+b=0), misclassified points (highlighted), and the accuracy curve across epochs.

AI Use

AI collaboration was used in this exercise (code + readme + comments)