AI foundations · 07
Do you always need a neural network?
No. For a table of measurements, a straight line, a few yes or no questions, or “look at the most similar examples” often works as well as a neural network. These classical models train in seconds, need less data and are easier to explain. Neural networks shine when the input is images, sound or text, and there is a lot of it.
What you will be able to do
Describe how linear models, decision trees, nearest neighbors and tree ensembles make a prediction, compare them on one small dataset using training and test accuracy, and choose a sensible first model for a problem.
Before you start
Features, labels and classification from How can examples become a rule?, and test sets and accuracy from Good at what, compared with what?
1 · You have already seen this
Different ways to make the same call
Ask a few gardeners how they tell whether a plant will do well, and you may get very different answers.
One goes by a rough rule: more sun is good, more water is bad. Another asks a couple of yes or no questions, such as “Does it get at least a few hours of sun?” A third thinks of plants they grew before and goes with the ones most like this one. Each of them learned from plants they had already seen.
Classical machine learning builds models that learn these same kinds of rules from a table of examples: a straight line, a few yes or no cut-offs, the most similar examples, or many trees that vote. They train in seconds and are easier to explain than a neural network. This page compares them on a small set of made-up plants. It shows how that works, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
From a table to a prediction
Classical machine learning in five steps. The next section goes deeper into step 2 and the worked example into step 4. The lab lets you do steps 4 and 5 for five models at once.
How a classical model is used
Five steps
- 1 · Make a tableEach row is one example and each column a feature, such as hours of sun. One column is the label to predict: did the plant thrive?
- 2 · Pick a familyEach family bets on a shape for the rule: a straight line, a few yes or no cut-offs, “similar examples have similar labels,” or many trees voting.
- 3 · Learn from training rowsThe model fits its rule to the labeled training examples. Nearest neighbors is the exception: it has no training step and simply keeps the data.
- 4 · Predict a new caseA new example goes through the rule, and out comes a label, such as “thrives” or “struggles.”
- 5 · Compare on held-out rowsScore each model on test examples it never learned from.
Going deeper · step 2
Five families, five ideas
Every model below learns from labeled examples. They differ in the shape of the rule they can learn.
A neural network is one way to learn a rule from examples, not the only one. Long before deep learning, people built models that predict from a table: each row is an example, each column a feature (such as hours of sun), and one column is the label to predict (did the plant thrive?). These “classical” methods are still widely used in science, business and government.
Each family makes a different bet about what the pattern looks like. A linear model bets on a straight line (or a flat surface, with more features). A decision tree bets on a few clean yes or no cut-offs. Nearest neighbors bets that similar examples have similar labels. Ensembles combine many trees so their separate mistakes partly cancel out.
| Family | How it predicts | Good for | Watch out for |
|---|---|---|---|
| Linear regression | Multiplies each feature by a learned weight, adds them up, adds an offset: a number comes out. | Predicting numbers when the pattern is roughly a straight line; easy to explain | Curved or “too much is bad” patterns |
| Logistic regression | The same weighted sum, squeezed into a probability between 0 and 1. With two features, its boundary is a straight line. | Yes or no questions; a strong first baseline | Despite its name, it classifies |
| Decision tree | Asks a chain of yes or no questions about the features, learned from the data. | Mixed tables; rules a person can read | Deep trees memorize their training examples |
| k-nearest neighbors | Finds the k most similar stored examples and lets them vote. It has no training step: it keeps the data. | Small datasets; “find me similar cases” | Slow with huge data; features need similar scales |
| Ensembles: random forest, gradient boosting | Combine many trees. A forest trains each tree on a random sample and lets them vote. Boosting adds small trees one at a time, each fixing the errors left so far. | Often the most accurate choice for tables of data | Hundreds of trees are hard to explain |
Going deeper · step 4
One plant, four answers
Here is how four models decide about the same new plant, using rules learned from 12 made-up training plants.
Worked example
One new plant, four ways to decide
- The new plantTest plant C gets 4.5 hours of sun and 4.5 glasses of water a week. (It really thrived.)
- One questionIs sun at least 4 hours? 4.5 is, so: thrives. Right.
- Small treeSun at least 4? Yes. Then: water at least 7.25 glasses? 4.5 is not, so: thrives. Right.
- 3 nearest neighborsDistance on the chart, by Pythagoras: to the training plant at (5, 3) it is √(0.5² + 1.5²) = √2.5 ≈ 1.58; to (6, 5.5) √(1.5² + 1²) = √3.25 ≈ 1.80; to (5, 6.5) √(0.5² + 2²) = √4.25 ≈ 2.06. All three thrived, so the vote is 3 to 0: thrives. Right.
- Straight lineScore = −0.82 + 0.45 × 4.5 − 0.40 × 4.5 = −0.82 + 2.025 − 1.80 ≈ −0.60. Below 0, so: struggles. Wrong.
- Why the line missesIts only idea is “more sun helps, more water hurts,” at fixed rates everywhere on the chart. It cannot say “water is fine up to a point.” The tree and the neighbors can.
Made-up plants. The line’s numbers are rounded from the fit computed in the lab below; the exact score is about −0.59.
3 · Try it
Compare five models on 18 plants
The models learn from 12 training plants. Six test plants stay hidden from them, so they show how each model does on new cases.
Predict first
Before you start: which model will get the most training plants right? Will the same model win on the test plants?
Show the answer
1 nearest neighbor gets all 12 training plants right, every time, on any dataset without duplicates: each training plant is its own nearest neighbor. That says nothing about new plants. On the 6 test plants it gets 4 right, the fewest of the five here. One unusual training plant (sunny, not overwatered, but it struggled anyway) misleads it about test plant B. Memorizing is not the same as learning.
Lab
Model comparison: which boundary fits?
Pick a model. The shaded background shows what it would predict at every spot on the chart, the counts show how many training and test plants it gets right, and rings mark its mistakes.
Simulated in your browser: all five models are fitted to 12 made-up plants as the page loads.
Sun is in hours per day; water is in glasses per week. All 18 plants are made up.
- Training plant 1: sun 1, water 3, struggled
- Training plant 2: sun 2, water 6, struggled
- Training plant 3: sun 3, water 1.5, struggled
- Training plant 4: sun 2.5, water 8.5, struggled
- Training plant 5: sun 5, water 3, thrived
- Training plant 6: sun 6, water 5.5, thrived
- Training plant 7: sun 7, water 2, thrived
- Training plant 8: sun 8.5, water 4, thrived
- Training plant 9: sun 5, water 6.5, thrived
- Training plant 10: sun 6.5, water 9, struggled
- Training plant 11: sun 9, water 8, struggled
- Training plant 12: sun 7, water 4, struggled
- Test plant A: sun 2, water 4, struggled
- Test plant B: sun 6.5, water 3.5, thrived
- Test plant C: sun 4.5, water 4.5, thrived
- Test plant D: sun 8, water 7.5, struggled
- Test plant E: sun 3.5, water 7, struggled
- Test plant F: sun 8.5, water 2, thrived
The rules each model learned from the 12 training plants: the stump asks “Is sun at least 4 hours?” (yes: thrives). The small tree asks the same, then for sunny plants “Is water at least 7.25 glasses?” (yes: struggles). The neighbor models copy the label of the 1 or 3 closest training plants. The line is score = −0.82 + 0.45 × sun − 0.40 × water, and a score of 0 or more means thrives.
| Model | Training plants right | Test plants right | Test plants it gets wrong |
|---|---|---|---|
| One question (decision stump) | 9 of 12 | 5 of 6 | D |
| Small tree (two levels) | 11 of 12 | 6 of 6 | none |
| 1 nearest neighbor | 12 of 12 | 4 of 6 | B, E |
| 3 nearest neighbors | 11 of 12 | 6 of 6 | none |
| Straight line (logistic regression) | 8 of 12 | 5 of 6 | C |
Six test plants is a tiny test. One plant moves the test score by about 17 percentage points, so use the differences here to learn how a comparison works; they do not settle which model wins. Real comparisons use hundreds of test examples, or repeat the split several times (cross-validation). Choosing the 1 or the 3 in nearest neighbors, or the depth of a tree, is done with a validation set, never with the final test set.
Show all 18 made-up plants as a table
| Plant | Sun (hours per day) | Water (glasses per week) | Result | Used for |
|---|---|---|---|---|
| 1 | 1 | 3 | struggled | training |
| 2 | 2 | 6 | struggled | training |
| 3 | 3 | 1.5 | struggled | training |
| 4 | 2.5 | 8.5 | struggled | training |
| 5 | 5 | 3 | thrived | training |
| 6 | 6 | 5.5 | thrived | training |
| 7 | 7 | 2 | thrived | training |
| 8 | 8.5 | 4 | thrived | training |
| 9 | 5 | 6.5 | thrived | training |
| 10 | 6.5 | 9 | struggled | training |
| 11 | 9 | 8 | struggled | training |
| 12 | 7 | 4 | struggled | training |
| A | 2 | 4 | struggled | test |
| B | 6.5 | 3.5 | thrived | test |
| C | 4.5 | 4.5 | thrived | test |
| D | 8 | 7.5 | struggled | test |
| E | 3.5 | 7 | struggled | test |
| F | 8.5 | 2 | thrived | test |
Choosing
When is simple enough?
A good habit: start with the simplest reasonable model as a baseline, and only add complexity when it clearly helps on held-out data.
Classical models are often enough
Tables, small data, explanations
- Tabular data: rows and columns of measurements, like sales records or sensor logs. A 2022 benchmark study found tree ensembles still beat deep learning on many medium-sized tables.
- Small data: hundreds or a few thousand examples. Big neural networks need far more, unless they start from a pretrained model.
- Need for explanation: a short tree or a few weights can be checked by a person.
- Little computing power: they train in seconds on a laptop.
Neural networks shine
Images, sound and language at scale
- Raw, high-dimensional input: a photo is millions of pixel values, and no single pixel means much on its own. Neural networks learn useful features by themselves. See How AI works.
- Audio and text: speech recognition, translation and chatbots are built on deep learning.
- Lots of data and computing power: performance keeps improving with more examples, where simpler models level off.
Which would you try first?
Which would you try first? (1) A bakery has 400 rows of past days: weekday, weather, holiday or not, and loaves sold. The owner wants a forecast she can understand. (2) A wildlife project has 2 million camera-trap photos to sort by species. (3) A library has 20,000 past loans with 10 columns each and wants to flag books likely to come back late.
Show the answer
(1) A linear model or a small tree, compared with the baseline “same as the same weekday last week.” The data is a small table and the owner wants to see why.
(2) A neural network, most likely a pretrained image model adapted to these species. Images, and millions of them, are where deep learning is strongest.
(3) Gradient boosting or a random forest, with logistic regression as the baseline. It is a medium-sized table, where tree ensembles are often the most accurate. If the library needs to explain each flag, the simpler model may still win. Whatever you pick, check it on held-out loans.
Going further
Two more families, and a warning
Optional for older students.
Naive Bayes uses probability: it asks how often each word or feature appeared with each label in the training data, and combines that evidence as if the features were independent. The “naive” assumption is usually false, but the method is fast and has long been used for sorting text, such as spam.
Support vector machines look for the boundary that leaves the widest gap to the nearest examples on each side. With a “kernel,” they can draw curved boundaries too.
The warning: flexible models can memorize. A tree with no depth limit, or 1 nearest neighbor, can get every training example right and still do worse on new ones. That is overfitting. A model that is too stiff, like a straight line for a curved pattern, underfits. Settings such as tree depth or k, called hyperparameters, control this trade-off, and they are chosen with validation data.
4 · Common mix-ups
What people get wrong about classical models
Four common beliefs, each with what to think instead.
Mix-up: Neural networks are always more accurate.
Better: On many tables of data, well-tuned tree ensembles match or beat them, and they are cheaper and easier to check.
Mix-up: If a model gets every training example right, it is perfect.
Better: It may have memorized them. Only held-out test data shows how it does on new cases.
Mix-up: “Classical” means outdated.
Better: These models are widely used today, and they are often the baseline that a neural network has to beat before anyone should switch.
Mix-up: Decision trees are always easy to explain.
Better: A short tree is. A forest of 500 deep trees is not.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
In the lab, 1 nearest neighbor gets 12 of 12 training plants right but fewer test plants than the small tree. What does that show?
Check your answer
Training accuracy flatters models that can memorize. 1 nearest neighbor (12/12 training, 4/6 test) copied an unusual plant exactly, while the small tree (11/12 training, 6/6 test) learned a simpler rule that held up on new plants. Judge models on held-out data.
Why does a straight line struggle with the plant data, even though it is a reasonable model?
Check your answer
The real pattern is “enough sun helps, but too much water hurts.” A straight line can only say “more of this, less of that” at the same rate everywhere. The pattern has a corner, which a tree can draw and a line cannot. That is underfitting.
A school wants to predict which students’ library books will be late, from 3,000 past loans in a spreadsheet. Your friend says, “Use a big neural network, it’s the best AI.” What would you suggest instead, and why?
Check your answer
Start with a simple baseline (for example, “never late”) and logistic regression or a small tree, then try a random forest or gradient boosting. It is a small table, so classical models are likely to do as well, train fast and be easier to explain. Compare everything on held-out loans, and think about whether flagging students is fair at all.
For younger learners
A grades 6–8 path
Same idea, no formulas: students act as the models on paper.
30 minutes · groups of three
Be the model
- Print the 12 training plants on a grid (the table under the lab lists them). Keep the 6 test plants on separate cards, face down.
- Group 1 draws one straight line. Group 2 writes one yes or no question. Group 3 uses “copy your nearest neighbor” with a ruler.
- Turn over the test cards one by one. Each group predicts, then checks.
- Count: how many training plants does each rule get right? How many test plants?
Exit question
Did the best memorizer win?
“The nearest-neighbor group got every training plant right. Did it get every test plant right? Why not?” Good answers mention the odd plant that struggled even with good sun and water.
For a hands-on start, Train the Machine shows how a learned rule can latch onto the wrong pattern.
Honest limits
What this page leaves out
What this page leaves out
How each model is fitted in detail (least squares, gradient descent, Gini impurity), feature scaling, regularization, and models for numbers, such as regression trees. Real projects also spend most of their time on data, not on picking a model. The plant data is invented and far too small to show which model is better in general.
Where to go next
Keep going
Next stop · 08
Can a computer find groups nobody labeled?
Clustering: learning without labels, and why discovered groups need interpreting.
Revisit · 06
Good at what, compared with what?
Baselines, test sets, precision and recall.
The neural branch
How AI works
Neurons, layers and the training loop behind deep learning.
Sources
Where this comes from
- scikit-learn User Guide, “Supervised learning”: linear models, nearest neighbors, decision trees, ensembles, naive Bayes and support vector machines.
- scikit-learn User Guide, “Ensembles”: random forests and gradient boosting.
- Google for Developers, Machine Learning Crash Course: linear regression, logistic regression, overfitting and data splits.
- Google for Developers, “Decision Forests” course: decision trees, random forests and gradient boosted trees.
- Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux (2022), “Why do tree-based models still outperform deep learning on tabular data?”
- Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 4th edition, contents: “Learning from Examples.”
AI words on this page
Machine learning, Feature, Label, Classification, Regression, Linear model, Decision tree, k-nearest neighbors, Random forest, Gradient boosting, Overfitting, Underfitting, Hyperparameter, Validation set, Test set, Neural network, Deep learning
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.