Nine regression models, one split: what my comparison can and can't say
16 November 2026, 4 min read
My house price project started with a simple plan: train every regressor I knew on the same data, put the scores in one table and keep the best. The table came out tidy, with a clear winner. Reading it a second time, I realised some of the rows were telling me less than they seemed to.
The setup
The data is scikit-learn's California housing set from the 1990 US census: 20,640 block groups, each with 8 numeric features (median income, house age, average rooms and bedrooms, population, average occupancy, latitude and longitude) and a target, the median house value in units of $100,000. Every column is numeric and none has missing values, so there was no cleaning step.
I split it once, 80/20 with random_state=42, and fitted nine models on the same 16,512 training rows. Each one was scored on the same 4,128 test rows with MAE, RMSE and R².
| Model | R² | RMSE | MAE |
|---|---|---|---|
| Hist gradient boosting | 0.835 | 0.465 | 0.311 |
| Extra trees | 0.806 | 0.504 | 0.327 |
| Random forest | 0.805 | 0.506 | 0.328 |
| Gradient boosting | 0.776 | 0.542 | 0.372 |
| Decision tree | 0.623 | 0.703 | 0.454 |
| Ridge | 0.576 | 0.746 | 0.533 |
| Linear regression | 0.576 | 0.746 | 0.533 |
| ElasticNet | 0.563 | 0.757 | 0.556 |
| Lasso | 0.532 | 0.783 | 0.582 |
In money, the winner's RMSE of 0.465 is about $46,500 and its MAE about $31,100. RMSE sitting well above MAE means a share of the errors are large ones, since squaring punishes big misses more.
The linear models hit a ceiling
Plain linear regression explains about 58% of the variance. Ridge with alpha=1.0 scored 0.575855 against linear regression's 0.575788, a difference too small to mean anything. Lasso (alpha=0.1) and ElasticNet (alpha=0.1, l1_ratio=0.5) did worse.
That made sense once I thought about what regularisation is for. A penalty helps a model that is overfitting. With 8 features and over 16,000 training rows, a straight line isn't overfitting; it's too simple for the data. Shrinking its coefficients can only take away from a fit that was already too weak.
There's a second reason not to read much into the Lasso and ElasticNet rows: I didn't scale the features. The penalty acts on coefficient sizes, and those depend on each feature's units, so population (in the thousands) and average bedrooms (around 1) aren't penalised on equal terms. The comparison is unfair to those two models until the features are standardised.
One tree memorises, many trees generalise
A single decision tree with default settings reached 0.623. With no depth limit it keeps splitting until it fits the training data almost perfectly, which is another way of saying it learns the noise too.
Averaging 100 such trees is what changed the picture. The random forest got 0.805 and Extra Trees 0.806. Each tree still overfits, but they overfit in different ways, and the average cancels much of it out.
Same 100 iterations, different models
The row that surprised me most was boosting. GradientBoostingRegressor and HistGradientBoostingRegressor both ran 100 boosting iterations with the same learning rate, yet one scored 0.776 and the other 0.835.
The difference is in the defaults. GradientBoostingRegressor builds trees with a maximum depth of 3. HistGradientBoostingRegressor grows each tree up to 31 leaves and bins every feature into at most 255 values first. So the "same" 100 iterations give the histogram version much larger trees, and more room to fit the data.
That's the main thing I took from this project: when you compare models on their defaults, you're comparing the defaults someone else picked, not the algorithms. A tuned GradientBoostingRegressor might close most of that gap. My table can't tell me, because I didn't tune either of them.
What the table can't tell me
All nine numbers come from one split. If I had picked a different random_state, every score would move a little, and some rows would swap places. Extra Trees beats the random forest by 0.0016 in R²; I wouldn't call that a win for either. The winner's lead of about 0.03 over the next model is more convincing, but I'd want cross-validation before I stated it as fact.
So I saved HistGradientBoostingRegressor as the current best model, and the repo's README calls these early experimental results rather than a benchmark. The next steps listed there are better feature preparation and model validation, and tuning the chosen model.
The data has limits no model can fix. Prices are from 1990, so the absolute numbers say nothing about today's market. Target values are capped at $500,000, so the model can't predict above that. And each row is a block-group average, not a single home.
Try it
The saved model also runs in your browser on the case study page: a JavaScript copy of it, made with an AI coding assistant for this site, gives the same predictions. You can change the inputs and see the estimate move at sangamgupta.in/work/house-price-prediction, and read the full notebook there or on GitHub.
Get new posts by email
One email when I publish something. No spam, unsubscribe any time.