Why I picked the clustering model with the lower score

In my customer segmentation project, three clustering algorithms competed and the one with the best score lost.

Algorithm	Clusters	Noise points	Silhouette score
K-Means	3	0	0.498
Hierarchical (Ward)	3	0	0.473
DBSCAN	2	47	0.578

If you only read the last column, DBSCAN wins. I shipped K-Means. This post is about why.

From transactions to three numbers per customer

The data is a table of 25,075 transactions: invoice, customer, date, product category, quantity, price, discount, total, country and payment method. After filling 150 missing countries and 100 missing payment methods with the most common value and dropping 75 duplicate rows, 25,000 rows were left, from 2,394 customers.

Clustering transactions directly makes no sense, so I aggregated them into one row per customer with the classic RFM features:

• Recency: days since the customer's last purchase, counted from the day after the latest invoice in the data.
• Frequency: how many transactions they made.
• Monetary: how much they spent in total.

The three features live on very different scales. Recency runs from 1 to a few hundred days, frequency from 1 to 37, and monetary value into the hundreds of thousands. K-Means works on distances, so without scaling the money column would decide everything. I standardised all three with StandardScaler before any clustering.

Choosing K

For K-Means I fitted K from 1 to 10 and recorded the inertia, the total squared distance from each customer to its cluster centre. It always falls as K grows, so the question is where it stops falling quickly. Instead of eyeballing the elbow plot, I let KneeLocator find it, and it picked K = 3.

The silhouette score for that model was 0.498. Silhouette compares how close each point is to its own cluster with how close it is to the next nearest one; it runs from -1 to 1, and higher means better-separated clusters.

What the three clusters mean

Here are the average RFM values per cluster:

Segment	Customers	Recency (days)	Frequency	Monetary
High-Value / Loyal	656	13.8	18.8	139,434
Regular / Mid-Value	1,511	36.0	8.1	38,418
Inactive / Low-Value	227	267.4	2.0	8,376

These are groups a business can act on. Keep the loyal group happy, give the regulars a reason to come back more often, and decide whether the inactive group is worth a win-back campaign. Each segment is big enough to matter and different enough to deserve its own plan.

Checking it against other algorithms

A single algorithm proves little, so I tried two more.

Hierarchical clustering with Ward linkage, cut at 3 clusters, gave a silhouette of 0.473 and almost the same segments: 788 high-value, 1,359 regular and 247 inactive customers, with similar averages. Two different methods finding the same structure made me more confident it is real, not an artefact of K-Means.

DBSCAN was harder. It doesn't take a number of clusters; it takes a neighbourhood radius (eps) and a minimum number of neighbours (min_samples), and it labels points that don't fit anywhere as noise. My first try, eps=0.8, put 2,390 customers in one cluster and 4 in noise. Smaller values went the other way: eps=0.2 gave six clusters, 179 noise points and a silhouette of only 0.078.

So I drew a k-distance graph and searched a grid of eps from 0.25 to 0.36 and min_samples from 4 to 8. The best combination I kept, eps=0.34 with min_samples=4, produced the 0.578 in the table.

Why the best score lost

Look at what that DBSCAN model actually did:

• one cluster with 2,343 customers, about 98% of everyone,
• one cluster with 4 customers,
• 47 customers marked as noise.

There's a second catch. Silhouette can't be computed for noise points, so I measured it only on the customers DBSCAN did assign. The 47 customers it gave up on were left out of its score, and they were not random: their average spend was 184,174, far above everyone else's. In other words, the most valuable customers were the ones DBSCAN couldn't place.

A segmentation that says "almost everyone is the same, plus four special people" gives a marketing team nothing to do. The high silhouette comes from one huge, dense blob and a tiny one far away from it, which is easy to separate and useless to act on.

DBSCAN is built for data with dense regions separated by sparse gaps, and for finding outliers. Customer RFM data mostly forms a continuous spread from occasional to frequent buyers, which is closer to what K-Means assumes.

What I took from it

A metric answers a narrow question. Silhouette asks how well separated the clusters are, not whether they are useful or whether every customer was counted. I chose K-Means because:

1. its three segments are each large enough to act on,
2. hierarchical clustering independently found nearly the same groups,
3. it assigns every customer, including the high spenders DBSCAN dropped,
4. it is simple to serve: the saved scaler and model turn three numbers into a segment instantly.

That last point mattered for the next step. I saved the scaler, the K-Means model and the segment names with joblib and served them from a FastAPI endpoint on AWS EC2, which a Streamlit dashboard calls. The Android client for the same API was built with AI help.

The limitations are real: RFM ignores what people buy, their channel and their demographics, and the segments describe this dataset's customers, so another business would need to refit them.

You can try the model and read the full case study at sangamgupta.in/work/customer-intelligence. The notebook and code are on GitHub.