The Data Journey: Messy to Meaningful - Phase 5

Introduction

So far, we have:

Collected → Cleaned → Transformed → Visualized → Built Logistic Regression

But there is one important question left:

Will our model work equally well on customers it has never seen before?

In the previous phase, we evaluated Logistic Regression on the same data used to build the model. This helped us understand the model, but it does not tell us how well it generalizes.

So now, we introduce a Train-Test Split and compare multiple ML algorithms like Decision Tree, Random Forest, SVM and Boosting on completely unseen test data.

1. Define X and y

We already created our customer-level dataset and target:

HighValueCustomer = 1 → High-value customer

HighValueCustomer = 0 → Other customers

Our predictors remain:

import pandas as pd

customer_features = pd.read_csv('Online Retail Phase 4 Output.csv',index_col = 0)
customer_features.head()

	CustomerID	TotalRevenue	NumberOfOrders	TotalQuantity	AvgOrderValue	HighValueCustomer
0	12347	3314.73	7	1893	473.532857	1
1	12348	90.20	3	140	30.066667	0
2	12349	999.15	1	523	999.150000	0
3	12350	294.40	1	196	294.400000	0
4	12352	1130.94	7	500	161.562857	1

features = [
    "NumberOfOrders",
    "TotalQuantity",
    "AvgOrderValue"
]

X = customer_features[features]
y = customer_features["HighValueCustomer"]

What are we doing?

We separate:

X → information about the customer

y → what we want to predict

2. Train-Test Split

Now comes an important change from the previous phase.

We divide our data into:

Training Data → Model learns

Testing Data → Model is evaluated

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y
)

Why stratify=y?

Our target has two classes:

0 → Not High Value

1 → High Value

stratify=y ensures that the train and test datasets maintain approximately the same class distribution.

3. Why use a Pipeline?

Now we are going to compare four different algorithms.

Instead of manually preprocessing the data separately for every model, we can create a Pipeline.

This gives us a consistent workflow:

Data -> Scaling -> ML Algorithm -> Prediction -> Test ROC-AUC

This is especially important for SVM, where feature scaling can significantly affect the model.

4. Import the Algorithms

We will compare:

• Decision Tree
• Random Forest
• Support Vector Machine
• Boosting

For boosting, let's use Gradient Boosting.

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

5. Create the Models

models = {

    "Decision Tree": Pipeline([
        ("model", DecisionTreeClassifier(
            random_state=42
        ))
    ]),

    "Random Forest": Pipeline([
        ("model", RandomForestClassifier(
            n_estimators=100,
            random_state=42
        ))
    ]),

    "SVM": Pipeline([
        ("scaler", StandardScaler()),
        ("model", SVC(
            probability=True,
            random_state=42
        ))
    ]),

    "Gradient Boosting": Pipeline([
        ("model", GradientBoostingClassifier(
            random_state=42
        ))
    ])
}

Why different pipelines?

For Decision Tree, Random Forest and Gradient Boosting, scaling isn't necessary.

For SVM, scaling is important because SVM is sensitive to the magnitude of features.

So the SVM pipeline becomes:

X -> StandardScaler -> SVM

6. Train All Four Models

Instead of writing four separate training blocks, we can use a loop.

results = {}

for name, model in models.items():

    model.fit(X_train, y_train)

    y_prob = model.predict_proba(X_test)[:, 1]

    auc = roc_auc_score(y_test, y_prob)

    results[name] = auc

    print(f"{name}: Test ROC-AUC = {auc:.4f}")

Decision Tree: Test ROC-AUC = 0.9865
Random Forest: Test ROC-AUC = 0.9996
SVM: Test ROC-AUC = 0.9991
Gradient Boosting: Test ROC-AUC = 0.9999

This is where the workflow becomes much cleaner.

Each model:

• Learns from X_train
• Predicts probabilities for X_test
• Gets evaluated using y_test
• Produces a Test ROC-AUC

7. Compare the Models

Let's turn the results into a DataFrame.

results_df = pd.DataFrame(
    results.items(),
    columns=["Model", "Test_ROC_AUC"]
)

results_df = results_df.sort_values(
    "Test_ROC_AUC",
    ascending=False
)

results_df

	Model	Test_ROC_AUC
3	Gradient Boosting	0.999856
1	Random Forest	0.999591
2	SVM	0.999076
0	Decision Tree	0.986502

8. Visual Comparison

Since this is also part of our data journey, let's visualize the comparison.

import matplotlib.pyplot as plt

plt.figure(figsize=(8, 5))

plt.bar(
    results_df["Model"],
    results_df["Test_ROC_AUC"]
)

plt.ylabel("Test ROC-AUC")
plt.xlabel("Model")
plt.title("Model Comparison on Unseen Test Data")

plt.ylim(0.5, 1.0)
plt.xticks(rotation=20)

plt.show()

One Important Point About Our Previous 0.993 AUC (Phase 4)

The previous Logistic Regression ROC-AUC of 0.993 was calculated on the same observations used to fit the model.

Therefore, it should not be directly compared with these new test ROC-AUC values.

If we want a fair comparison, we would also train Logistic Regression on X_train and evaluate it on X_test.

We can add it as a baseline:

from sklearn.linear_model import LogisticRegression

models["Logistic Regression"] = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(
        max_iter=1000,
        random_state=42
    ))
])

results = {}

for name, model in models.items():

    model.fit(X_train, y_train)

    y_prob = model.predict_proba(X_test)[:, 1]

    auc = roc_auc_score(y_test, y_prob)

    results[name] = auc

results_df = pd.DataFrame(
    results.items(),
    columns=["Model", "Test_ROC_AUC"]
).sort_values(
    "Test_ROC_AUC",
    ascending=False
)

results_df

	Model	Test_ROC_AUC
3	Gradient Boosting	0.999856
1	Random Forest	0.999591
2	SVM	0.999076
4	Logistic Regression	0.995390
0	Decision Tree	0.986502

Interpretation

• Gradient Boosting — 0.999856: Highest test ROC-AUC among the models evaluated.
• Random Forest — 0.999591: Very close to Gradient Boosting.
• SVM — 0.999076: Also shows extremely strong discrimination.
• Logistic Regression — 0.995390: The simpler baseline still performs extremely strongly.
• Decision Tree — 0.986502: Strong performance, but lower than the other approaches on this test set.

The interesting takeaway is that all five models achieve very high Test ROC-AUC, suggesting that the engineered customer-level features contain strong predictive information for identifying high-value customers.

Because the AUC values are extremely high, this is worth investigating further rather than simply concluding that the problem is solved

Conclusion

From one model to multiple models. From in-sample evaluation to unseen data.

By introducing a Train-Test Split and comparing Logistic Regression, Decision Tree, Random Forest, SVM and Gradient Boosting, we moved from simply building a model to evaluating how different approaches generalize.

The next question is no longer “Which model predicts?”

It is:

“Why did the model make this prediction?” 🔍

🚀 Next Phase: Model Explainability with SHAP & LIME