The Data Journey: Messy to Meaningful | Phase 4

So far, we’ve: 🧹 Cleaned the data ⚙️ Transformed it and created useful features 📊 Visualized it to uncover patterns and relationships

Now comes the exciting part — turning those patterns into a model.

In Phase 4, we step into BLR (Binary Logistic Regression) using statsmodels to understand how different factors influence the likelihood of an outcome.

From understanding the data → explaining the outcome. 🔍

Let’s build our first statistical model.

import pandas as pd

df = pd.read_csv('Online Retail Phase 3 Output.csv')
df.columns

Index(['CustomerID', 'InvoiceNo', 'StockCode', 'Description', 'Quantity',
       'InvoiceDate', 'UnitPrice', 'Country', 'Revenue', 'Year', 'Month',
       'Day', 'Hour'],
      dtype='object')

I wouldn't directly put these into BLR.

Instead, let's create a customer-level problem.

For example:

Can we predict whether a customer is a high-value customer?

Now BLR has a very clear purpose.

We can create:

• HighValueCustomer = 1 → High-value customer

• HighValueCustomer = 0 → Other customer

Then create customer-level features such as:

• Total Revenue
• Total Quantity
• Number of Orders
• Average Order Value
• Number of Products
• Recency
• Country

This is much more relatable than simply throwing Revenue, Quantity, UnitPrice etc. into a model.

But our data is at transaction level!

"Should one customer appear three times in our modelling dataset?"

NO!!

We need:

Key concept

Transaction-level data → Customer-level dataset

This is where Feature Engineering becomes meaningful.

Let's Start With Feature Engineering

Creating meaningful variables from existing data that help a model learn the problem better.

1. Total Revenue

customer_features = df.groupby('CustomerID').agg(
    TotalRevenue=('Revenue', 'sum')
).reset_index()

1. Number of orders

orders = df.groupby('CustomerID')['InvoiceNo'].nunique()

customer_features['NumberOfOrders'] = (
    customer_features['CustomerID'].map(orders)
)

1. Total Quantity

quantity = df.groupby('CustomerID')['Quantity'].sum()

customer_features['TotalQuantity'] = (
    customer_features['CustomerID'].map(quantity)
)

1. Average Order Value

customer_features['AvgOrderValue'] = (
    customer_features['TotalRevenue'] /
    customer_features['NumberOfOrders']
)

Now we can see the transformation:

RAW TRANSACTIONS

   ↓

Customer behaviour

   ↓

Customer-level features

   ↓

BLR

Now we need our target

X → Features

y → Target

Define high-value customers.

For example, use the 75th percentile of TotalRevenue:

threshold = customer_features['TotalRevenue'].quantile(0.75)

customer_features['HighValueCustomer'] = (
    customer_features['TotalRevenue'] >= threshold
).astype(int)

Instead of arbitrarily saying ₹X makes someone high-value, we're using the distribution of our own dataset. Customers in the top 25% of revenue become our high-value group.

customer_features

	CustomerID	TotalRevenue	NumberOfOrders	TotalQuantity	AvgOrderValue	HighValueCustomer
0	12347	3314.73	7	1893	473.532857	1
1	12348	90.20	3	140	30.066667	0
2	12349	999.15	1	523	999.150000	0
3	12350	294.40	1	196	294.400000	0
4	12352	1130.94	7	500	161.562857	1
...	...	...	...	...	...	...
4186	18280	137.00	1	40	137.000000	0
4187	18281	46.92	1	52	46.920000	0
4188	18282	113.13	2	51	56.565000	0
4189	18283	2002.63	16	1353	125.164375	1
4190	18287	960.76	3	778	320.253333	0

4191 rows × 6 columns

If our target is:

HighValueCustomer

and we define it using:

TotalRevenue

then we cannot use TotalRevenue as a predictor.

Otherwise we're effectively telling the model:

"Predict whether someone is high-value using the exact variable used to define high-value."

That's leakage.

#So our features could be:
    
X = customer_features[
    [
        'NumberOfOrders',
        'TotalQuantity',
        'AvgOrderValue'
    ]
]

y = customer_features['HighValueCustomer']

import statsmodels.formula.api as smf
model = smf.logit('HighValueCustomer ~ NumberOfOrders + TotalQuantity + AvgOrderValue',data = customer_features).fit()

model.summary()

Optimization terminated successfully.
         Current function value: 0.097776
         Iterations 11

Logit Regression Results

Dep. Variable:	HighValueCustomer	No. Observations:	4191
Model:	Logit	Df Residuals:	4187
Method:	MLE	Df Model:	3
Date:	Tue, 15 Sep 2026	Pseudo R-squ.:	0.8261
Time:	11:44:32	Log-Likelihood:	-409.78
converged:	True	LL-Null:	-2357.0
Covariance Type:	nonrobust	LLR p-value:	0.000

	coef	std err	z	P>|z|	[0.025	0.975]
Intercept	-11.3420	0.556	-20.384	0.000	-12.433	-10.251
NumberOfOrders	0.7134	0.059	12.015	0.000	0.597	0.830
TotalQuantity	0.0080	0.000	16.222	0.000	0.007	0.009
AvgOrderValue	0.0072	0.001	10.324	0.000	0.006	0.009

Possibly complete quasi-separation: A fraction 0.21 of observations can be
perfectly predicted. This might indicate that there is complete
quasi-separation. In this case some parameters will not be identified.

BLR Model Interpretation

• LLR p-value = 0.000: Model is statistically significant.
• Pseudo R² = 0.826: Strong model fit compared with the null model.
• All predictors have p-value < 0.001: NumberOfOrders, TotalQuantity, and AvgOrderValue are statistically significant.
• All coefficients are positive: Higher values of these variables are associated with a higher likelihood of being a High-Value Customer.

Model Evaluation

from sklearn.metrics import roc_auc_score, classification_report

# Predicted probabilities
y_prob = model.predict(customer_features)

# Convert probabilities to class predictions
y_pred = (y_prob >= 0.5).astype(int)

# ROC-AUC
roc_auc = roc_auc_score(y, y_prob)
print("ROC-AUC:", roc_auc)

# Classification Report
print(classification_report(y, y_pred))

ROC-AUC: 0.9932750714662173
              precision    recall  f1-score   support

           0       0.97      0.98      0.97      3143
           1       0.93      0.91      0.92      1048

    accuracy                           0.96      4191
   macro avg       0.95      0.94      0.95      4191
weighted avg       0.96      0.96      0.96      4191

BLR Model Evaluation

The model gives an excellent in-sample performance:

• ROC-AUC = 0.993 → Very strong ability to distinguish high-value customers from others.
• Accuracy = 96% → 96% of the existing observations are classified correctly.
• Class 0: Precision = 97%, Recall = 98%
• Class 1: Precision = 93%, Recall = 91% → The model identifies most high-value customers correctly.

⚠️ But there is a catch...

These results look very high because we evaluated the model on the same data used to build it.

So, we don't yet know how the model performs on unseen customers.

The real test is not: "How well did the model learn?" It is: "How well does the model perform on data it has never seen?"

Therefore, in the next phase, we'll introduce a Train–Test Split and evaluate our model properly.

🚀 Next Phase: Machine Learning Algorithms

Now that we've understood how BLR works and how a statistical model can explain the outcome, it's time to move from statistical modelling to Machine Learning.

We'll train models such as Decision Trees, Random Forest, SVM, etc., compare their performance on unseen data, and see which model works best for our problem.

From explaining patterns → to learning patterns → to making predictions. 🚀