📊 Data Preparation & Analysis
Building, scoring and trusting predictive models
0.5.dev0+git.20260808.d6e9440 - August 08, 2026 18:54 UTC

Data Preparation & Analysis

This course covers the applied predictive-modelling workflow end to end — framing a prediction problem, exploring and preparing data, fitting models, and, most importantly for scikit-plots, evaluating them with the right chart for the right question. It is the practical companion to the Terminology reference: terminology defines the metrics, this course shows the workflow that produces and reads them — as an ordered, self-contained sequence of 56 lessons across 8 stages.

Read it at any depth:

• newcomers — the intuition behind analysis, associations and model evaluation;

• practitioners — choosing between regression, trees, ROC, lift and threshold tuning;

• reviewers — diagnostics (residuals, outliers, separation) before shipping a model.

Warning

Report performance on held-out test data, never the data a model was fit on. The sampling-and-partitioning stage exists precisely so the numbers you quote are honest.

Note

Follow the lessons in order with Next ▶, or jump in by stage below. Snippets use real scikitplot / scikit-learn / mlxtend calls, and the evaluation charts (ROC, lift, calibration, residuals) are scikit-plots’ specialty. This course pairs with the Terminology reference (which defines every metric used).

---

📋 Stage 1 — Foundations · 6 lessons

Why we analyse data, the CRISP-DM process, big data, and how numbers are stored. · beginner

• 01 · Why Do We Analyze Data? — The goals of data analysis — turning raw records into decisions, predictions, and understanding.

• 02 · The Process of Data Analysis — The end-to-end workflow from question to insight, and where each technique fits.

• 03 · CRISP-DM for Data Science — The Cross-Industry Standard Process: six phases from business understanding to deployment.

• 04 · Big Data: Definition, Characteristics, Evolution, and Business Impact — The volume, velocity and variety of big data, and why it reshaped analytics.

• 05 · The First Step in Knowing Your Data — Inspecting types, ranges and missingness before any modelling begins.

• 06 · IEEE 754 Floating-Point Standard — How computers store real numbers — precision, rounding, and the pitfalls that follow.

🔗 Stage 2 — Associations & Correlation · 10 lessons

Measuring relationships between variables: correlation, statistical tests and effect size. · beginner

• 07 · Discovering Associations Through Data: From Everyday Patterns to Chicago Taxi Trips (September 2022) — Finding relationships in data, illustrated with the Chicago taxi-trips dataset.

• 08 · Taxi Trips – 2022 dataset from the City of Chicago open data portal — The open dataset used throughout — its fields, scale, and how to load it.

• 09 · Objective Selection of the Bin Width for a Time Histogram — Choosing histogram bin width by a principled rule rather than guesswork.

• 10 · Measuring Associations in Data — How to quantify whether and how strongly two variables move together.

• 11 · Measuring Associations Between Two Continuous Variables — Covariance and correlation for two numeric variables, and what they capture.

• 12 · Correlation Coefficients in Python (Pearson, Spearman, Kendall) — Three correlation measures — linear, rank, and concordance — computed in Python.

• 13 · Karl Pearson — The statistician behind the correlation coefficient and much of modern statistics.

• 14 · Harald Cramér — The mathematician whose V measures association between categorical variables.

• 15 · What Are Statistical Tests? — The logic of hypothesis testing: null hypotheses, p-values, and significance.

• 16 · Eta Squared (η²): Effect Size in ANOVA — An effect-size measure for how much a grouping explains a numeric variable’s variance.

🛒 Stage 3 — Market Basket & Association Rules · 6 lessons

Mining frequent itemsets and association rules for cross-selling with Apriori. · intermediate

• 17 · Understanding Market Baskets and Ideal Customers — What transaction baskets reveal about customer behaviour and product affinity.

• 18 · What Can Association Rules Tell Us? — Reading if-then rules over co-purchased items to guide merchandising.

• 19 · How Association Rules Are Discovered: Concepts, Scale, Measures, and the Apriori Approach — Support, confidence and lift, and how Apriori tames the combinatorial search.

• 20 · Apriori: Frequent Itemsets via the Apriori Algorithm — Finding frequent itemsets efficiently using the downward-closure property.

• 21 · association_rules: Generating Association Rules from Frequent Itemsets (mlxtend) — Turning frequent itemsets into ranked rules with mlxtend in Python.

• 22 · Cross-Selling — Using discovered rules to recommend complementary products.

🧩 Stage 4 — Sampling, Partitioning & Segmentation · 8 lessons

Sampling and train / test partitioning, then grouping observations by clustering and RFM. · intermediate

• 23 · Stratified Random Sampling — Sampling that preserves subgroup proportions for representative data.

• 24 · Linear Congruential Random Number Generator (LCG) — A classic pseudo-random generator — how reproducible randomness is produced.

• 25 · Partitioning Observations to Train Objective Models — Splitting data into train and test sets to measure honest performance.

• 26 · Putting Similar Observations into Clusters — The idea of grouping records by similarity — unsupervised segmentation.

• 27 · Clustering — Algorithms like k-means that partition observations into homogeneous groups.

• 28 · Recency, Frequency, and Monetary Value (RFM) — Three behavioural dimensions that summarise a customer’s value.

• 29 · RFM Analysis — Scoring and ranking customers on recency, frequency, and monetary value.

• 30 · Creating Segments of Observations for Business Reasons (RFM) — Turning RFM scores into actionable customer segments.

📈 Stage 5 — Regression · 7 lessons

Linear and multiple regression, feature selection, and feature importance. · intermediate

• 31 · Least Squares Regression — Fitting a line by minimising the sum of squared residuals.

• 32 · Multiple Linear Regression — Predicting an outcome from several features with a linear model.

• 33 · Feature Importance in Linear Regression — Judging which predictors matter most from coefficients and standardisation.

• 34 · Forward Selection: Definition and Core Idea — Building a model by adding the most useful feature one at a time.

• 35 · Forward Selection and Model Interpretation in Linear Regression — Applying forward selection to a regression and reading the result.

• 36 · Understanding Forward and Backward Stepwise Regression — Adding and removing features by a criterion to search model space.

• 37 · How Shapley Values Work — Fairly attributing a prediction to each feature using cooperative game theory.

🎯 Stage 6 — Classification & Logistic Regression · 6 lessons

Modelling binary outcomes with logistic regression, maximum likelihood, and model fit. · advanced

• 38 · Logistic Regression: Modeling Binary Outcomes via Odds and Log-Odds — Modelling a yes/no probability through the log-odds linear form.

• 39 · Maximum Likelihood (MLE): Fitting a Distribution to Observed Data — Choosing parameters that make the observed data most probable.

• 40 · Assessing Model Fit in Logistic Regression — Predictive-power and pseudo-R² measures for a fitted logistic model.

• 41 · Complete and Quasi-Complete Separation in Logistic Regression — When perfect class separation breaks maximum-likelihood estimation.

• 42 · Forward Selection with Nested Models and Deviance Tests — Comparing nested logistic models via the deviance likelihood-ratio test.

• 43 · Interpreting and Assessing a Forward-Selection Logistic Regression Model for College Student Retention — A worked case study fitting and reading a student-retention model.

🌳 Stage 7 — Decision Trees · 6 lessons

CART decision trees as piecewise models, the interactions they capture, and profiling clusters. · advanced

• 44 · Motivation of Decision Trees: An Incremental Model of Decision-Making — Why splitting data by questions yields an interpretable predictor.

• 45 · The CART Algorithm — Classification and Regression Trees — recursive binary splitting by impurity.

• 46 · Decision Trees as Piecewise Models and Their Predictive Structure — Reading a tree as a piecewise-constant model over feature regions.

• 47 · How CART Decision Trees Model Interactions — Why trees naturally capture feature interactions through nested splits.

• 48 · Cluster Profiling Using Decision Trees — Describing clusters by training a tree to predict cluster membership.

• 49 · Using Decision Trees to Explain Clustering Results — Turning opaque clusters into readable rules with a surrogate tree.

📊 Stage 8 — Model Evaluation · 7 lessons

Judging models: prediction quality, classification metrics, ROC / AUC, lift, and residuals. · advanced

• 50 · Assessing the Quality of Prediction Models — The general question of how good a model is, and by which yardstick.

• 51 · Binary Classification Models – Conceptual Framework and Evaluation Metrics — The confusion matrix and the metrics derived from it for two classes.

• 52 · Nominal Classification Models: Model State and Evaluation Metrics — Extending evaluation to multi-class, unordered outcomes.

• 53 · Binary Classification Model Evaluation and Threshold Optimization — Choosing a decision threshold to balance precision, recall, and cost.

• 54 · Identifying Outliers Using Residuals and Studentized Residuals — Spotting influential and anomalous points from scaled residuals.

• 55 · AUC–ROC Curve: Evaluating Classification Model Performance — The ROC curve and the area under it as threshold-free performance measures.

• 56 · Lift Analysis for Direct Mail Campaigns: Concept, Process, and Business Value — Ranking prospects by model score to target a campaign efficiently.

🔤 Every lesson, A–Z index

🔠 Open the full alphabetical index

Apriori: Frequent Itemsets via the Apriori Algorithm Assessing Model Fit in Logistic Regression Assessing the Quality of Prediction Models association_rules: Generating Association Rules from Frequent Itemsets (mlxtend) AUC–ROC Curve: Evaluating Classification Model Performance Big Data: Definition, Characteristics, Evolution, and Business Impact Binary Classification Model Evaluation and Threshold Optimization Binary Classification Models – Conceptual Framework and Evaluation Metrics Cluster Profiling Using Decision Trees Clustering Complete and Quasi-Complete Separation in Logistic Regression Correlation Coefficients in Python (Pearson, Spearman, Kendall) Creating Segments of Observations for Business Reasons (RFM) CRISP-DM for Data Science Cross-Selling Decision Trees as Piecewise Models and Their Predictive Structure Discovering Associations Through Data: From Everyday Patterns to Chicago Taxi Trips (September 2022) Eta Squared (η²): Effect Size in ANOVA Feature Importance in Linear Regression Forward Selection and Model Interpretation in Linear Regression Forward Selection with Nested Models and Deviance Tests Forward Selection: Definition and Core Idea Harald Cramér How Association Rules Are Discovered: Concepts, Scale, Measures, and the Apriori Approach How CART Decision Trees Model Interactions How Shapley Values Work Identifying Outliers Using Residuals and Studentized Residuals IEEE 754 Floating-Point Standard

Interpreting and Assessing a Forward-Selection Logistic Regression Model for College Student Retention Karl Pearson Least Squares Regression Lift Analysis for Direct Mail Campaigns: Concept, Process, and Business Value Linear Congruential Random Number Generator (LCG) Logistic Regression: Modeling Binary Outcomes via Odds and Log-Odds Maximum Likelihood (MLE): Fitting a Distribution to Observed Data Measuring Associations Between Two Continuous Variables Measuring Associations in Data Motivation of Decision Trees: An Incremental Model of Decision-Making Multiple Linear Regression Nominal Classification Models: Model State and Evaluation Metrics Objective Selection of the Bin Width for a Time Histogram Partitioning Observations to Train Objective Models Putting Similar Observations into Clusters Recency, Frequency, and Monetary Value (RFM) RFM Analysis Stratified Random Sampling Taxi Trips – 2022 dataset from the City of Chicago open data portal The CART Algorithm The First Step in Knowing Your Data The Process of Data Analysis Understanding Forward and Backward Stepwise Regression Understanding Market Baskets and Ideal Customers Using Decision Trees to Explain Clustering Results What Are Statistical Tests? What Can Association Rules Tell Us? Why Do We Analyze Data?

Tags: purpose: reference topic: data analysis topic: data preparation