The First Step in Knowing Your Data#

Stage 1 · 📋 Foundations · Lesson 05 of 56 · beginner

◀ Previous · Big Data: Definition, Characteristics, Evolution, and Business Impact · Next · IEEE 754 Floating-Point Standard ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Look before you model#

Before fitting anything, look at the data. This is CRISP-DM’s Data Understanding phase, and skipping it is how projects quietly go wrong — a mis-typed column, a hidden pile of missing values, or an outlier that wrecks a model. The first step is always profiling what you actually have.

What to check#

A good first pass answers a handful of questions:

Shape — how many rows and columns?
Types — which columns are numeric, categorical, or dates?
Ranges and distributions — the min, max, centre and spread of each numeric field;
Missingness — how many values are absent, and where?
Cardinality — how many distinct values do categorical fields take?
Duplicates and oddities — repeated rows, impossible values, surprising codes.

A first pass in pandas#

In practice a few pandas calls cover most of it:

df.shape            # (rows, columns)
df.info()           # column names, dtypes, non-null counts
df.describe()       # count, mean, std, min, quartiles, max (numeric)
df.isnull().sum()   # missing values per column
df.nunique()        # distinct values per column

Understanding, not just numbers#

Numbers alone are not understanding. The real goal is to know what each variable means — its units, how it was collected, what a missing value signifies — so that later choices (which features to use, how to handle gaps) are informed rather than mechanical. Every stage that follows, from association measures to model evaluation, rests on this first honest look.

Hint