Topic 8.1: Data Quality

A large hospital network collects patient records across twenty clinics over five years. Each clinic used a slightly different intake form — one clinic recorded age as a number, another as a date of birth, and a third skipped it entirely for some patients. Several patients appear twice because they visited two different branches. Blood pressure was sometimes recorded as a single average number rather than systolic/diastolic. Before any doctor could study health trends in this data, an entire team spent weeks just making the records usable.

This is not an unusual situation — it is the default. Real-world data is collected by humans, entered by hand, exported from systems that were never designed to work together, and accumulated over years without consistent standards. The result is almost never clean. It is raw: unverified, inconsistent, and full of gaps.

The hospital example maps precisely onto every data science project. Before analysis can begin, someone must unify the age formats, remove duplicate patients, decide what to do about missing values, and verify that all recorded values fall within plausible ranges. That process — systematic, careful, and skilled — is what Week 8 is about.

Data quality is not a single property — it is a combination of six measurable dimensions. A dataset can excel on one dimension and fail on another. Understanding each dimension separately allows an analyst to diagnose precisely what is wrong and choose the correct remediation strategy.

These six dimensions are not independent. A value can be complete (present) but inaccurate (wrong). A dataset can be consistent (all the same format) but invalid (all the same wrong format). A full quality assessment checks all six dimensions simultaneously.

The most important distinction to establish at the start of Week 8 is the difference between raw data and analysis-ready data. These are not just different stages — they represent entirely different fitness levels for the task at hand.

A useful test: ask What would happen if I ran an analysis on this data right now? If the answer involves plausible errors — averages pulled up by a typo, categories that should be the same counted as different, or a model that silently ignores half the rows — then the data is not analysis-ready. The job of data preparation is to close every one of those gaps.

It is tempting to view data preparation as a preliminary chore — something you get through before the 'real' analysis begins. This framing is inaccurate and dangerous. A model trained on dirty data does not produce dirty results; it produces confident results that are wrong, which is far more harmful than an obviously broken output.

Professional data preparation is a structured diagnostic and remediation process. It follows a consistent sequence: first inspect the data to understand what problems exist, then address each problem type in a planned order, then validate the result before passing the data to any downstream process.

Each phase requires judgment, not just syntax. Knowing which method to use — and when — is the skill that separates a careful analyst from someone who merely runs code. The decisions made in data preparation directly determine the validity of every conclusion drawn from the data.

The W8 competition dataset is a concrete illustration of quality problems across multiple dimensions. Looking at the raw dataset through the lens of all six quality dimensions reveals exactly what needs to be fixed before any analysis or modelling can begin.

None of these problems cause a Pandas error. The dataset loads, the columns are named correctly, and the data appears superficially reasonable. The problems only become visible when each quality dimension is explicitly audited — which is exactly what Topic 8.4 (Initial Inspection) covers.

The W8 dataset violates three of the six quality dimensions: completeness (missing scores and level values), consistency (seven city spellings for three cities and four date format conventions), and uniqueness (duplicate rows that overcount students). Accuracy, validity, and timeliness happen to be intact for this dataset — the scores are within the valid range and the competition data is current. But no single dataset violates all six dimensions at once — the art of data quality is knowing which dimensions to check and in what order.

Data quality problems are not abstract technical concerns — they translate directly into wrong decisions, wasted resources, and failed projects. Understanding what can go wrong when quality is ignored makes the case for rigorous preparation more concrete than any statistic.

Each row in the table above describes a consequence that looks like an analysis error but is actually a data quality error. The analyst runs the correct code, the code executes without errors, and the output is presented to decision-makers as factual. The damage is done before anyone realises the data was never trustworthy.

Quality awareness is not negativity — it is the professional default. Treating every new dataset as likely to have quality issues (because real-world data almost always does) is what separates an analyst who produces trustworthy results from one who produces fast results. Speed without quality is not an advantage in data science.