Data cleaning, feature selection and dimensionality reduction
Repair data quality, select informative features and reason about high-dimensional data without drifting beyond the Guide.
What you need to be able to do
- handle outliers, duplicates, incorrect/irrelevant/misformatted and missing data;
- describe imputation, deletion, predictive modelling, normalization and standardization;
- describe filter, wrapper and embedded feature selection;
- describe overfitting, computational complexity, sparsity, distance metrics, visualization, sample-size and memory issues in high dimensions;
- recognize PCA/LDA as beyond required course depth.
Rapid Recall Deck
Attempt each card first. If a term is new, flip and study it, then answer it again without looking. Mark only secure recall as “Know it”.
Build a cleaning plan
Age Income City Purchased 19 24000 Austin Yes 19 24000 Austin Yes ? 31000 Austin No 220 42000 Boston Yes 31 "51k" BOSTON No
Diagnose before changing
Verify the duplicate before consolidating/removing it. Handle missing age using justified imputation, deletion or predictive modelling. Validate age 220 before treating it as error/outlier. Convert 51k to a consistent numeric format and standardize city representation. Filter irrelevant data only when it does not support the modelling objective.
Cleaning choices change model behaviour
Missing values
Compare imputation, deletion and predictive modelling. Why is none universally best?
Deletion loses records and can bias the dataset; imputation preserves rows but introduces estimates; predictive modelling adds assumptions/complexity. Choice depends on the pattern/amount of missingness and consequences for the task.
Normalization / standardization from a concrete distance calculation
Suppose a later model compares two records using numerical distance. One feature is Age (roughly 18–70) and another is Income (roughly 20,000–100,000). You do not need to know K-NN yet. Explain why using the raw values could let Income dominate the distance, and how normalization or standardization helps.
The Income values are numerically thousands of times larger than Age values, so changes in Income could dominate a raw distance calculation even when Income is not thousands of times more important. Normalization or standardization rescales the numerical features so their magnitudes are more comparable before distance-based modelling.
Feature selection strategies
Filter
Score/select features using data criteria before model-subset search.
Wrapper
Evaluate candidate feature subsets through model performance.
Embedded
Selection occurs during model training.
Why remove weak features?
Removing genuinely uninformative/noisy features can reduce overfitting, computation and memory use and can simplify interpretation/deployment while preserving the information that matters to prediction.
Curse of dimensionality
10 features → 5,000 features, same sample size
More dimensions increase overfitting opportunities and compute/memory costs; examples become sparse, distance measures can lose usefulness, visualization becomes difficult and many more samples may be required to represent the larger feature space adequately.
Technical cleaning meets dataset curation
Scope boundary
Do you need to construct PCA or LDA calculations?
No. PCA and LDA are explicitly beyond the course scope; explain the importance of reducing dimensions rather than learning those statistical procedures.
Visionary Studios
Build a dataset-curation plan covering intellectual property and bias mitigation.
Track provenance/licensing, investigate copyrighted material, remove/correct unintended duplicates and bad metadata, audit representation/bias, apply justified bias mitigation, document missing-data decisions and retain an auditable record of curation choices.
Student-facing mini-set
Paper 1 mini-set
1. Describe one reason for data cleaning before training. 2. Distinguish filter, wrapper and embedded feature selection. 3. Describe two curse-of-dimensionality problems and explain why reducing the number of variables while preserving relevant information can help. 4. Explain why a perfectly formatted dataset can still be ethically biased.
Cleaning prevents poor-quality, duplicated, missing, misformatted or irrelevant data from teaching false patterns. Filter methods select features independently of repeated target-model subset testing; wrapper methods compare subsets through model performance; embedded selection occurs during training. High dimensions can cause sparsity, overfitting, computation/memory growth, weak distance metrics, visualization difficulty or larger sample requirements; reducing variables while preserving relevant information can lessen these problems. Formatting alone does not guarantee representative or fair data.
Finish the learning cycle
Case study / exam preparation
Add one credible source on dataset provenance/IP and one on bias mitigation.
Paper 1 — short practice
Rebuild the full preprocessing pipeline and all three feature-selection methods.
Case-study checkpoint
Do not spend revision time on PCA/LDA mathematics.