Feature Preprocessing for Machine Learning: One-Hot Encoding and Scaling

Feature preprocessing is the set of transformations that convert raw inputs into a form a Machine Learning algorithm can consume, applied before model training. When a dataset mixes feature types, we must prepare the data before feeding it into a Machine Learning algorithm. This happens when inputs (also called features or covariates) include categories such as gender or geographic region alongside features on different numerical scales, for instance a person’s weight or height.

A Machine Learning algorithm typically requires data in a specific type — often numerical only. ML algorithms also perform better or converge faster when data is preprocessed before training. Because the step happens before model training, we call it preprocessing. This article focuses on two main feature-preprocessing methods: feature scaling (normalisation) and feature standardisation.

Data Exploration with Pandas: info(), describe(), groupby()

To decide what we do with the data and apply Machine Learning to it, we need to analyse the dataset. We want to determine what features we have, whether they are helpful for our ML goals, how clean the dataset is, the presence of missing or noisy data. Quite often, we need also to perform data cleaning or wrangling.

Visualising features, building tables, dropping irrelevant columns, and converting data types all help here. To start playing with data, we first download our dataset — the Medical Cost Personal Datasets, originally drawn from Brett Lantz’s book Machine Learning with R and mirrored on both Kaggle and GitHub. We use the Pandas library to pull it directly from GitHub:

# Importing libraries further used in code
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf

insurance = pd.read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")
insurance.head(10)

The table shows the first ten rows of the insurance charges dataset we just downloaded. Our main goal is to predict medical insurance charges having a person’s age, sex, BMI, number of children, geographic region, and smoking status.

We can use Pandas functions such as info() and describe() to explore our features more in-depth. The info() function prints out a summary of the data with column names data types and finds any missing values.

insurance.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1338 entries, 0 to 1337
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   age       1338 non-null   int64  
 1   sex       1338 non-null   object 
 2   bmi       1338 non-null   float64
 3   children  1338 non-null   int64  
 4   smoker    1338 non-null   object 
 5   region    1338 non-null   object 
 6   charges   1338 non-null   float64
dtypes: float64(2), int64(2), object(3)
memory usage: 73.3+ KB

If we need to change a data type, we use the astype() function. For instance, we want to change the age and number of children from int64 (default type assigned when we downloaded the dataset) into int8, which needs less memory for storage. If you are interested in playing with different data types, read the “Overview of Pandas Data Types” by Chris Moffitt.

insurance['age']= insurance['age'].astype('int8')
insurance['children']= insurance['children'].astype('int8')

Running info() again shows we’ve already saved 18KB by switching to int8.

insurance.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1338 entries, 0 to 1337
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   age       1338 non-null   int8   
 1   sex       1338 non-null   object 
 2   bmi       1338 non-null   float64
 3   children  1338 non-null   int8   
 4   smoker    1338 non-null   object 
 5   region    1338 non-null   object 
 6   charges   1338 non-null   float64
dtypes: float64(2), int8(2), object(3)
memory usage: 55.0+ KB

The describe() function is helpful for numerical features to get their statistical characteristics such as counts, mean, standard deviation.

insurance.describe()

We can do further data analysis with a grouping function to observe that, on average, smokers face far larger medical insurance charges.

insurance.groupby("smoker")["charges"].mean()

smoker
no      8434.268298
yes    32050.231832
Name: charges, dtype: float64

We can also draw plots with Pandas. For instance, we can plot a histogram of the “bmi” feature.

insurance["bmi"].plot(kind="hist")

Since the data exploration is not our main topic, you can read more about it at “Data Exploration 101 with Pandas” by Günter Röhrich. On possible tools to automate the data exploration, read the article by Abdishakur “4 Tools to Speed Up Exploratory Data Analysis (EDA) in Python”.

Data Preprocessing: Normalisation vs. Standardisation

To prepare our data for ML, we transform it into a more machine-readable form. For instance, we can convert string categories into numerical features and rescale numerical features with normalisation or standardisation.

Normalisation rescales data to a common range (0 to 1); scikit-learn implements it as [MinMaxScaler][1]. Standardisation removes the mean and divides each value by the standard deviation; scikit-learn implements it as [StandardScaler][2]. Both methods can improve performance or speed up convergence. The contrast between them:

Linear and logistic regression, nearest neighbours, and neural networks all benefit from feature scaling. Neural networks often do better with normalisation — but test both on your own data and see which one trains faster or scores better.

Transforming Features with MinMaxScaler and OneHotEncoder

Let’s transform features with [MinMaxScaler][1] and [OneHotEncoder][3] (scikit-learn) for our insurance charges dataset. Both methods are combined in a single preprocessing step with make_column_transformer(). We fit the column transformer on the training data only, then apply it to the testing data — this ordering prevents data leakage, where test-set statistics contaminate training. Read more in “How to Avoid Data Leakage When Performing Data Preparation” by Jason Brownlee.

Creating and Evaluating a Keras Neural Network on Preprocessed Data

🔒 Subscribe to keep reading.

Conclusion: A One-Step Column Transformer Before Model Training

🔒 Subscribe to keep reading.

References

🔒 Subscribe to keep reading.