5 Important things to Keep in Mind during Data Preprocessing! (Specific to Predictive Models).

Harsh Kumar Jangir
Harsh Kumar Jangir 12 Jan, 2021 • 5 min read

This article was published as a part of the Data Science Blogathon.

5 Important things to Keep in Mind during Data Preprocessing

Hello, There Data science has been a vastly growing and improving field. New algorithms new models are being created more frequently than ever before. However, the model implementations and the machine learning algorithm comes into play after Data Preprocessing. Below is just a sneak peek of some important things.

Data Preprocessing

The first question arises is what is Data Preprocessing? Keeping it in layman terms Before grinding the wheat, we clean it, after that, we wash it so that all the garbage is removed, Data Preprocessing also does the same with data. It cleans the data and removes the garbage so that in the end only Quality data is received.

That’s why this is a very important step when creating a model.

Below are the 5 things to keep in mind during Data Preprocessing:

 

1. KNOW YOUR DATA

 

know your data

 

It will depend on the model we are building suppose we are building a model on forecasting generally the data will be of concentration of atmospheric gases or suppose we are building a model for sale generally the data we will get will be of how many sales a person made. For every model, the data will be different and it is characteristics are also going to be different. 

It is a good habit to get some information about the data or basically about the parameters present in it. Either by looking up about the parameters on the internet or talking to the person expert in that field for example building a predictive model for forecasting we can talk to a chemistry expert which can tell us some information about the parameters present in it and what are they how do they behave this step makes many things clear about the dataset. And let us understand what is going on in the dataset.

In a Predictive model, we are generally trying to predict a parameter. We do this by finding out the effect of other parameters present in the dataset. The variable which is targeted is called Dependent Variable and the rest variable is called the Independent Variable. For example, suppose we are building a Model that predicts ozone meaning it tells us the concentration of Ozone for the next 24 hours using the previously recorded data. Now Ozone is our Dependent variable and rest parameters are our independent variables.

 

2. IDENTIFY THE VARIABLE/PARAMETERS

IDENTIFY THE VARIABLE/PARAMETERS data preprocessing

There are two types of variable :

  1. Categorical Variables
  2. Continuous Variables

Categorical Variables

Categorical variables are variables that generally define two or more categories for example gender is a categorical variable as it two categories male or female. Categorical variables are treated differently than Continuous variables.

Continuous Variables

Continuous variables are variables that can take any continuous value depending upon the parameter for example Weight, height, age are continuous variables.

Once we have identified these two types of variables then we can proceed further accordingly.

For example categorical variable we can use OneHotEncoder which basically convert the categorical variable into a continuous variable so that it can be fed to the model. For example, gender contains Male and Females so OneHotEncoder will convert Male to 0 and Female to 1.

3. FINDING CORRELATION BETWEEN PARAMETERS

CORRELATION BETWEEN PARAMETERS

Finding Correlation values between parameters (Which is generally between Independent and Dependent variables) is a great way to find out the relations between parameters. The correlation value lies between -1 to 1. -1 and 1 being the best value which means that if a correlation value between parameter near -1 or 1 then those parameters are strongly correlated.

If it is 1 then those both parameter is directly proportional to each other and If it is -1 then those both parameters are inversely proportional.

f those parameters value lies near 0 then they are weakly correlated which means that they both don’t depend upon each other.

From this analysis, we can remove those parameters which do not affect our independent variable. For example in the prediction model of Ozone. Carbon monoxide and Ozone have a correlation value of -0.6 and wind speed and Ozone have a correlation value of -0.1. From this, we can remove the wind speed parameter from our dataset, and that way we can increase our accuracy of the model which otherwise would have been compromised if we have included the wind speed parameter in our dataset.

 

4. TREATING MISSING VALUES  AND EXCEPTION RIGHT

TREATING MISSING VALUES  AND EXCEPTION RIGHT

All the data is recorded so it is very obvious that there can be some errors while recording the data. There might be some missing values some NaN values. Some absurd values for example in the human age parameter there is a value of 200 which is not possible in practical life so this is a mistake in the dataset. And all these mistakes affect our dataset integrity so they need to clarify before inputting the dataset into the model.

For treating missing values we can do a couple of things depending upon the dataset. We can replace the missing values by the mean of that parameter or by the mode of that parameter. Generally, we replace it with the mean of the parameter.

Sometimes we need to remove that row from the dataset where the missing value is present. Either we delete the entire column or delete the entire row which expands to other columns.

5. SCALARIZING THE DATA

Scaling Data

Scalarization depends upon the type of model we are using however scalarization is a very important concept in Data Preprocessing. Suppose we have a dataset that contains parameters A, B, C, D.

Parameter A contains  values  between 0-10

Parameter B contains values between 10-1000

Parameter C contains values between 1000-10,000

Parameter D contains values between 0-1

So we see that the values of the parameters range very much and one parameter might totally over dominate the others so to reduce this and to increase the accuracy and for creating a balanced dataset where all the parameters are Standardized meaning lying in the same range. We scalarize the values

The most commonly used Scalar in python is the Standard scalar which scalarizes all the values between the range -3 to +3.

Hola! That’s the end of it. Hope everyone learned something from it. It was just a small a very brief tip about Data Preprocessing.

This was a blog written for the competition of Blogathon which is happening right now. And actually my first ever blog. Right now a college student in the final of Msc aspiring to become a data scientist. Here is my LinkedIn profile https://www.linkedin.com/in/harsh-kumar-jangir-5545a0174/

 

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion.

blogathondata preprocessing
Harsh Kumar Jangir
Harsh Kumar Jangir 12 Jan 2021

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68
Machine Learning
Become a full stack data scientist

6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling

Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?

Reading Data Files into Python Different Variable Datatypes

Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability

Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable

Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables

Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN

Sample and Population Central Limit Theorem Confidence Interval and Margin of Error

Bivariate Analysis Introduction

Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables

Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables

Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables

Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL

Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model

Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification

Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2

Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building

Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression

Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python

Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation

Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R

Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees

Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools

Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes

Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification

Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?

Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python

Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt

Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R

Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis

Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models

Understand Basics of Recommendation Engine with Case Study

8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model

Introduction to Dask Working with CuML

Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values

Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML

Pickle and Joblib Introduction to Model Deployment

Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask

Model Deployment in Android Model Deployment in Iphone