{"id":23659,"date":"2018-08-05T04:00:00","date_gmt":"2018-08-05T04:00:00","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=23659"},"modified":"2026-04-28T11:58:25","modified_gmt":"2026-04-28T06:28:25","slug":"python-ml-data-preprocessing","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/python-ml-data-preprocessing\/","title":{"rendered":"Data Preprocessing, Analysis & Visualization – Python Machine Learning"},"content":{"rendered":"

Today in this Python Machine Learning Tutorial<\/strong><\/a>, we will discuss Data Preprocessing, Analysis & Visualization. Moreover, in this Data Preprocessing in Python machine learning<\/strong><\/a>, we will look at rescaling, standardizing, normalizing, and binarizing the data. Also, we will see different steps in Data Analysis, Visualization, and Python Data Preprocessing Techniques.<\/p>\n

Data preprocessing remains one of the significant stages in the machine learning process because it aids in enhancing the models\u2019 accuracy and effectiveness. Data preparation is a process that is typically used to clean data and make arrangements that are favorable for the analysis process.<\/p>\n

On the other hand, Data preprocessing has an essential role, a prerequisite step in simplifying the data so that it can be ingested by the machine learning algorithm. Cleaning the data entails checking for pertinent features that keep errors and biases away and making sure the data is correctly formatted for the model.<\/p>\n

So, let’s start machine learning with Python Data Preprocessing.<\/p>\n

\"Data<\/a>

Data Preprocessing, Analysis & Visualization – Python Machine Learning<\/p><\/div>\n

Data Preprocessing in Python Machine Learning<\/strong><\/h3>\n

Machine Learning algorithms<\/strong><\/a> don\u2019t work so well with processing raw data. Before we can feed such data to an ML algorithm, we must preprocess it. In other words, we must apply some transformations to it. With data preprocessing, we convert raw data into a clean data set.<\/span><\/p>\n

\"Data<\/a><\/p>\n

Some ML models<\/strong> need information to be in a specified format. For instance, the Random Forest algorithm does not take null values. To preprocess data, we will use the library scikit-learn or sklearn in this tutorial.<\/span><\/p>\n

Python Data Preprocessing Techniques<\/h3>\n

Let\u2019s talk about seven such techniques for Data Preprocessing in Python Machine Learning.<\/span>
\nLet’s have a look at data Structure in Python<\/a><\/strong><\/p>\n

1. Rescaling Data using Python<\/strong><\/h4>\n

For data with attributes of varying scales, we can rescale attributes to possess the same scale. We rescale attributes into the range 0 to 1 and call it normalization. We use the MinMaxScaler class from scikit-learn. Let\u2019s take an example.<\/span><\/p>\n

>>> import pandas, scipy, numpy\r\n>>> from sklearn.preprocessing import MinMaxScaler\r\n>>> df=pandas.read_csv( 'http:\/\/archive.ics.uci.edu\/ml\/machine-learning-databases\/wine-quality\/winequality-red.csv ',sep=';')\r\n>>> array=df.values\r\n>>> #Separating data into input and output components\r\n>>> x=array[:,0:8]\r\n>>> y=array[:,8]\r\n>>> scaler=MinMaxScaler(feature_range=(0,1))\r\n>>> rescaledX=scaler.fit_transform(x)\r\n>>> numpy.set_printoptions(precision=3) #Setting precision for the output\r\n>>> rescaledX[0:5,:]<\/pre>\n
\"Data<\/a><\/p>\n
This gives us values between 0 and 1. Rescaling data proves useful with neural networks, optimization algorithms and those that use distance measures like k-nearest neighbors and weight inputs like regression.<\/p>\n
2. Standardizing Data in Python<\/strong><\/h4>\n
With standardization, we can take attributes with a Gaussian distribution and different means and standard deviations and transform them into a standard Gaussian distribution with a mean of 0 and a standard deviation of 1. For this, we use the StandardScaler class. Let\u2019s take an example.<\/span>
\nDo you know about Python Statistics<\/a>?<\/strong><\/p>\n
>>> from sklearn.preprocessing import StandardScaler\r\n>>> scaler=StandardScaler().fit(x)\r\n>>> rescaledX=scaler.transform(x)\r\n>>> rescaledX[0:5,:]<\/pre>\n
\"Data<\/a><\/p>\n
3. Normalizing Data using Python<\/h4>\n
In this task, we rescale each observation to a length of 1 (a unit norm). For this, we use the Normalizer class. Let\u2019s take an example.<\/span><\/p>\n
>>> from sklearn.preprocessing import Normalizer\r\n>>> scaler=Normalizer().fit(x)\r\n>>> normalizedX=scaler.transform(x)\r\n>>> normalizedX[0:5,:]<\/pre>\n
\"Data<\/a>
Normalizing Data in Data Preprocessing<\/p><\/div>\n
4. Binarizing Data using Python<\/strong><\/h4>\n
Using a binary threshold, it is possible to transform our data by marking the values above it 1 and those equal to or below it, 0. For this purpose, we use the Binarizer class. Let\u2019s take an example.<\/span>
\nLearn about Python Data Science Environment Setup<\/a><\/strong><\/p>\n
>>> from sklearn.preprocessing import Binarizer\r\n>>> binarizer=Binarizer(threshold=0.0).fit(x)\r\n>>> binaryX=binarizer.transform(x)\r\n>>> binaryX[0:5,:]<\/pre>\n
This marks 0 over all values equal to or less than 0, and marks 1 over the rest. When you want to turn probabilities into crisp values, this functionality comes in handy.<\/span><\/p>\n
\"Data<\/a>
Binarizing Data in Data Preprocessing<\/p><\/div>\n
5. Mean Removal using Python<\/strong><\/h4>\n
We can remove the mean from each feature to center it on zero.<\/span><\/p>\n
>>> from sklearn.preprocessing import scale\r\n>>> data_standardized=scale(df)\r\n>>> data_standardized.mean(axis=0)<\/pre>\n
array([ 3.555e-16, \u00a01.733e-16, -8.887e-17, -1.244e-16, \u00a03.910e-16,<\/strong>
\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0-6.221e-17, \u00a04.444e-17, 2.364e-14, \u00a02.862e-15, 6.754e-16,<\/strong>
\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a01.066e-16, \u00a08.887e-17])<\/strong><\/p>\n
>>> data_standardized.std(axis=0)<\/pre>\n
array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])<\/strong><\/p>\n
6. One Hot Encoding using Python<\/strong><\/h4>\n
When dealing with a few and scattered numerical values, we may not need to store them. Then, we can perform one-hot encoding. For k distinct values, we can transform the feature into a k-dimensional vector with one value of 1 and 0 for the rest of the values.<\/span><\/p>\n
>>> from sklearn.preprocessing import OneHotEncoder\r\n>>> encoder=OneHotEncoder()\r\n>>> encoder.fit([[0,1,6,2],\r\n[1,5,3,5],\r\n[2,4,2,7],\r\n[1,0,4,2]\r\n])<\/pre>\n
OneHotEncoder(categorical_features=’all’, dtype=<class ‘numpy.float64’>,<\/strong>
\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0handle_unknown=’error’, n_values=’auto’, sparse=True)<\/strong><\/p>\n
>>> encoder.transform([[2,4,3,4]]).toarray()<\/pre>\n
array([[0., 0., 1., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0.]])<\/strong><\/p>\n
7. Label Encoding using Python<\/strong><\/h4>\n
Some labels can be words or numbers. Usually, training data is labelled with words to make it readable. Label encoding converts word labels into numbers to let algorithms work on them. Let\u2019s take an example.<\/span>
\nLet’s discuss Python Packages<\/a><\/strong><\/p>\n
>>> from sklearn.preprocessing import LabelEncoder\r\n>>> label_encoder=LabelEncoder()\r\n>>> input_classes=['Havells','Philips','Syska','Eveready','Lloyd']\r\n>>> label_encoder.fit(input_classes)<\/pre>\n
LabelEncoder()<\/strong><\/span><\/p>\n
>>> for i,item in enumerate(label_encoder.classes_):\r\nprint(item,'-->',i)<\/pre>\n
Eveready –> 0<\/strong>
\nHavells –> 1<\/strong>
\nLloyd –> 2<\/strong>
\nPhilips –> 3<\/strong>
\nSyska –> 4<\/strong>
\nThis gives us a set of numeric labels that map to these words. Let\u2019s confirm this:<\/span><\/p>\n
>>> labels=['Lloyd','Syska','Philips']\r\n>>> label_encoder.transform(labels)<\/pre>\n
array([2, 4, 3], dtype=int32)<\/strong><\/p>\n
>>> label_encoder.inverse_transform(label_encoder.transform(labels))<\/pre>\n
array([‘Lloyd’, ‘Syska’, ‘Philips’], dtype='<U8′)<\/strong><\/p>\n
4. Analyzing Data in Python Machine Learning<\/strong><\/h3>\n
Assuming that you have loaded your dataset using pandas (which, if you haven\u2019t, refer to the <\/span>Python Pandas Tutorial<\/a><\/strong> to learn how), let\u2019s find out more about our data.<\/span><\/p>\n
a. Describing the dataset using ML<\/strong><\/h4>\n
Using the method describe(), we can find out parameters like count, mean, std, and max.<\/span><\/p>\n
>>> df.describe()<\/pre>\n
\"Data<\/a>
Analyzing Data in Python Machine Learning<\/p><\/div>\n
b. Shape of the dataset<\/strong><\/h4>\n
Simply, the shape tuple will give us the dimensions of the dataset.<\/span><\/p>\n
>>> df.shape<\/pre>\n
(1599, 12)<\/strong><\/p>\n
c. Extracting data from the dataset<\/span><\/h4>\n
Now, if we want only the first ten rows from the dataset, we can call the head() method on it. To this, we can pass it the argument 10.<\/span><\/p>\n
>>> df.head(10)<\/pre>\n
\"Data<\/a>
Extracting data from the dataset – Data Analyzing<\/p><\/div>\n
d. Performing operations around a variable<\/h4>\n
We can perform certain operations on a variable. For instance, here, we demonstrate how to group data on a variable. For this, we use the groupby() function.<\/span>
\nHave a look at Python NumPy<\/a><\/strong><\/p>\n
>>> df.groupby('quality').size()<\/pre>\n
\"Data<\/a>
Performing operations around a variable in data Analyzing<\/p><\/div>\n
Visualizing Data-Univariate Plots in Python Machine Learning<\/strong><\/h3>\n
Finally, when we want to visualize data as plots and charts to learn more about it, we can use pandas with Matplotlib. We will discuss two kinds of plots- univariate and multivariate.<\/span>
\nA univariate plot suggests we\u2019re only examining one variable.<\/span><\/p>\n
a. Histograms<\/strong><\/h4>\n
Since histograms<\/a><\/strong> group data into bins and give us an idea of how many observations each bin holds, this is a good way to visualize data for ML. The shapes of the bins tell us whether an attribute is Gaussian, skewed, or has an exponential distribution. It also hints at outliers.<\/span><\/p>\n
>>> import matplotlib.pyplot as plt\r\n>>> df.hist()\r\n>>> plt.show()<\/pre>\n
\"Data<\/a>
Histograms in Python Machine Learning<\/p><\/div>\n
Perhaps the attributes \u2018total sulfur dioxide\u2019, \u2018free sulfur dioxide\u2019, and \u2018residual sugar\u2019 have an exponential distribution. Attributes \u2018density\u2019, \u2018pH\u2019, \u2018fixed acidity\u2019, and \u2018Volatile acidity\u2019 have Gaussian or nearly Gaussian distributions.<\/span>
\nLet’s discuss Python Descriptive Statistics<\/a><\/strong><\/p>\n
b. Density Plots<\/strong><\/h4>\n
A density plot appears to be an abstracted histogram. Each bin has a smooth curve drawn through its top. Your eyes can rest now.<\/span><\/p>\n
Features of Density Plots:<\/strong><\/p>\n