Experiment: Perform Data Cleaning, Normalization, and Visualization on IoT Sensor Dataset

Aim

To perform data cleaning, normalization, and visualization on an IoT sensor dataset and prepare the data for further analysis.

Objectives

To understand the importance of data preprocessing in IoT applications.
To inspect and identify missing, duplicate, inconsistent, and anomalous sensor data.
To clean an IoT sensor dataset using appropriate data-cleaning techniques.
To normalize sensor values to a standard numerical range.
To visualize sensor data using suitable graphs.
To compare the dataset before and after data cleaning and normalization.
To understand how data preprocessing improves the quality and usability of IoT sensor data.

Components Required

Sr. No.	Component / Software	Quantity
1	Computer or laptop	1
2	IoT sensor dataset in CSV format	1
3	Python	1
4	Jupyter Notebook / Google Colab / Python IDE	1
5	Pandas library	1
6	NumPy library	1
7	Matplotlib library	1

Theory

Data preprocessing is the process of preparing raw data for analysis by identifying and correcting problems such as missing values, duplicate records, incorrect data types, inconsistent values, and anomalous observations.

IoT systems continuously generate sensor data. The collected data may contain errors due to sensor limitations, communication problems, transmission errors, incorrect entries, or abnormal operating conditions. Therefore, raw IoT data should be cleaned and transformed before it is used for analysis or visualization.

The main stages of data preprocessing performed in this experiment are:

Data inspection: Examining the structure, columns, data types, number of records, and statistical properties of the dataset.
Data cleaning: Identifying and handling missing values, duplicate records, invalid values, and anomalous observations.
Data transformation: Converting data into suitable formats for analysis, such as converting timestamp values into a proper date-time format.
Data normalization: Scaling numerical sensor values to a standard range so that they can be compared and processed more easily.
Data visualization: Representing the processed data graphically to identify patterns, trends, variations, and anomalies.

The dataset used in this experiment is an IoT temperature dataset obtained from the Cognifront cloud platform. The dataset contains the following fields:

Column	Description
ID	Unique record identifier generated for the sensor data record
Name	Name of the sensor variable. In this dataset, it is DHT-Temp
Value	Temperature value recorded by the sensor
Date	Date and time at which the sensor reading was recorded

The original dataset contains 547 records and four columns: ID, Name, Value and Date. The Value column contains temperature readings, while the Date column contains the corresponding timestamps.

Missing values are records where one or more required fields contain no valid value. Missing values may be removed or replaced depending on the nature of the dataset and the purpose of the analysis.

Duplicate records are repeated records that contain the same data more than once. Duplicate records can occur because of repeated data transmission or storage operations and may need to be removed.

Anomalies are observations that differ significantly from the normal pattern of the dataset. In a temperature dataset, an unusually high or low value may represent a sensor error, transmission error, or abnormal environmental condition. Anomalies should be investigated before deciding whether they should be removed, corrected, or retained.

Normalization is the process of transforming numerical values to a common scale. In this experiment, Min-Max normalization is used to transform the temperature values into the range 0 to 1.

The Min-Max normalization formula is:

Normalized Value = (X - Xmin) / (Xmax - Xmin)

where:

X is the original sensor value.
Xmin is the minimum value in the dataset.
Xmax is the maximum value in the dataset.

After normalization, the smallest value becomes 0 and the largest value becomes 1.

Data visualization is used to represent the processed sensor data graphically. A line graph can be used to show temperature variation with time, while a histogram can be used to observe the distribution of temperature values.

The overall data-processing flow is:

Raw IoT CSV Dataset
        ↓
   Data Inspection
        ↓
    Data Cleaning
        ↓
   Data Transformation
        ↓
     Normalization
        ↓
     Visualization
        ↓
  Processed Dataset

Dataset Used

The dataset used for this experiment is a DHT11 temperature dataset exported from the Cognifront cloud platform.

The original dataset contains:

Parameter	Value
Number of records	547
Number of columns	4
Sensor variable	DHT-Temp
Minimum temperature	27.1 °C
Maximum temperature	27.8 °C
Missing values	None in the original dataset
Duplicate timestamps	None in the original dataset

Program

Install the required Python libraries before running the program.

pip install pandas numpy matplotlib

The following program loads the IoT dataset, inspects the data, performs data cleaning, normalizes the temperature values using Min-Max normalization, and generates visualizations.

from google.colab import files
dht_data = files.upload()

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# Load the dataset
df = pd.read_csv('anomalous-dht11-data.csv')

print("Original Dataset")
print(df.head())

print("\nDataset Information")
print(df.info())

print("\nMissing Values")
print(df.isnull().sum())

print("\nDuplicate Records")
print(df.duplicated().sum())

print("\nStatistical Summary")
print(df.describe())

# Convert Date column to datetime
df["Date"] = pd.to_datetime(df["Date"], errors="coerce")

# Convert Value column to numeric
df["Value"] = pd.to_numeric(df["Value"], errors="coerce")

# Remove rows with missing required values
df = df.dropna(subset=["ID", "Name", "Value", "Date"])

# Remove duplicate records
df = df.drop_duplicates()

"""# Now to remove *outliers*, we will use 2 methods:

1.  Using IQR (Interquartile Range) method
2.  Using Domain-based method

For our particular dataset, you will see why Domain-based method is more applicable by looking at the output of IQR method.

**1) IQR (Interquartile Range) method**
"""

# Detect temperature outliers using IQR
Q1 = df["Value"].quantile(0.25)
Q3 = df["Value"].quantile(0.75)
IQR = Q3 - Q1

lower_limit = Q1 - 1.5 * IQR
upper_limit = Q3 + 1.5 * IQR

outliers = df[
    (df["Value"] < lower_limit) |
    (df["Value"] > upper_limit)
]

print("\nDetected Outliers")
print(outliers)

# Remove detected outliers
df_clean = df[
    (df["Value"] >= lower_limit) &
    (df["Value"] <= upper_limit)
].copy()

"""**2) Domain-based method**"""

lower_limit = 15
upper_limit = 50

df_clean = df[
    (df["Value"] >= lower_limit) &
    (df["Value"] <= upper_limit)
].copy()

# Min-Max normalization
min_value = df_clean["Value"].min()
max_value = df_clean["Value"].max()

df_clean["Normalized_Value"] = (
    (df_clean["Value"] - min_value) /
    (max_value - min_value)
)

# Save cleaned and normalized dataset
df_clean.to_csv(
    "cleaned_normalized_dht11_data.csv",
    index=False
)

print("\nCleaned Dataset")
print(df_clean.head())

print("\nNormalized Dataset")
print(df_clean[["Value", "Normalized_Value"]].head())

# Visualization 1: Temperature vs Time
plt.figure(figsize=(10, 5))

plt.plot(
    df_clean["Date"],
    df_clean["Value"]
)

plt.xlabel("Date and Time")
plt.ylabel("Temperature (°C)")
plt.title("DHT11 Temperature vs Time")
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

# Visualization 2: Temperature distribution
plt.figure(figsize=(8, 5))

plt.hist(
    df_clean["Value"],
    bins=10
)

plt.xlabel("Temperature (°C)")
plt.ylabel("Frequency")
plt.title("Distribution of DHT11 Temperature Values")
plt.tight_layout()
plt.show()

# Visualization 3: Normalized temperature
plt.figure(figsize=(10, 5))

plt.plot(
    df_clean["Date"],
    df_clean["Normalized_Value"]
)

plt.xlabel("Date and Time")
plt.ylabel("Normalized Temperature")
plt.title("Normalized DHT11 Temperature vs Time")
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

Code Explanation

files.upload(): Opens a file-upload interface in Google Colab so that the IoT sensor CSV dataset can be uploaded.

pd.read_csv(): Loads the uploaded CSV dataset into a Pandas DataFrame.

df.head(): Displays the first few records of the dataset.

df.info(): Displays information about the columns, data types and number of non-null records.

isnull().sum(): Counts the number of missing values in each column.

duplicated().sum(): Counts the number of duplicate records in the dataset.

df.describe(): Displays statistical information such as count, mean, standard deviation, minimum and maximum values of numerical columns.

pd.to_datetime(): Converts the Date column into a proper date-time format. Invalid date values are converted to missing values using errors="coerce".

pd.to_numeric(): Converts the Value column into numerical data. Invalid values are converted to missing values using errors="coerce".

dropna(): Removes records containing missing values in the required ID, Name, Value or Date fields.

drop_duplicates(): Removes duplicate records from the dataset.

Q1 and Q3: Represent the first and third quartiles of the temperature data and are used for IQR-based outlier detection.

IQR: Represents the Interquartile Range and is calculated as Q3 − Q1.

IQR outlier limits: Values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR are identified as potential outliers.

IQR outlier detection: Identifies and displays potential outliers using the statistical IQR method. In this experiment, the IQR method is demonstrated for comparison but is not used for the final cleaning because the normal temperature readings are closely clustered.

Domain-based outlier detection: Defines an acceptable temperature range of 15 °C to 50 °C. Values outside this range are treated as anomalous temperature readings.

df_clean: Stores the final cleaned dataset after applying the domain-based temperature limits.

Min-Max normalization: Transforms the cleaned temperature values into the range 0 to 1 using the formula:

Normalized Value = (X - Xmin) / (Xmax - Xmin)

to_csv(): Saves the cleaned and normalized dataset as a new CSV file named cleaned_normalized_dht11_data.csv.

plt.plot(): Creates a line graph showing the variation of temperature with respect to date and time.

plt.hist(): Creates a histogram showing the distribution and frequency of the cleaned temperature values.

Normalized temperature plot: Displays the normalized temperature values against time to show the sensor readings after Min-Max normalization.

Procedure

Obtain the DHT11 sensor dataset in CSV format from the Cognifront cloud platform.
Make a copy of the original CSV file and use the copy for data-cleaning experiments.
Open the Python environment such as Jupyter Notebook, Google Colab or a Python IDE.
Install the required Pandas, NumPy and Matplotlib libraries.
Load the CSV dataset using Pandas.
Inspect the dataset and identify its columns, data types and number of records.
Check the dataset for missing values.
Check the dataset for duplicate records.
Convert the Date column into the appropriate date-time format.
Convert the sensor Value column into numerical format.
Identify anomalous sensor values using statistical methods such as the IQR method.
Remove or handle the identified invalid or anomalous records as required.
Normalize the cleaned sensor values using Min-Max normalization.
Save the cleaned and normalized dataset as a new CSV file.
Plot the temperature values against time using a line graph.
Plot the distribution of temperature values using a histogram.
Plot the normalized temperature values against time.
Compare the original and processed datasets and observe the effect of data cleaning and normalization.

Introducing Manual Anomalies

The original Cognifront dataset is relatively clean. Therefore, for demonstrating data-cleaning techniques, controlled anomalies can be introduced into a copy of the original dataset.

Do not modify the original downloaded dataset. Keep the original file as the raw dataset and create a separate file such as:

anomalous-dht11-data.csv

The following anomalies can be introduced for demonstration:

Anomaly	Example	Purpose
Missing value	Replace a Value entry with an empty cell	Demonstrates missing-value handling
Duplicate record	Copy one complete row and insert it again	Demonstrates duplicate detection
Extreme temperature	Change one Value to 80 °C	Demonstrates outlier detection
Very low temperature	Change one Value to -20 °C	Demonstrates anomalous-value detection
Invalid value	Replace one Value with text such as "ERROR"	Demonstrates invalid data handling

For example, if the original data contains:

296900,DHT-Temp,27.5,2026-10-07 08:49:35

the Value can be changed to:

296900,DHT-Temp,80,2026-10-07 08:49:35

Similarly, another record can be changed to:

296950,DHT-Temp,,2026-10-07 08:51:15

to demonstrate a missing sensor value.

An existing row can also be copied and inserted again to create a duplicate record.

The modified dataset can then be processed using the Python program. The program should identify the missing, duplicate and anomalous records and produce a cleaned dataset.

Note: These manually introduced anomalies are artificial and are used only to demonstrate data-cleaning techniques. They should not be presented as actual errors produced by the DHT11 sensor or Cognifront cloud platform.

Observations and Output

The following observations can be recorded during the experiment:

Sr. No.	Operation	Observation
1	Dataset inspection	Dataset contains 547 records and 4 columns
2	Missing-value check	Original dataset contains no missing values
3	Duplicate check	Original dataset contains no duplicate records
4	Data type conversion	Date converted to datetime and Value converted to numeric
5	Anomaly detection	Artificial anomalous records are identified in the modified dataset
6	Data cleaning	Missing, duplicate and anomalous records are handled
7	Normalization	Temperature values are converted to the range 0 to 1
8	Visualization	Temperature variation and distribution are represented graphically
9	CSV export	Cleaned and normalized data is saved as a new CSV file

Result

The IoT sensor dataset was successfully inspected, cleaned and normalized using Python. Missing values, duplicate records and anomalous sensor readings were identified and handled. The temperature values were normalized using Min-Max normalization, and the processed data was visualized using suitable graphs. The cleaned and normalized dataset was successfully exported as a CSV file.

Viva Questions

What is data preprocessing?
Why is data cleaning required for IoT sensor data?
What is the difference between raw data and cleaned data?
What is a missing value?
What is a duplicate record?
What is an anomaly in a sensor dataset?
Why are artificial anomalies introduced during this experiment?
What is the IQR method?
How is an outlier detected using the IQR method?
What is data normalization?
What is Min-Max normalization?
What is the range of values produced by Min-Max normalization?
Why is the Date column converted into datetime format?
What is the purpose of the Pandas library?
What is the purpose of NumPy?
What is Matplotlib used for?
What is the purpose of a histogram?
What information can be obtained from a line graph of sensor data?
Why should the original dataset be preserved before data cleaning?
Why should manually introduced anomalies not be considered actual sensor errors?
What is the purpose of exporting the processed dataset as a CSV file?