In Python, NaN stands for Not a Number. It is used to represent values that are not present in a dataset or file. It is categorized as a special floating-point value and can only be converted to a float data type. Dealing with NaN type is necessary while working on datasets. There are several ways and built-in functions in Python to remove NaN values. In this article, we shall be looking into such ways in Python to remove nan from the list.
Why should we remove NaN values?
While performing data analysis, it is important to remove the NaN values. NaN basically represents data that either does not exist or was not collected. Because of missing data, it might mislead the model. It might affect the accuracy and predictions of the model. Hence, it is important to remove nan values.
Why cannot we use np.nan == np.nan?
When we perform np.nan == np.nan in Python, the output is False as they are not equal.
import numpy as np
print(np.nan == np.nan)
The output is:
False
The reason behind it is that Python does not consider an equivalence relation between two NaN values. Since NaN values are not defined, two NaN values are not equal.
Ways to remove nan from the list
Let us now look at 5 easy and effective ways in Python of removing nan values from a list.
- Using Numpy’s isnan() function
- By using Math’s isnan() function
- Using Pandas isnull() function
- Using for loop
- With list comprehension
1. Python Remove nan from List Using Numpy’s isnan() function
The isnan() function in numpy will check in a numpy array if the element is NaN or not. It returns a numpy array as an output that contains boolean values. The value will be False where the item is not NaN and True where it is NaN.
For that, first, we will have to import the numpy library.
import numpy as np
from numpy import nan
Using np.array(), we shall create a numpy array containing three integer values and three NaN values.
my_array = np.array([10, 25, nan, 15,nan,nan]
Then, we shall wrap the np.logical_not() function around the output of the isnan() function. We do that because we want the non-NaN values to be printed into the new array. By using logical_not(), it will convert the False values into True and vice – versa. So, for non-NaN values, the value will be True, and for NaN values, it will be false. We shall save the new array into the ‘new_array’ variable.
new_array = my_array[np.logical_not(np.isnan(my_array))]
print(new_array)
The output is:
[10. 25. 15.]
The entire code is:
import numpy as np
from numpy import nan
my_array = np.array([10, 25, nan, 15,nan,nan])
new_array = my_array[np.logical_not(np.isnan(my_array))]
print(new_array)
2. By using Math’s isnan() function
Like numpy, python’s math library also has isnan() function. It will return a boolean value – True if the number is NaN and False if it is not NaN.
To use math’s isnan() function, we will first have to import the math library.
import math
from numpy import nan
Then, we shall create a list containing integer values and NaN.
my_list = [10, 25, nan, 15,nan,nan]
The isnan() in the math library will check for one individual number at a time. So, we shall use list comprehension here to iterate over one item and save the new list into ‘new_list’.
new_list = [item for item in my_list if not(math.isnan(item)) == True]
print(new_list)
The output is:
[10, 25, 15]
The Entire Code is:
import math
from numpy import nan
my_list = [10, 25, nan, 15,nan,nan]
new_list = [item for item in my_list if not(math.isnan(item)) == True]
print(new_list)
3. Python Remove nan from List Using Pandas isnull() function
The pandas library in python has a function named isnull() which can be used in python to remove NaN values from the list.
First, we will import the pandas library.
import pandas as pd
from numpy import nan
Then we shall use list comprehension here and run a for loop over the list ‘my_list’. We shall check using not(pd.isnull()) whether the list item is NaN or not and accordingly append that item into a new list named ‘new_list’.
my_list = ['Mike', 'Harry', nan, 'Emma',nan,nan]
new_list = [item for item in my_list if not(pd.isnull(item)) == True]
print(new_list)
The output is:
['Mike', 'Harry', 'Emma']
Also, Read | How to Convert Numpy Array to Pandas Dataframe
The Entire Code is:
import pandas as pd
from numpy import nan
my_list = ['Mike', 'Harry', nan, 'Emma',nan,nan]
new_list = [item for item in my_list if not(pd.isnull(item)) == True]
print(new_list)
Alternatively, we can also use the isna() function present in pandas similarly.
import pandas as pd
from numpy import nan
my_list = ['Mike', 'Harry', nan, 'Emma',nan,nan]
new_list = [item for item in my_list if not(pd.isna(item)) == True]
print(new_list)
The output is:
['Mike', 'Harry', 'Emma']
4. Python Remove nan from List Using for loop
This is the most basic and effective method for removing nan values from the python list. We will run a for loop over the length of the list. If we encounter a not NaN value, we shall append that value to a new list. The new list will not contain any nan values.
First, we will have to import nan from the numpy library.
from numpy import nan
Now, we shall create a list named my_list. The list contains three string values and three NaN values. We shall also define an empty list named ‘new_list.’
my_list = ['Mike', 'Harry', nan, 'Emma',nan,nan]
new_list = []
Now, we shall run a for loop over my_list. Inside the for loop, we shall place an if condition, which will check if the current list item is a NaN value or not. If it is not NaN, then we will append it to the list ‘new_list’. Then at the end, we shall print that list.
for item in my_list:
if str(item) != 'nan':
new_list.append(item)
print(new_list)
The output is:
['Mike', 'Harry', 'Emma']
The entire code is:
from numpy import nan
my_list = ['Mike', 'Harry', nan, 'Emma',nan,nan]
new_list = []
for item in my_list:
if str(item) != 'nan':
new_list.append(item)
print(new_list)
5. With list comprehension
List comprehension is an effective way of generating new sequences from already existing sequences. It is a compact piece of one-line code with which we can loop over a sequence.
The syntax of list comprehension is:
[expression for item in list if-condition]
The expression is the item to be included in the sequence. The expression is followed by a for loop. We can also mention an if condition at the end if required. The code works similarly to using a for loop. The only difference is that it has lesser lines of code and thus more efficient.
from numpy import nan
my_list = ['Mike', 'Harry', nan, 'Emma',nan,nan]
new_list = [item for item in my_list if str(item) != 'nan']
print(new_list)
The output is:
['Mike', 'Harry', 'Emma']
That sums up different ways in python to remove NaN values from the list. If you have any questions in your mind or any thoughts to share, don’t forget to leave them in the comments below.
Until next time, Keep Learning!