Pandas dataframe filter with Multiple conditions
Selecting or filtering rows from a dataframe can be sometime tedious if you don’t know the exact methods and how to filter rows with multiple conditions
In this post we are going to see the different ways to select rows from a dataframe using multiple conditions
Let’s create a dataframe with 5 rows and 4 columns i.e. Name, Age, Salary_in_1000 and FT_Team(Football Team)
import pandas as pd
df=pd.DataFrame({'Name':['JOHN','ALLEN','BOB','NIKI','CHARLIE','CHANG'],
'Age':[35,42,63,29,47,51],
'Salary_in_1000':[100,93,78,120,64,115],
'FT_Team':['STEELERS','SEAHAWKS','FALCONS','FALCONS','PATRIOTS','STEELERS']})
df
Output:
- | Name | Age | Salary_in_1000 | FT_Team |
---|---|---|---|---|
0 | JOHN | 35 | 100 | STEELERS |
1 | ALLEN | 42 | 93 | SEAHAWKS |
2 | BOB | 63 | 78 | FALCONS |
3 | NIKI | 29 | 120 | FALCONS |
4 | CHARLIE | 47 | 64 | PATRIOTS |
5 | CHANG | 51 | 115 | STEELERS |
Selecting Dataframe rows on multiple conditions using these 5 functions
In this section we are going to see how to filter the rows of a dataframe with multiple conditions using these five methods
a) loc b) numpy where c) Query d) Boolean Indexing e) eval
What’s the Condition or Filter Criteria ?
Get all rows having salary greater or equal to 100K and Age < 60 and Favourite Football Team Name starts with ‘S’
Using loc with multiple conditions
loc is used to Access a group of rows and columns by label(s) or a boolean array
As an input to label you can give a single label or it’s index or a list of array of labels
Enter all the conditions and with & as a logical operator between them
df.loc[(df['Salary_in_1000']>=100) & (df['Age']< 60) & (df['FT_Team'].str.startswith('S')),['Name','FT_Team']]
Output:
Name | FT_Team | |
---|---|---|
0 | JOHN | STEELERS |
5 | CHANG | STEELERS |
Using np.where with multiple conditions
numpy where can be used to filter the array or get the index or elements in the array where conditions are met. You can read more about np.where in this post
Numpy where with multiple conditions and & as logical operators outputs the index of the matching rows
import numpy as np
idx = np.where((df['Salary_in_1000']>=100) & (df['Age']< 60) & (df['FT_Team'].str.startswith('S')))
Output:
(array([0, 5], dtype=int64),)
The output from the np.where, which is a list of row index matching the multiple conditions is fed to dataframe loc function
df.loc[idx]
Output:
Name | Age | Salary_in_1000 | FT_Team | |
---|---|---|---|---|
0 | JOHN | 35 | 100 | STEELERS |
5 | CHANG | 51 | 115 | STEELERS |
Using Query with multiple Conditions
It is used to Query the columns of a DataFrame with a boolean expression
df.query('Salary_in_1000 >= 100 & Age < 60 & FT_Team.str.startswith("S").values')
Output:
Name | Age | Salary_in_1000 | |
---|---|---|---|
0 | JOHN | 35 | 100 |
5 | CHANG | 51 | 115 |
pandas boolean indexing multiple conditions
It is a standrad way to select the subset of data using the values in the dataframe and applying conditions on it
We are using the same multiple conditions here also to filter the rows from pur original dataframe with salary >= 100 and Football team starts with alphabet ‘S’ and Age is less than 60
df[(df['Salary_in_1000']>=100) & (df['Age']<60) & df['FT_Team'].str.startswith('S')][['Name','Age','Salary_in_1000']]
Output:
Name | Age | Salary_in_1000 | |
---|---|---|---|
0 | JOHN | 35 | 100 |
5 | CHANG | 51 | 115 |
Pandas Eval multiple conditions
Evaluate a string describing operations on DataFrame column. It Operates on columns only, not specific rows or elements
df[df.eval("Salary_in_1000>=100 & (Age <60) & FT_Team.str.startswith('S').values")]
Output:
Name | Age | Salary_in_1000 | |
---|---|---|---|
0 | JOHN | 35 | 100 |
5 | CHANG | 51 | 115 |
Conclusion:
In this post we have seen that what are the different methods which are available in the Pandas library to filter the rows and get a subset of the dataframe
And how these functions works: loc works with column labels and indexes, whereas eval and query works only with columns and boolean indexing works with values in a column only
Let me know your thoughts in the comments section below if you find this helpful or knows of any other functions which can be used to filter rows of dataframe using multiple conditions