pandas.DataFrame.drop#
Remove rows or columns by specifying label names and corresponding axis, or by specifying directly index or column names. When using a multi-index, labels on different levels can be removed by specifying the level. See the user guide <advanced.shown_levels> for more information about the now unused levels.
Parameters labels single label or list-like
Index or column labels to drop. A tuple will be used as a single label and not treated as a list-like.
Whether to drop labels from the index (0 or ‘index’) or columns (1 or ‘columns’).
index single label or list-like
Alternative to specifying axis ( labels, axis=0 is equivalent to index=labels ).
columns single label or list-like
Alternative to specifying axis ( labels, axis=1 is equivalent to columns=labels ).
level int or level name, optional
For MultiIndex, level from which the labels will be removed.
inplace bool, default False
If False, return a copy. Otherwise, do operation inplace and return None.
errors <‘ignore’, ‘raise’>, default ‘raise’
If ‘ignore’, suppress error and only existing labels are dropped.
Returns DataFrame or None
DataFrame without the removed index or column labels or None if inplace=True .
If any of the labels is not found in the selected axis.
Label-location based indexer for selection by label.
Return DataFrame with labels on given axis omitted where (all or any) data are missing.
Return DataFrame with duplicate rows removed, optionally only considering certain columns.
How to drop a list of rows from Pandas dataframe?
Then I want to drop rows with certain sequence numbers which indicated in a list, suppose here is [1,2,4], then left:
How or what function can do that ?
15 Answers 15
Use DataFrame.drop and pass it a Series of index labels:
Note that it may be important to use the «inplace» command when you want to do the drop in line.
If the DataFrame is huge, and the number of rows to drop is large as well, then simple drop by index df.drop(df.index[]) takes too much time.
In my case, I have a multi-indexed DataFrame of floats with 100M rows x 3 cols , and I need to remove 10k rows from it. The fastest method I found is, quite counterintuitively, to take the remaining rows.
Let indexes_to_drop be an array of positional indexes to drop ( [1, 2, 4] in the question).
In my case this took 20.5s , while the simple df.drop took 5min 27s and consumed a lot of memory. The resulting DataFrame is the same.
I solved this in a simpler way — just in 2 steps.
Make a dataframe with unwanted rows/data.
Use the index of this unwanted dataframe to drop the rows from the original dataframe.
Example:
Suppose you have a dataframe df which as many columns including ‘Age’ which is an integer. Now let’s say you want to drop all the rows with ‘Age’ as negative number.
Hope this is much simpler and helps you.
You can also pass to DataFrame.drop the label itself (instead of Series of index labels):
Which is equivalent to:
If I want to drop a row which has let’s say index x , I would do the following:
If I would want to drop multiple indices (say these indices are in the list unwanted_indices ), I would do:
Here is a bit specific example, I would like to show. Say you have many duplicate entries in some of your rows. If you have string entries you could easily use string methods to find all indexes to drop.
And now to drop those rows using their indexes
Use only the Index arg to drop row:-
For multiple rows:-
In a comment to @theodros-zelleke’s answer, @j-jones asked about what to do if the index is not unique. I had to deal with such a situation. What I did was to rename the duplicates in the index before I called drop() , a la:
where rename_duplicates() is a function I defined that went through the elements of index and renamed the duplicates. I used the same renaming pattern as pd.read_csv() uses on columns, i.e., «%s.%d» % (name, count) , where name is the name of the row and count is how many times it has occurred previously.
Determining the index from the boolean as described above e.g.
can be more memory intensive than determining the index using this method
applied like so
This method is useful when dealing with large dataframes and limited memory.
To drop rows with indices 1, 2, 4 you can use:
The tilde operator
negates the result of the method isin . Another option is to drop indices:
Look at the following dataframe df
Lets drop all the rows which has an odd number in column1
Create a list of all the elements in column1 and keep only those elements that are even numbers (the elements that you dont want to drop)
keep_elements = [x for x in df.column1 if x%2==0]
All the rows with the values [2, 4, 6, 8, 10] in its column1 will be retained or not dropped.
We make the column1 as index and drop all the rows that are not required. Then we reset the index back. df
As Dennis Golomazov’s answer suggests, using drop to drop rows. You can select to keep rows instead. Let’s say you have a list of row indices to drop called indices_to_drop . You can convert it to a mask as follows:
You can use this index directly:
The nice thing about this method is that mask can come from any source: it can be a condition involving many columns, or something else.
The really nice thing is, you really don’t need the index of the original DataFrame at all, so it doesn’t matter if the index is unique or not.
The disadvantage is of course that you can’t do the drop in-place with this method.
Delete Rows & Columns in DataFrames Quickly using Pandas Drop
At the start of every analysis, data needs to be cleaned, organised, and made tidy. For every Python Pandas DataFrame, there is almost always a need to delete rows and columns to get the right selection of data for your specific analysis or visualisation. The Pandas Drop function is key for removing rows and columns.
Pandas Drop Cheatsheet
Removing columns and rows from your DataFrame is not always as intuitive as it could be. It’s all about the “DataFrame drop” command. The drop function allows the removal of rows and columns from your DataFrame, and once you’ve used it a few times, you’ll have no issues.
Sample DataFrame
For this post, we’re using data from the WHO COVID tracker, downloaded as at the 1st January 2020 (data here). If you’d like to work with up-to-date data, please change the source URL for the read_csv function in the loading script to this one.
Delete or Drop DataFrame Columns with Pandas Drop
Delete columns by name
Deleting columns by name from DataFrames is easy to achieve using the drop command. There are two forms of the drop function syntax that you should be aware of, but they achieve the same result:
Delete column with pandas drop and axis=1
The default way to use “drop” to remove columns is to provide the column names to be deleted along with specifying the “axis” parameter to be 1.
Delete column with pandas drop “columns” parameter
Potentially a more intuitive way to remove columns from DataFrames is to use the normal “drop” function with the “columns” parameter specifying a single column name or a list of columns.
Delete columns by column number or index
The drop function can be used to delete columns by number or position by retrieving the column name first for .drop. To get the column name, provide the column index to the Dataframe.columns object which is a list of all column names. The name is then passed to the drop function as above.
WARNING: This method can end up in multiple columns being deleted if the names of the columns are repeated (i.e. you have two columns with the same name as the one at index 3).
When you have repeating columns names, a safe method for column removal is to use the iloc selection methodology on the DataFrame. In this case, you are trying to “select all rows and all columns except the column number you’d like to delete”.
To remove columns using iloc, you need to create a list of the column indices that you’d like to keep, i.e. a list of all column numbers, minus the deleted ones.
To create this list, we can use a Python list comprehension that iterates through all possible column numbers ( range(data.shape[1]) ) and then uses a filter to exclude the deleted column indexes ( x not in
). The final deletion then uses an iloc selection to select all rows, but only the columns to keep ( .iloc[:,
).
Delete DataFrame Rows with Pandas Drop
There are three different ways to delete rows from a Pandas Dataframe. Each method is useful depending on the number of rows you are deleting, and how you are identifying the rows that need to be removed.
Deleting rows using “drop” (best for small numbers of rows)
Delete rows based on index value
To delete rows from a DataFrame, the drop function references the rows based on their “index values“. Most typically, this is an integer value per row, that increments from zero when you first load data into Pandas. You can see the index when you run “data.head()” on the left hand side of the tabular view. You can access the index object directly using “data.index” and the values through “data.index.values”.
To drop a specific row from the data frame – specify its index value to the Pandas drop function.
It can be useful for selection and aggregation to have a more meaningful index. For our sample data, the “name” column would make a good index also, and make it easier to select country rows for deletion from the data.
Delete rows based on row number
At times, the DataFrame index may not be in ascending order. To delete a row based on it’s position in the DataFrame, i.e. “delete the second row”, we still use the index of the DataFrame, but select the row from the index directly as we delete. We can also use these index selections to delete multiple rows, or index from the bottom of the DataFrame using negative numbers. For example:
Deleting rows based on a column value using a selection (iloc/loc)
The second most common requirement for deleting rows from a DataFrame is to delete rows in groups, defined by values on various columns. The best way to achieve this is through actually “selecting” the data that you would like to keep. The “drop” method is not as useful here, and instead, we are selecting data using the “loc” indexer and specifying the desired values in the column(s) we are using to select.
There is a full blog post on Pandas DataFrame iloc and loc selection on this blog, but a basic example is here:
Note – if you get the Pandas error: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() , then you have most likely left out the parenthesis “( )” around each condition of your loc selection.
Deleting rows by truncating the DataFrame
One final way to remove rows from the DataFrame is to use Python “slice” notation. Slice notation is well summarised in this StackOverflow post:
The slice notation makes it easy to delete many rows from a DataFrame, while retaining the selected “slice”. For example:
Dropping “inplace” or returning a new DataFrame
The drop function can be used to directly alter a Pandas DataFrame that you are working with, or, alternatively, the return the result after columns or rows have been dropped. This behaviour is controlled with the “inplace” parameter. Using inplace=True can reduce the number of reassignment commands that you’ll need in your application or script. Note that if inplace is set as True, there is no return value from the drop function.
Further Reading and Links
As deleting columns and rows is one of the key operations for DataFrames, there’s a tonne of of excellent content out there on the drop function, that should explain any unusual requirement you may have. I’d be interested in any element of removing rows or columns not covered in the above tutorial – please let me know in the comments.
How to Use the Pandas Drop Technique
In this tutorial, I’ll explain how to drop rows and columns from a dataframe using the Pandas drop method.
I’ll explain what the drop method does, explain the syntax, and show you clear examples.
If you need something specific, you can click on any of the following links.
Table of Contents:
The drop technique is fairly simple to use, but there are a few important details that you should know. So, let’s start with a quick explanation of what it does and how it works.
A quick introduction to Pandas Drop
The Pandas drop method deletes rows and columns from Python dataframes and Series objects.
data wrangling” or data cleaning. So to master data wrangling in Python, you really need to know how to use this technique.
Having said that, exactly how you use it depends on the syntax. That being the case, let’s take a look at the syntax of the drop() method.
The Syntax of Pandas drop
In this section, I’ll show you the syntax to:
We’ll look at those separately, and then I’ll explain some optional parameters afterwards.
A quick note
One quick note before we look at the syntax.
All of these syntax explanations assume that you’ve already imported Pandas and that you have a Pandas dataframe available (or a Series).
You can import Pandas with the following code:
And if you need a refresher on how to create dataframes, you can read our tutorial on Pandas dataframes.
syntax: delete a column
First, let’s look at the syntax to delete a single column.
To delete a column, you type the name of your dataframe, and then .drop() to call the method.
syntax: delete multiple columns
The syntax to delete multiple columns is similar to the syntax to delete a single column.
You type the name of your dataframe and .drop() to call the method. You also still use the columns parameter.
syntax: delete rows
Finally, let’s look at the syntax to delete a row or rows.
The syntax to delete rows is very similar to the previous to syntax variations.
You call the method by typing the name of the dataframe and then .drop() to call the method.
But here, to delete rows, you use the labels parameter.
the dataframe index. You can use either a single row label, or multiple labels inside of a Python list.
This is fairly simple to do, but to do it properly, you really need to understand Python dataframe indexes. If you need a refresher, you can read our tutorial on Pandas indexes.
I’ll show you an example of how to delete rows in example 3.
The parameters of Pandas drop
Now that we’ve looked at the basic syntax of Pandas drop, let’s look at some parameters.
The important parameters that I think you should know are:
- columns
- labels
- inplace
There are a few other parameters, but I think several of them are simply confusing for most beginners and there are a few unnecessary parameters. So, the three above are the ones I recommend using.
Let’s discuss each of them.
columns
The columns parameter enables you to specify the columns that you want to delete.
The argument to this parameter can be a single column name or a list of column names. The column names themselves must be enclosed inside quotes.
I’ll show you how to use the columns parameter in example 1 and example 2.
labels
The labels parameter enables you to specify the rows that you want to delete.
The argument to this parameter can be a single row label or a list of row labels.
The format of the labels depends on how you’ve structured the index. If the labels are integers, the labels you provide will be integers. But if the index labels are strings, then you’ll provide strings to this parameter.
I’ll show you how to use the labels parameter in example 3.
inplace
The inplace parameter enables you to modify your dataframe directly.
Remember: by default, the drop() method produces a new dataframe and leaves the original dataframe unchanged. That’s because by default, the inplace parameter is set to inplace = False .
If you set inplace = True , the drop() method will delete rows or columns directly from the original dataframe. Said differently, if you set inplace = True , Pandas will overwrite your data instead of producing a new dataframe as an output.
Be careful when you use this parameter, since it will overwrite your data.
The output of Pandas drop
By default, the drop() technique outputs a new dataframe and leaves your original dataframe unchanged.
That’s because by default, the inplace parameter is set to inplace = False .
If you set inplace = True , Pandas will directly modify the data you’re operating on instead of producing a new object. Be careful when you use inplace = True , since it will overwrite your data.
Examples: how to drop rows and columns of a Pandas dataframe
Now that we’ve looked at the syntax, let’s take a look at how we can use the drop() method to delete rows and columns of a Python dataframe.
Examples:
Run this code first
Before you run any of the examples, you’ll need to run some preliminary code first.
Specifically, you need to:
- import Pandas
- create a dataframe
Import Pandas
Fist, let’s import Pandas.
You can do that with the following code:
Obviously, we’ll need Pandas to use the Pandas drop technique. We’ll also need Pandas to create our data. Let’s do that next.
Create Dataframe
Here, we’ll create a simple dataframe called sales_data .
To do this, we’ll call the pd.DataFrame() function, but we’ll also set the dataframe index with the set_index() method.
This dataframe contains mock sales data. We’ll be able to use this in our examples.
Let’s quickly print it out so we can see the contents:
As you can see, this dataframe has 3 columns: region , sales , and expenses .
The dataframe also has an index with the names of the salespeople in the data. We’ll be able to use the index to reference the rows and delete specific rows.
So now that we have our dataframe, let’s run some examples.
EXAMPLE 1: Delete a single column from a dataframe
First, let’s start very simple.
Here, we’re going to delete a single column from our dataframe.
To do this, we’ll call the drop method, and we’ll use the columns parameter.
Let’s take a look:
Explanation
This is fairly simple, but let me explain.
Here, we deleted the expenses column.
To do this we typed the name of the dataframe, and then .drop() to call the method.
Inside the parenthesis, we used the code columns = ‘expenses’ to specify that we want to drop the expenses column. Note that the name of the column is inside quotation marks (i.e., it’s presented as a string).
In the output, we see that the entire expenses column has been removed.
Also note: the output is a new dataframe and the original data remains unchanged. This is because by default, the inplace parameter is set to inplace = False . When inplace = False , drop() will output a new dataframe but leave the original dataframe unchanged.
I’ll show you how to directly modify the original dataframe in example 4.
EXAMPLE 2: Delete multiple columns from a dataframe
Next, let’s delete multiple columns from a Pandas dataframe.
To do this, we’ll still use the columns parameter.
But instead of providing a single column name as the argument, we’ll provide a list of column names.
Specifically, here, we’ll delete the region variable and the expenses variable.
Let’s take a look:
Explanation
In the output we see that both the region variable and the expenses variable have been removed.
To do this, we called the drop() method, but we used the columns parameter to specify multiple variables to drop.
Specifically, inside the parenthesis, we used the code columns = [‘region’,’expenses’] to indicate that we want to remove the region variable and the expenses variable. Notice that the names of these variables are inside quotations (i.e., they are presented as strings). Furthermore, they are passed to the columns parameter as a list of variable names.
Keep in mind that here, we only deleted two variables. But if you have a larger dataframe and you want to delete many more variables, you can simply create a list of all of the names you want to delete.
EXAMPLE 3: Drop specific rows from a dataframe
Now, let’s drop some rows from our dataframe.
Deleting rows is very similar to deleting columns. But instead of using the columns we’ll use the labels parameter.
By using the labels parameter, we can specify specific rows to delete by the index label.
Let’s take a look:
Explanation
Here, we deleted the records for William and Paulo. We did this with the code drop(labels = [‘William’,’Paulo’]) .
The labels parameter enables us to delete rows by index label and the list of values (i.e., [‘William’,’Paulo’] ) indicate exactly which rows to remove.
This is fairly simple, but to really understand it, you need to understand what a dataframe index is. If you need a refresher, you should check out our tutorial on Pandas indexes.
Note that in this example, we deleted multiple rows, so we presented the labels inside of a Python list, similar to deleting multiple columns like we did in example 2.
EXAMPLE 4: Delete columns and modify the data “in place”
Finally, let’s directly modify our data by deleting a column “in place.”
Remember: when we use the drop() method, the technique produces a new dataframe as the output by default, and leaves the original dataframe unchanged.
We can change this behavior by setting inplace = True .
Let’s take a look, and then I’ll explain.
Create dataframe copy
Before we run the example, we’ll first create a copy of the data.
That’s because we’ll be directly modifying our data. As a safeguard, we’ll work with a copy right now.
If you inspect this data, you’ll see that it’s the same as sales_data .
Drop column “in place”
Ok. Now, we’ll drop a column directly from sales_data_copy .
And let’s print out the data:
Explanation
When you run the code and then look at sales_data_copy , you can see that the expenses variable has been permanently removed from the dataframe.
Remember: when we use the drop() technique with inplace = True , Pandas will directly operate on the dataframe.
This is in contrast with inplace = False . If you set inplace = False (which is the default behavior), Pandas will produce a new dataframe and leave the original unchanged.
So when you use inplace = True , Pandas will directly change your data. This can be dangerous. Before you use this, you should test your code to make sure that it works properly!
Frequently asked questions about Pandas drop
Now that we’ve looked at some examples, let’s look at some common questions about the drop() technique.
Frequently asked questions:
Question 1: I used the drop method, but my dataframe is unchanged. Why?
If you use the drop method, you might notice that your original dataframe remains unchanged after you call the method.
For example, in example 1, we used the following code:
If you print out sales_data after you run the code, you’ll realize that sales_data is unchanged. The expenses column is still there.
That’s because the drop() method produces a new dataframe, and leaves both original dataframes unchanged.
By default, the output of the method is sent to the console. We can see the output in the console, but to save it, we need to store it with a name.
For example, you could store the output like this:
You can name the output whatever you want. You could even name it with the original name sales_data .
Alternatively, you can set inplace = True , which will also overwrite your original dataset. I showed an example of this in example 4.
But be careful, if you use either of these techniques, they will overwrite your original dataset. Make sure that you check your code so it works properly before you overwrite an input dataframe.
Question 2: What does the axis parameter do?
The axis parameter is an alternative way of controlling whether you delete rows or columns.
Personally, I think that the use of the axis parameter for the drop() method is very poorly designed. I won’t go into the details, but the way the Pandas developers implemented this parameter makes it very confusing to work with.
The good news is that there’s another way. You can completely skip using the axis parameter.
Instead, you can use the columns parameter when you want to delete columns, and you can use the labels parameter when you want to delete rows.
I show the syntax for using these other parameters these in the syntax section, and I show examples of deleting columns and rows in example 1, example 2, and example 3.
Leave your other questions in the comments below
Do you have any other questions about the Pandas drop method?
Is there something else that you need to understand that I haven’t covered here?
If so, leave your question in the comments section below.
To learn more about Pandas, sign up for our email list
This tutorial should have given you a good introduction to the Pandas drop technique, but if you really want to master data manipulation and data science in Python, there’s a lot more to learn.
So if you’re ready to learn more about Pandas and more about data science, then sign up for our email newsletter.
We publish FREE tutorials almost every week on:
- Base Python
- NumPy
- Pandas
- Scikit learn
- Machine learning
- Deep learning
- … and more.
When you sign up for our email list, we’ll deliver these free tutorials directly to your inbox.
Sign up for FREE data science tutorials
If you want to master data science fast, sign up for our email list.
When you sign up, you’ll receive FREE weekly tutorials on how to do data science in R and Python.