pandas.DataFrame.duplicated#
Only consider certain columns for identifying duplicates, by default use all of the columns.
keep <‘first’, ‘last’, False>, default ‘first’
Determines which duplicates (if any) to mark.
first : Mark duplicates as True except for the first occurrence.
last : Mark duplicates as True except for the last occurrence.
False : Mark all duplicates as True .
Boolean series for each duplicated rows.
Equivalent method on index.
Equivalent method on Series.
Remove duplicate values from Series.
Remove duplicate values from DataFrame.
Consider dataset containing ramen rating.
By default, for each set of duplicated values, the first occurrence is set on False and all others on True.
By using ‘last’, the last occurrence of each set of duplicated values is set on False and all others on True.
Dealing with duplicates in pandas DataFrame
Sometimes, you get a messy dataset. For example, you may have to deal with duplicates, which will skew your analaysis.
Checking for duplicates
First of all, you may want to check if you have duplicate records. If you don’t, you may not need the rest of this post at all. This checks if the whole row appears elsewhere with the same values in each column.
df.duplicated()
You can see that this returns a pandas Series, not a DataFrame.
df.duplicated(‘col1’)
This checks if there are duplicate values in a particular column of your DataFrame.
Getting rid of duplicates
Getting rid of duplicate records is easy. Just use:
df.drop_duplicates()
In this case, it’s pointless as I have no duplicates but you can see that when you run this, it returns a DataFrame without duplicates.
Dropping duplicates from a particular column
Sometimes, you may want to drop duplicates just from one column.
df.drop_duplicates([‘col1’])
Here, you drop duplicates from column1. Alternatively, you can add ‘keep’ and indicate whether you’d like to keep the first argument (keep=’first’), the last argument (keep=’last’) from the duplicates or drop all the duplicates altogether (keep=False). The default is ‘first’ so if you are happy with that, you don’t need to include this.
How to find duplicate names using pandas?
I have a pandas.DataFrame with a column called name containing strings. I would like to get a list of the names which occur more than once in the column. How do I do that?
But it doesn’t filter out the singleton names.
6 Answers 6
If you want to find the rows with duplicated name (except the first time we see that), you can try this
A one liner can be:
the index contains a method for finding duplicates, columns does not seem to have a similar method..
value_counts will give you the number of duplicates as well.
Most of the responses given demonstrate how to remove the duplicates, not find them.
The following will select each row in the data frame with a duplicate ‘name’ field. Note that this will find each instance, not just duplicates after the first occurrence. The keep argument accepts additional values that can exclude either the first or last occurrence.
Pandas Get List of All Duplicate Rows
Pandas DataFrame.duplicated() function is used to get/find/select a list of all duplicate rows(all or selected columns) from pandas. Duplicate rows means, having multiple rows on all columns. Using this method you can get duplicate rows on selected multiple columns or all columns. In this article, I will explain these with several examples.
1. Quick Examples of Get List of All Duplicate Items
If you are in a hurry, below are some quick examples of how to get a list of all duplicate rows in pandas DataFrame.
Now, let’s create a DataFrame with a few duplicate rows on all columns. Our DataFrame contains column names Courses , Fee , Duration , and Discount .
Yields below output.
2. Select Duplicate Rows Based on All Columns
You can use df[df.duplicated()] without any arguments to get rows with the same values on all columns. It takes defaults values subset=None and keep=‘first’ . The below example returns two rows as these are duplicate rows in our DataFrame.
Yields below output.
You can set 'keep=False' in the duplicated function to get all the duplicate items without eliminating duplicate rows.
Yields below output.
3. Get List of Duplicate Last Rows Based on All Columns
You want to select all the duplicate rows except their last occurrence, we must pass a keep argument as ”last" . For instance, df[df.duplicated(keep='last')] .
Yields below output.
4. Get List Of Duplicate Rows Using Single Columns
You want to select duplicate rows based on single columns then pass the column name as an argument.
Yields below output.
5. Get List Of Duplicate Rows Using Multiple Columns
To get/find duplicate rows on the basis of multiple columns, specify all column names as a list.
Yields below output.
6. Get List Of Duplicate Rows Using Sort Values
Let’s see how to sort the results of duplicated() method. You can sort pandas DataFrame by one or multiple (one or more) columns using sort_values() method.
Yields below output.
You can use sort_values("Discount") instead to sort after duplicate filter.
Yields below output.
7. Complete Example For Get List of All Duplicate Items
Conclusion
In this article, you have learned how to get/select a list of all duplicate rows (all or multiple columns) using pandas DataFrame duplicated() method with examples.