How to use Pandas Value_Counts
This tutorial will explain how to use the Pandas value_counts method to count the values in a Python dataframe.
It explains what value_counts does, how the syntax works, and it provides step-by-step examples.
If you need something specific, you can click on any of the following links.
Table of Contents:
Ok. Let’s get into the details.
A quick introduction to the Pandas value_counts method
First, let’s just start with an explanation of what the value_counts technique does.
Essentially, value_counts counts the unique values of a Pandas object. We often use this technique to do data wrangling and data exploration in Python.
The syntax of value_counts
Ok. Let’s look at the syntax of the Pandas value_counts technique.
Here, I’ll divide this up into different sections, so we can look at the syntax for how to use value_counts on Series objects and how to use value counts on dataframes.
A quick note
The following syntax explanations assume that you’ve imported Pandas, and that you’ve already created a Pandas dataframe or a Pandas series.
You can import Pandas with this code:
And for more information about dataframes, you can read our introduction to Pandas dataframes.
syntax to use value_counts on a Pandas dataframe
First, let’s look at the syntax for how to use value_counts on a dataframe.
This is really simple. You just type the name of the dataframe then .value_counts() .
syntax to use value_counts on a Pandas Series
Next, let’s look at the syntax to use value_counts on a Series object.
The syntax for a Series is almost the same as the syntax for a dataframe:
You simply type the name of the Series object, and then .value_counts() .
Additionally, there are some optional parameters that you can use, which we’ll discuss in the parameters section.
syntax to use value_counts on a dataframe column
Finally, let’ look at how to use value_counts on a column inside of a dataframe.
Remember: individual dataframe columns are Series objects.
So to call value_counts on a column, we first use “dot syntax” to retrieve an individual column. For example, if your dataframe is named your_dataframe and the column you want to retrieve is called column , you would start by typing your_dataframe.column .
The parameters of value counts
The Pandas value_counts technique has several parameters that you can use which will change how the technique works and what exactly it does.
- ascending
- sort
- normalize
- subset
- dropna
In addition, there is the bins parameter, which I rarely use and won’t discuss here.
It’s important to note that all of these parameters are optional.
It’s also important to note that most of these parameters – ascending , sort , and normalize – are used for both the Series syntax and the dataframe syntax.
On the other hand, subset is only available when you use value_counts on dataframes, and dropna is only available when you use value_counts on Series.
Having said all of that, let’s look at each of these parameters individually.
ascending
By default, value_counts will sort the data by numeric count in descending order.
The ascending parameter enables you to change this.
When you set ascending = True , value counts will sort the data by count from low to high (i.e., ascending order).
I’ll show you an example of this in example 4.
The sort parameter controls how the output is sorted.
By default, value_counts sorts the data by the numeric count.
You can change this and sort the data by categories instead by setting sort = False .
I’ll show you an example of this in example 5.
normalize
The normalize parameter changes the form of the output.
By default, value_counts shows the count of the unique values.
But if you set normalize = True , value_counts will display the proportion of total records instead of the raw count.
I’ll show you an example of this in example 6.
subset
The subset parameter enables you to specify a subset of columns on which to apply value_counts, when you use value_counts on a dataframe.
The argument to this parameter should be a list (or list-like object) of column names.
So for example, if you want to use value counts on var_1 and var_2 in a dataframe, you would use the code your_dataframe.value_counts(subset = [‘val_1′,’var_2’]) .
NOTE: again, this parameter is works when you use value_counts on a whole dataframe.
I’ll show you an example of this in example 7.
dropna
The dropna parameter enables you to show ‘NA’ values (i.e., NaN values).
You can do this by setting dropna = False .
NOTE: this parameter is only available for Pandas Series objects and individual dataframe columns. This parameter will not work if you use value_counts on a whole dataframe.
I’ll show you an example of this in example 2.
Examples: Get Value Counts for Pandas Dataframes and Series Objects
Now that we’ve looked at the syntax, let’s look at some examples of how to use the value_counts technique.
Examples:
Run this code first
Before you run the examples, you’ll need to run some preliminary code in order to:
- import necessary packages
- get a dataframe
- create a dataframe subset that we can work with
Let’s do those one at a time.
Import Packages
First, let’s import two packages that we’ll need.
Specifically, we’ll need to import Pandas and Seaborn.
You can do that with the following code:
Obviously, we’ll need Pandas to use the value_counts() technique. But we’ll also need Seaborn, because we’ll be using the titanic dataframe which we can load from Seaborn’s pre-installed datasets.
Get dataframe
Next, let’s get the dataframe we’ll be working with.
In the following examples, we’ll be using the titanic dataset, or some subset of it.
So here, let’s load the dataset from Seaborn:
Additionally, let’s print it out, so we can see the contents:
There are 15 columns in this dataframe, which will be a little difficult to work with if we’re using the value_counts() technique.
That said, let’s quickly create a subset that we can use with some of our examples.
Create Dataframe Subset
Now, let’s create a subset of the titanic dataframe.
Here, we’ll create a subset that contains two variables: sex and embarked .
To subset down to these two variables, we’ll use the Pandas filter method:
For some of our examples, this subset will simply be easier to work with, since it has only 2 variables.
EXAMPLE 1: Use value_counts on a dataframe column
First, let’s use the value_counts technique on a single column.
Here, we’ll use value_counts on the embarked variable in the titanic dataframe.
Let’s run the code, and then I’ll explain:
Explanation
The code to perform this operation is a single line of code, but in some sense, it’s a two step process.
In this code, we’re:
- retrieving the embarked variable with “dot syntax”
- calling the value_counts() method
So, we’re retrieving the embarked variable with the code titanic.embarked .
But after that, we’re calling the value counts method with .value_counts() .
In the output, you see the unique values of the embarked variable – S , C , and Q – and the counts associated with each of those values.
EXAMPLE 2: Include ‘NA’ values in the counts (Series only)
Next, let’s include the ‘NA’ values (i.e., NaN ) in the output. This will enable us to see the number of ‘missing’ values for the variable, if there are any.
Keep in mind that here, we’re still going to operate on a single dataframe variable.
Explanation
Here, we’ve called value_counts() just like we did in example 1.
The only difference is that we included the code dropna = False inside the parenthesis.
As you can see in the output, there is now a count of the number of NaN values (i.e., “missing” values).
This can be useful if you need to identify missing values to clean them up, etc.
NOTE: This will only work if you use value_counts() on a Pandas Series or a dataframe column. It will not work if you try to use value_counts on an entire Pandas dataframe (like in example 3).
EXAMPLE 3: Use value_counts on an entire Pandas dataframe
In the last two examples, we used value_counts on a single column of a dataframe (i.e., a Pandas series object).
Now, let’s use value_counts on a whole dataframe.
Here, we’re going to use value counts on the titanic_subset dataframe. (Remember, we created this subset earlier. It has only two variables to make it easier to work with.)
Ok. Let’s run the code:
Explanation
This is really straight forward.
To do this, we simply typed the name of the dataframe, and then .value_counts() .
You can see that the output is a count of the unique combinations of the variables in the dataframe.
Notice as well that the output is sorted in descending order. That’s the default, but we can change it as well, which we’ll do in the next example.
EXAMPLE 4: Sort the output in ascending order
In this example, we’ll sort the output in ascending order.
Remember that by default, value_counts sorts the output in descending order.
We can change that behavior though with the ascending parameter.
Let’s take a look:
Explanation
Here, we see the counts of the unique combinations of values in the dataframe.
But now, because we set ascending = True , the output is sorted in ascending order … it’s sorted from low to high.
EXAMPLE 5: Sort by category (instead of count)
Now, let’s remove the sorting altogether.
To do this, we’ll call the method with sort = False .
Explanation
Notice in the output, the data are not sorted by the value counts (i.e., the numbers).
Instead, the data are sorted by the categories. The unique categorical values in both variables are sorted in alphabetical order.
Personally, I think this is easier to read, but it does depend on what you’re doing.
There may be some applications where this is better, and there may be some instances where it’s better to sort the data by the numeric counts (like the default behavior).
In any case, you have a choice.
EXAMPLE 6: Compute proportions (i.e., normalize the value counts)
In this example, let’s compute the proportions of each unique combination of values.
In the previous examples, value_counts provided a count of the number of values.
Here, we’ll tell value_counts to compute the percent of total records, using the normalize parameter:
Explanation
The output here is somewhat similar to the output for example 3, in the sense that it’s sorted in descending order of frequency.
But instead of showing the raw counts of each unique combination of categories, it’s showing the proportion. Notice that if you add all the numbers up, they add up to 1.
So again, the numbers represent the proportion of total records accounted for by each unique combination.
EXAMPLE 7: Operate on a subset of dataframe columns (dataframes only)
In the previous examples, I’ve shown you how to use value_counts on a pandas Series, a small Pandas dataframe (with only 2 columns), or a single dataframe column.
Here, I’ll show you how to operate a large dataframe with many columns.
But we’ll use the subset parameter to reduce the size and complexity of the output.
So here, we’ll be working with the full titanic dataframe, which has 15 columns. We’ll use the subset parameter to operate only on two of those variables: sex and embarked .
Let’s take a look.
Explanation
Here, we’re working with the full titanic dataset. Remember: this is the full dataset with 15 variables (instead of the smaller titanic_subset dataframe, which only has 2 variables).
So here, we’re taking the full titanic dataframe with 15 variables and using value_counts on only 2 variables. To do this, we’re setting subset = [‘sex’,’embarked’] .
Notice that syntactically, each variable we want to include is presented as a string (inside of quotation marks). And the collection of variable names is organized into a Python list.
Frequently asked questions about value_counts
Now that you’ve learned about value_counts and seen some examples, let’s review some frequently asked questions.
Frequently asked questions:
Question 1: Can you use the dropna parameter when you operate on a dataframe?
The dropna parameter is very useful for identifying missing values, but unfortunately, you can only use this parameter when you operate on a single dataframe column or Pandas series.
Leave your other questions in the comments below
Do you have any other questions about the Pandas value_counts technique?
Is there something that you’re struggling with that I haven’t covered here?
If so, leave your question in the comments section below.
To learn more about Pandas, sign up for our email list
This tutorial should have helped you understand the value_counts technique, and how it works.
But if you want to master data cleaning and data wrangling with Pandas, there’s a lot more to learn.
And there’s even more to learn if you want to learn data science in Python, more broadly.
That said, if you’re ready to learn more about Pandas and data science in Python, then sign up for our email list.
When you sign up, you’ll get free tutorials on:
- NumPy
- Pandas
- Base Python
- Scikit learn
- Machine learning
- Deep learning
- … and more.
We publish free data science tutorials every week. When you sign up for our email list, we’ll deliver these free tutorials directly to your inbox.
Sign up for FREE data science tutorials
If you want to master data science fast, sign up for our email list.
When you sign up, you’ll receive FREE weekly tutorials on how to do data science in R and Python.
Pandas Count Occurrences in Column – i.e. Unique Values
In this Pandas tutorial, you are going to learn how to count occurrences in a column. There are occasions in data science when you need to know how many times a given value occurs. This can happen when you, for example, have a limited set of possible values that you want to compare. Another example can be if you want to count the number of duplicate values in a column. Furthermore, we may want to count the number of observations there is in a factor or we need to know how many men or women there are in the data set, for example.
Table of Contents
Outline
In this post, you will learn how to use Pandas value_counts() method to count the occurrences in a column in the dataframe. First, we start by importing the needed packages and then we import example data from a CSV file. Second, we will start looking at the value_counts() method and how we can use this to count distinct occurrences in a column. Third, we will count the number of occurrences of a specific value in the dataframe. In the last section, we will have a look at an alternative method that also can be used: the groupby() method together with size() and count() . Now, let’s start by importing Pandas and some example data to play around with!
How do you Count the Number of Occurrences in a data frame?
To count the number of occurrences in e.g. a column in a dataframe you can use Pandas value_counts() method. For example, if you type df['condition'].value_counts() you will get the frequency of each unique value in the column “condition”.
Now, before we use Pandas to count occurrences in a column, we are going to import some data from a .csv file.
Importing the Packages and Data
In the code example above, we first imported Pandas and then we created a string variable with the URL to the dataset. In the last line of code, we imported the data and named the dataframe “df”. Note, we used the index_col parameter to set the first column in the .csv file as index column. Briefly explained, each row in this dataset includes details of a person who has been arrested. This means, and is true in many cases, that each row is one observation in the study. If you store data in other formats refer to the following tutorials:
In this tutorial, we are mainly going to work with the “sex” and “age” columns. It may be obvious but the “sex” column classifies an individual’s gender as male or female. The age is, obviously, referring to a person’s age in the dataset. We can take a quick peek of the dataframe before counting the values in the chosen columns:
If you have another data source and you can also add a new column to the dataframe. Although, we get some information about the dataframe using the head() method you can get a list of column names using the column() method. Many times, we only need to know the column names when counting values. Note, if needed you can also use Pandas to rename a column in the dataframe.
Of course, in most cases, you would count occurrences in your own data set but now we have data to practice counting unique values with. In fact, we will now jump right into counting distinct values in the column “sex”. That said, we are ready to use Pandas to count occurrences in a column, in our dataset.
How to Count Occurences in a Column with Pandas value_counts()
Here’s how to count occurrences (unique values) in a column in Pandas dataframe:
As you can see, we selected the column “sex” using brackets (i.e. df['sex'] ), and then we just used the value_counts() method. Note, if we want to store the counted values as a variable we can create a new variable. For example, gender_counted = df['sex'].value_counts() would enable us to fetch the number of men in the dataset by its index (0, in this case).
As you can see, the method returns the count of all unique values in the given column in descending order, without any null values. By glancing at the above output we can, furthermore, see that there are more men than women in the dataset. In fact, the results show us that the vast majority are men.
Now, as with many Pandas methods, value_counts() has a couple of parameters that we may find useful at times. For example, if we want the reorder the output such as that the counted values (male and female, in this case) are shown in alphabetical order we can use the ascending parameter and set it to True :
Note, both of the examples above will drop missing values. That is, they will not be counted at all. There are cases, however, when we may want to know how many missing values there are in a column as well. In the next section, we will therefore have a look at another parameter that we can use (i.e., dropna ). First, however, we need to add a couple of missing values to the dataset:
In the code above, we used Pandas iloc method to select rows and NumPy’s nan to add the missing values to these rows that we selected. In the next section, we will count the occurrences including the 10 missing values we added, above.
Pandas Count Unique Values and Missing Values in a Column
Here’s a code example to get the number of unique values as well as how many missing values there are:
Looking at the output we can see that there are 10 missing values (yes, yes, we already knew that!).
Getting the Relative Frequencies of the Unique Values
Now that we have counted the unique values in a column we will continue by using another parameter of the value_counts() method: normalize . Here’s how we get the relative frequencies of men and women in the dataset:
This may be useful if we not only want to count the occurrences but want to know e.g. what percentage of the sample that are male and female. Before moving on to the next section, let’s get some descriptive statistics of the age column by using the describe() method:
Naturally, counting age as we did earlier, with the column containing gender, would not provide any useful information. Here’s the data output from the above code:
We can see that there are 5226 values of age data, a mean of 23.85, and a standard deviation of 8.32. Naturally, counting the unique values of the age column would produce a lot of headaches but, of course, it could be worse. In the next example, we will have a look at counting age and how we can bin the data. This is useful if we want to count e.g. continuous data.
Creating Bins when Counting Distinct Values
Another cool feature of the value_counts() method is that we can use the method to bin continuous data into discrete intervals. Here’s how we set the parameter bins to an integer representing the number of bins to create bins:
For each bin, the range of age values (in years, naturally) is the same. One contains ages from 11.45 to 22.80 which is a range of 10.855. The next bin, on the other hand, contains ages from 22.80 to 33.60 which is a range of 11.8. in this example, you can see that all ranges here are roughly the same (except the first, of course). However, each range of age values can contain a different count of the number of persons within this age range. We can see that most people, that are arrested are under 22.8, followed by under 33.6. It kind of makes sense, in this case, right? In the next section, we will have a look at how we can use count the unique values in all columns in a dataframe.
Count the Frequency of Occurrences Across Multiple Columns
Naturally, it is also possible to count the occurrences in many columns using the value_counts() method. Now, we are going to start by creating a dataframe from a dictionary:
As you can see in the output, above, we have a smaller data set which makes it easier to show how to count the frequency of unique values in all columns. If you need, you can convert a NumPy array to a Pandas dataframe, as well. That said, here’s how to use the apply() method:
What we did, in the code example above, was to use the method with the value_counts method as the only parameter. This will apply this method to all columns in the Pandas dataframe. However, this really not a feasible approach if we have larger datasets. In fact, the unique counts we get for this rather small dataset is not that readable:
Counting the Occurences of a Specific Value in Pandas Dataframe
It is, of course, also possible to get the number of times a certain value appears in a column. Here’s how to use Pandas value_counts() , again, to count the occurrences of a specific value in a column:
In the example above, we used the dataset we imported in the first code chunk (i.e., Arrest.csv). Furthermore, we selected the column containing gender and used the value_counts() method. Because we wanted to count the occurrences of a certain value we then selected Male. The output shows us that there are 4783 occurrences of this certain value in the column.
As often, when working with programming languages, there are more approaches than one to solve a problem. Therefore, in the next example, we are going to have a look at some alternative methods that involve grouping the data by category using Pandas groupby() method.
Counting the Frequency of Occurrences in a Column using Pandas groupby Method
In this section, we are going to learn how to count the frequency of occurrences across different groups. For example, we can use size() to count the number of occurrences in a column:
Another method to get the frequency we can use is the count() method:
Now, in both examples above, we used the brackets to select the column we want to apply the method on. Just as in the value_counts() examples we saw earlier. Note that this produces the exact same output as using the previous method and to keep your code clean I suggest that you use value_counts() . Finally, it is also worth mentioning that using the count() method will produce unique counts, grouped, for each column. This is clearly redundant information:
Conclusion: Pandas Count Occurences in Column
In this Pandas tutorial, you have learned how to count occurrences in a column using 1) value_counts() and 2) groupby() together with size() and count() . Specifically, you have learned how to get the frequency of occurrences in ascending and descending order, including missing values, calculating the relative frequencies, and binning the counted values.
Как посчитать количество значений в столбце pandas
Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric python packages. Pandas is one of those packages and makes importing and analyzing data much easier.
Pandas series is a One-dimensional ndarray with axis labels. The labels need not be unique but must be a hashable type. The object supports both integer- and label-based indexing and provides a host of methods for performing operations involving the index.
value_counts() Method: Count Unique Occurrences of Values in a Column
In pandas, for a column in a DataFrame, we can use the value_counts() method to easily count the unique occurences of values.
There’s additional interesting analyis we can do with value_counts() too. We’ll try them out using the titanic dataset.
Import Module¶
Get Titanic Dataset¶
We’ll use the titanic dataset included in the seaborn library.
Below is a preview of the first few rows of the dataset.
Each row includes details of a person who boarded the famous Titanic cruise ship.
In this tutorial, we’re just going to utilize the sex and fare columns. The sex column classifies the person’s gender as male or female. The fare column indicates the dollar amount each person paid to board the Titanic.
| survived | pclass | sex | age | sibsp | parch | fare | embarked | class | who | adult_male | deck | embark_town | alive | alone | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 3 | male | 22.0 | 1 | 0 | 7.2500 | S | Third | man | True | NaN | Southampton | no | False |
| 1 | 1 | 1 | female | 38.0 | 1 | 0 | 71.2833 | C | First | woman | False | C | Cherbourg | yes | False |
| 2 | 1 | 3 | female | 26.0 | 0 | 0 | 7.9250 | S | Third | woman | False | NaN | Southampton | yes | True |
| 3 | 1 | 1 | female | 35.0 | 1 | 0 | 53.1000 | S | First | woman | False | C | Southampton | yes | False |
| 4 | 0 | 3 | male | 35.0 | 0 | 0 | 8.0500 | S | Third | man | True | NaN | Southampton | no | True |
Fun with value_counts() ¶
Here is the simple use of value_counts() we call on the sex column that returns us the count of occurences of each of the unique values in this column.
Now, we want to do the same operation, but this time sort our outputted values in the sex column, male and female, so that values that start with the letter a appear at the top and values that start with letter z appear at the bottom. This is considered ascending order.
f is before m in the alphabet so we see female before male.
In our value_counts method, we’ll set the argument ascending to True .
Often times, we want to know what percentage of the whole is for each value that appears in the column. For example, if we took the two counts above, 577 and 314 and we sum them up, we’d get 891. So, what percentage of people on the titanic were male. The calculation is 577/891 x 100 = 64.75%.
To calculate this in pandas with the value_counts() method, set the argument normalize to True .
Before we try a new value_counts() argument, let’s take a look at some basic descriptive statistics of the fare column. To accomplish this, we’ll call the describe() method on the column.
There’s 891 values of fare data, a mean of 32 and a standard deviation of 49 which indicates a fairly wide spread of data.
Another interesting feature of the value_counts() method is that it can be used to bin continuous data into discrete intervals. We set the argument bins to an integer representing the number of bins to create.
For each bin, the range of fare amounts in dollar values is the same. One contains fares from 73.19 to 146.38 which is a range of 73.19. Another bin contains fares from 146.38 to 73.19 which is also a range of 73.19. See how the ranges are same! However, inside each range of fare values can contain a different count of the number of tickets bought by passengers of the Titanic.
We can see most people paid under 73.19 for their ticket.
Thank you for reading my content! This project is available on GitHub.
Copyright © Dan Friedman, 2020 .