Pandas apply() Function to Single & Multiple Column(s)
Using pandas.DataFrame.apply() method you can execute a function to a single column, all and list of multiple columns (two or more). In this article, I will cover how to apply() a function on values of a selected single, multiple, all columns. For example, let’s say we have three columns and would like to apply a function on a single column without touching other two columns and return a DataFrame with three columns.
Please enable JavaScript
1. Quick Examples of pandas Apply Function to a Column
If you are in a hurry, below are some of the quick examples of how to apply a function to a single and multiple columns (two or more) in pandas DataFrame.
2. pandas.DataFrame.apply() Function Syntax
If you are a learner let’s see the syntax of apply() method and executing some examples of how to apply it on a single column, multiple, and all columns. Our DataFrame contains column names A , B , and C .
Below is a syntax of pandas.DataFrame.apply()
Let’s create a sample DataFrame to work with some examples.
Yields below output.
3. Pandas Apply Function to Single Column
We will create a function add_3() which adds value 3 column value and use this on apply() function. To apply it to a single column, qualify the column name using df["col_name"] . The below example applies a function to a column B .
Yields below output. This applies the function to every row in DataFrame for a specified column.
4. Pandas Apply Function to All Columns
In some cases we would want to apply a function on all pandas columns, you can do this using apply() function. Here the add_3() function will be applied to all DataFrame columns.
Yields below output.
5. Pandas Apply Function to Multiple List of Columns
Similarly using apply() method, you can apply a function on a selected multiple list of columns. In this case, the function will apply to only selected two columns without touching the rest of the columns.
Yields below output
6. Apply Lambda Function to Each Column
You can also apply a lambda expression using the apply() method, the Below example, adds 10 to all column values.
Yields below output.
7. Apply Lambda Function to Single Column
You can apply the lambda function for a single column in the DataFrame. The following example subtracts every cell value by 2 for column A – df["A"]=df["A"].apply(lambda x:x-2) .
Yields below output.
Similarly, you can also apply the Lambda function to all & multiple columns in pandas, I will leave this to you to explore.
8. Using pandas.DataFrame.transform() to Apply Function Column
Using DataFrame.apply() method & lambda functions the resultant DataFrame can be any number of columns whereas with transform() function the resulting DataFrame must have the same length as the input DataFrame.
Yields below output.
9. Using pandas.DataFrame.map() to Single Column
Here is another alternative using map() method.
Yields below output.
10. DataFrame.assign() to Apply Lambda Function
You can also try assign()
Yields below output.
11. Using Numpy function on single Column
Use df['A']=df['A'].apply(np.square) to select the column from DataFrame as series using the [] operator and apply NumPy.square() method.
Yields below output.
12. Using NumPy.square() Method
You can also do the same without using apply() function and directly using Numpy.
Yields same output as above.
13. Multiple columns Using NumPy.square() and Lambda Function
Apply a lambda function to multiple columns in DataFrame using Dataframe apply(), lambda, and Numpy functions.
pandas.DataFrame.apply#
Objects passed to the function are Series objects whose index is either the DataFrame’s index ( axis=0 ) or the DataFrame’s columns ( axis=1 ). By default ( result_type=None ), the final return type is inferred from the return type of the applied function. Otherwise, it depends on the result_type argument.
Parameters func function
Function to apply to each column or row.
Axis along which the function is applied:
0 or ‘index’: apply function to each column.
1 or ‘columns’: apply function to each row.
Determines if row or column is passed as a Series or ndarray object:
False : passes each row or column as a Series to the function.
True : the passed function will receive ndarray objects instead. If you are just applying a NumPy reduction function this will achieve much better performance.
These only act when axis=1 (columns):
‘expand’ : list-like results will be turned into columns.
‘reduce’ : returns a Series if possible rather than expanding list-like results. This is the opposite of ‘expand’.
‘broadcast’ : results will be broadcast to the original shape of the DataFrame, the original index and columns will be retained.
The default behaviour (None) depends on the return value of the applied function: list-like results will be returned as a Series of those. However if the apply function returns a Series these are expanded to columns.
args tuple
Positional arguments to pass to func in addition to the array/series.
**kwargs
Additional keyword arguments to pass as keywords arguments to func .
Returns Series or DataFrame
Result of applying func along the given axis of the DataFrame.
For elementwise operations.
Only perform aggregating type operations.
Only perform transforming type operations.
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
Using a numpy universal function (in this case the same as np.sqrt(df) ):
Using a reducing function on either axis
Returning a list-like will result in a Series
Passing result_type=’expand’ will expand list-like results to columns of a Dataframe
Returning a Series inside the function is similar to passing result_type=’expand’ . The resulting column names will be the Series index.
Passing result_type=’broadcast’ will ensure the same shape result, whether list-like or scalar is returned by the function, and broadcast it along the axis. The resulting column names will be the originals.
Tutorial: How to Use the Apply Method in Pandas
The apply() method is one of the most common methods of data preprocessing. It simplifies applying a function on each element in a pandas Series and each row or column in a pandas DataFrame. In this tutorial, we’ll learn how to use the apply() method in pandas — you’ll need to know the fundamentals of Python and lambda functions. If you aren’t familiar with these or need to brush up your Python skills, you might like to try our free Python Fundamentals course.
Let’s dive right in.
Applying a Function on a Pandas Series
Series form the basis of pandas. They are essentially one-dimensional arrays with axis labels called indices.
There are different ways of creating a Series object (e.g., we can initialize a Series with lists or dictionaries). Let’s define a Series object with two lists containing student names as indices and their heights in centimeters as data:
The code above returns the content of the students object and its data type.
The data type of the students object is Series, so we can apply any functions on its data using the apply() method. Let’s see how we can convert the heights of the students from centimeters to feet:
The students’ heights are converted to feet with two decimal places. To do so, we first defined a function that does the conversion, then pass the function name without parentheses to the apply() method. The apply() method takes each element in the Series and applies the cm_to_feet() function on it.
Applying a Function on a Pandas DataFrame
In this section, we’re going to learn how to use the apply() method to manipulate columns and rows in a DataFrame.
First, let’s create a dummy DataFrame containing the personal details of a company’s employees using the following snippet:
| EmployeeName | Department | HireDate | Sex | Birthdate | Weight | Height | Kids | |
|---|---|---|---|---|---|---|---|---|
| 0 | Callen Dunkley | Accounting | 2010 | M | 04/09/1982 | 78 | 176 | 2 |
| 1 | Sarah Rayner | Engineering | 2018 | F | 14/04/1981 | 80 | 160 | 1 |
| 2 | Jeanette Sloan | Engineering | 2012 | F | 06/05/1997 | 66 | 169 | 0 |
| 3 | Kaycee Acosta | HR | 2014 | F | 08/01/1986 | 67 | 157 | 1 |
| 4 | Henri Conroy | HR | 2014 | M | 10/10/1988 | 90 | 185 | 1 |
| 5 | Emma Peralta | HR | 2018 | F | 12/11/1992 | 57 | 164 | 0 |
| 6 | Martin Butt | Data Science | 2020 | M | 10/04/1991 | 115 | 195 | 2 |
| 7 | Alex Jensen | Data Science | 2018 | M | 16/07/1995 | 87 | 180 | 0 |
| 8 | Kim Howarth | Accounting | 2020 | M | 08/10/1992 | 95 | 174 | 3 |
| 9 | Jane Burnett | Data Science | 2012 | F | 11/10/1979 | 57 | 165 | 1 |
NOTE
In this section, we’ll work on dummy requests initiated by the company’s HR team. We’ll learn how to use the apply() method by going through different scenarios. We’ll explore a new use case in each scenario and solve it using the apply() method.
Scenario 1
Let’s assume that the HR team wants to send an invitation email that starts with a friendly greeting to all the employees (e.g., Hey, Sarah!). They asked you to create two columns for storing the employees’ first and last names separately, making referring to the employees’ first names easy. To do so, we can use a lambda function that splits a string into a list after breaking it by the specified separator; the default separator character of the split() method is any white space. Let’s look at the code:
| EmployeeName | Department | HireDate | Sex | Birthdate | Weight | Height | Kids | FirstName | LastName | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Callen Dunkley | Accounting | 2010 | M | 04/09/1982 | 78 | 176 | 2 | Callen | Dunkley |
| 1 | Sarah Rayner | Engineering | 2018 | F | 14/04/1981 | 80 | 160 | 1 | Sarah | Rayner |
| 2 | Jeanette Sloan | Engineering | 2012 | F | 06/05/1997 | 66 | 169 | 0 | Jeanette | Sloan |
| 3 | Kaycee Acosta | HR | 2014 | F | 08/01/1986 | 67 | 157 | 1 | Kaycee | Acosta |
| 4 | Henri Conroy | HR | 2014 | M | 10/10/1988 | 90 | 185 | 1 | Henri | Conroy |
| 5 | Emma Peralta | HR | 2018 | F | 12/11/1992 | 57 | 164 | 0 | Emma | Peralta |
| 6 | Martin Butt | Data Science | 2020 | M | 10/04/1991 | 115 | 195 | 2 | Martin | Butt |
| 7 | Alex Jensen | Data Science | 2018 | M | 16/07/1995 | 87 | 180 | 0 | Alex | Jensen |
| 8 | Kim Howarth | Accounting | 2020 | M | 08/10/1992 | 95 | 174 | 3 | Kim | Howarth |
| 9 | Jane Burnett | Data Science | 2012 | F | 11/10/1979 | 57 | 165 | 1 | Jane | Burnett |
In the code above, we applied the lambda function on the EmployeeName column, which is technically a Series object. The lambda function splits the employees’ full names into first and last names. Thus, the code creates two more columns that contain the first and last names of employees.
Scenario 2
Now, let’s assume that the HR team wants to know every employee’s age and the average age of the employees because they want to determine if an employee’s age influences job satisfaction and work engagement.
To get the job done, the first step is to define a function that gets an employee’s date of birth and returns their age:
The calculate_age() function gets a person’s date of birth in a proper format and, after performing a simple calculation on it, returns their age.
The next step is to apply the function on the Birthdate column of the DataFrame using the apply() method, as follows:
| EmployeeName | Department | HireDate | Sex | Birthdate | Weight | Height | Kids | FirstName | LastName | Age | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Callen Dunkley | Accounting | 2010 | M | 04/09/1982 | 78 | 176 | 2 | Callen | Dunkley | 39 |
| 1 | Sarah Rayner | Engineering | 2018 | F | 14/04/1981 | 80 | 160 | 1 | Sarah | Rayner | 40 |
| 2 | Jeanette Sloan | Engineering | 2012 | F | 06/05/1997 | 66 | 169 | 0 | Jeanette | Sloan | 24 |
| 3 | Kaycee Acosta | HR | 2014 | F | 08/01/1986 | 67 | 157 | 1 | Kaycee | Acosta | 36 |
| 4 | Henri Conroy | HR | 2014 | M | 10/10/1988 | 90 | 185 | 1 | Henri | Conroy | 33 |
| 5 | Emma Peralta | HR | 2018 | F | 12/11/1992 | 57 | 164 | 0 | Emma | Peralta | 29 |
| 6 | Martin Butt | Data Science | 2020 | M | 10/04/1991 | 115 | 195 | 2 | Martin | Butt | 30 |
| 7 | Alex Jensen | Data Science | 2018 | M | 16/07/1995 | 87 | 180 | 0 | Alex | Jensen | 26 |
| 8 | Kim Howarth | Accounting | 2020 | M | 08/10/1992 | 95 | 174 | 3 | Kim | Howarth | 29 |
| 9 | Jane Burnett | Data Science | 2012 | F | 11/10/1979 | 57 | 165 | 1 | Jane | Burnett | 42 |
The single-line statement above applies the calculate_age() function on each element of the Birthdate column and stores the returned values in the Age column.
The last step is to calculate the average age of the employees, as follows:
Scenario 3
The HR manager of the company is exploring options for healthcare coverage for all employees. Potential providers require information about the employees. Since the DataFrame contains the weight and height of each employee, let’s assume the HR manager asked you to provide a Body Mass Index (BMI) for every employee so she can get quotes from potential healthcare providers.
To do the task, first, we need to define a function that calculates the Body Mass Index (BMI). The formula for the BMI is weight in kilograms divided by height in meters squared. Because the employees’ heights are measured in centimeters, we need to divide the heights by 100 to obtain the heights in meters. Let’s implement the function:
The next step is to apply the function on the DataFrame:
The lambda function takes each row’s weight and height values, then applies the calc_bmi() function on them to calculate their BMIs. The axis=1 argument means to iterate over rows in the DataFrame.
| EmployeeName | Department | HireDate | Sex | Birthdate | Weight | Height | Kids | FirstName | LastName | Age | BMI | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Callen Dunkley | Accounting | 2010 | M | 04/09/1982 | 78 | 176 | 2 | Callen | Dunkley | 39 | 25.18 |
| 1 | Sarah Rayner | Engineering | 2018 | F | 14/04/1981 | 80 | 160 | 1 | Sarah | Rayner | 40 | 31.25 |
| 2 | Jeanette Sloan | Engineering | 2012 | F | 06/05/1997 | 66 | 169 | 0 | Jeanette | Sloan | 24 | 23.11 |
| 3 | Kaycee Acosta | HR | 2014 | F | 08/01/1986 | 67 | 157 | 1 | Kaycee | Acosta | 36 | 27.18 |
| 4 | Henri Conroy | HR | 2014 | M | 10/10/1988 | 90 | 185 | 1 | Henri | Conroy | 33 | 26.30 |
| 5 | Emma Peralta | HR | 2018 | F | 12/11/1992 | 57 | 164 | 0 | Emma | Peralta | 29 | 21.19 |
| 6 | Martin Butt | Data Science | 2020 | M | 10/04/1991 | 115 | 195 | 2 | Martin | Butt | 30 | 30.24 |
| 7 | Alex Jensen | Data Science | 2018 | M | 16/07/1995 | 87 | 180 | 0 | Alex | Jensen | 26 | 26.85 |
| 8 | Kim Howarth | Accounting | 2020 | M | 08/10/1992 | 95 | 174 | 3 | Kim | Howarth | 29 | 31.38 |
| 9 | Jane Burnett | Data Science | 2012 | F | 11/10/1979 | 57 | 165 | 1 | Jane | Burnett | 42 | 20.94 |
The last step is to categorize the employees according to the BMI measurement. A BMI of less than 18.5 is Group One, between 18.5 and 24.9 is Group Two, between 25 and 29.9 is Group Three, and over 30 is Group Four. To implement the solution, we will define a function that returns the various BMI indicators, then apply it on the BMI column of the DataFrame to see each employee falls into which category:
| EmployeeName | Department | HireDate | Sex | DoB | Weight | Height | Kids | FirstName | LastName | Age | BMI | BMI_Indicator | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Callen Dunkley | Accounting | 2010 | M | 04/09/1982 | 78 | 176 | 2 | Callen | Dunkley | 39 | 25.18 | Group Three |
| 1 | Sarah Rayner | Engineering | 2018 | F | 14/04/1981 | 80 | 160 | 1 | Sarah | Rayner | 40 | 31.25 | Group Four |
| 2 | Jeanette Sloan | Engineering | 2012 | F | 06/05/1997 | 66 | 169 | 0 | Jeanette | Sloan | 24 | 23.11 | Group Two |
| 3 | Kaycee Acosta | HR | 2014 | F | 08/01/1986 | 67 | 157 | 1 | Kaycee | Acosta | 36 | 27.18 | Group Three |
| 4 | Henri Conroy | HR | 2014 | M | 10/10/1988 | 90 | 185 | 1 | Henri | Conroy | 33 | 26.30 | Group Three |
| 5 | Emma Peralta | HR | 2018 | F | 12/11/1992 | 57 | 164 | 0 | Emma | Peralta | 29 | 21.19 | Group Two |
| 6 | Martin Butt | Data Science | 2020 | M | 10/04/1991 | 115 | 195 | 2 | Martin | Butt | 30 | 30.24 | Group Four |
| 7 | Alex Jensen | Data Science | 2018 | M | 16/07/1995 | 87 | 180 | 0 | Alex | Jensen | 26 | 26.85 | Group Three |
| 8 | Kim Howarth | Accounting | 2020 | M | 08/10/1992 | 95 | 174 | 3 | Kim | Howarth | 29 | 31.38 | Group Four |
| 9 | Jane Burnett | Data Science | 2012 | F | 11/10/1979 | 57 | 165 | 1 | Jane | Burnett | 42 | 20.94 | Group Two |
Scenario 4
Let’s assume the new year is around the corner and the company management has announced that those employees who have more than ten years of experience will get an extra bonus. The HR manager wants to know who is qualified to get the bonus.
To prepare the requested information, you need to apply the following lambda function on the HireDate column, which returns True if the difference between the current year and the hire year is greater than or equal to ten years otherwise False .
Running the code above creates a pandas Series that contains True or False values, called a Boolean mask.
To display the qualified employees, we use the Boolean mask to filter the DataFrame rows. Let’s run the following statement and see the result:
| EmployeeName | Department | HireDate | Sex | DoB | Weight | Height | Kids | FirstName | LastName | Age | BMI | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Callen Dunkley | Accounting | 2010 | M | 04/09/1982 | 78 | 176 | 2 | Callen | Dunkley | 39 | 25.18 |
| 2 | Jeanette Sloan | Engineering | 2012 | F | 06/05/1997 | 66 | 169 | 0 | Jeanette | Sloan | 24 | 23.11 |
| 9 | Jane Burnett | Data Science | 2012 | F | 11/10/1979 | 57 | 165 | 1 | Jane | Burnett | 42 | 20.94 |
Scenario 5
Let’s assume that tomorrow is Mother’s Day, and the company has planned a Mother’s Day gift for all its female employees who have children. The HR team asked you to prepare a list of the employees who are eligible for the gift. To do the task, we need to write a simple lambda function that considers the Sex and Kids columns to provide the desired result, as follows:
| EmployeeName | Department | HireDate | Sex | Birthdate | Weight | Height | Kids | FirstName | LastName | Age | BMI | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Sarah Rayner | Engineering | 2018 | F | 14/04/1981 | 80 | 160 | 1 | Sarah | Rayner | 40 | 31.25 |
| 3 | Kaycee Acosta | HR | 2014 | F | 08/01/1986 | 67 | 157 | 1 | Kaycee | Acosta | 36 | 27.18 |
| 9 | Jane Burnett | Data Science | 2012 | F | 11/10/1979 | 57 | 165 | 1 | Jane | Burnett | 42 | 20.94 |
Running the code above returns the list of employees who will receive the gifts.
The lambda function returns True if a female employee has at least one child; otherwise, it returns False . The result of applying the lambda function on the DataFrame is a Boolean mask that we directly used to filter the DataFrame’s rows.
Conclusion
In this tutorial, we learned what the apply() method does and how to use it by going through different examples. The apply() method is a powerful and efficient way to apply a function on every value of a Series or DataFrame in pandas. Since the apply() method uses C extensions for Python, it performs faster when iterating through all the rows of a pandas DataFrame. However, it isn’t a general rule as it’s slower when performing the same operation through a column.
About the author
Mehdi Lotfinejad
Mehdi is a Senior Data Engineer and Team Lead at ADA. He is a professional trainer who loves writing data analytics tutorials.
Применение функции по строкам или столбцам
Произвольные функции могут применяться вдоль осей DataFrame с помощью метода apply() , который, как и методы описательной статистики, принимает необязательный аргумент axis :
Метод apply() также отправляет имя строкового метода.
Тип возвращаемого значения функции, переданной в apply() влияет на тип окончательного вывода из DataFrame.apply для поведения по умолчанию:
Если примененная функция возвращает Series , конечным результатом будет DataFrame . Столбцы соответствуют индексу Series возвращаемой примененной функцией.
Если примененная функция возвращает любой другой тип, конечным результатом будет Series .
Это поведение по умолчанию можно переопределить с помощью result_type , который принимает три параметра: reduce , broadcast и expand . Они будут определять, как возвращаемые значения списков будут расширяться (или нет) в DataFrame .
apply() сочетании с некоторой смекалкой можно использовать, чтобы ответить на многие вопросы о наборе данных. Например, предположим, что мы хотим извлечь дату, когда произошло максимальное значение для каждого столбца:
Вы также можете передать дополнительные аргументы и аргументы ключевого слова методу apply() . Например, рассмотрим следующую функцию, которую вы хотели бы применить:
Затем вы можете применить эту функцию следующим образом:
Еще одной полезной функцией является возможность передачи методов Series для выполнения некоторой операции Series над каждым столбцом или строкой:
Наконец, apply() принимает raw аргумент, который по умолчанию равен False, который преобразует каждую строку или столбец в серию перед применением функции. Если установлено значение True, переданная функция вместо этого получит объект ndarray, что положительно скажется на производительности, если вам не нужны функции индексирования.
Aggregation API
API агрегации позволяет выразить, возможно, несколько операций агрегации в одном кратком виде. Этот API похож на все объекты pandas, см. Groupby API , Window API и resample API . Точкой входа для агрегации является DataFrame.aggregate() или псевдоним DataFrame.agg() .
Мы будем использовать аналогичную стартовую рамку,описанную выше:
Использование одной функции эквивалентно apply() . Вы также можете передавать именованные методы в виде строк. Они вернут Series агрегированного вывода:
Одиночные агрегаты в Series это вернет скалярное значение:
Агрегирование с использованием нескольких функций
Вы можете передать несколько аргументов агрегации в виде списка. Результаты каждой из переданных функций будут строкой в DataFrame . Они естественно названы из функции агрегирования.
Несколько функций дают несколько рядов:
В Series несколько функций возвращают Series , индексированную именами функций:
Передача lambda функции даст строку с именем <lambda> :
Передавая именованную функцию,вы получите это имя для строки:
Агрегирование с помощью диктанта
Передача словаря имен столбцов в скаляр или список скаляров в DataFrame.agg позволяет вам настроить, какие функции применяются к каким столбцам. Обратите внимание, что результаты не находятся в каком-либо определенном порядке, вместо этого вы можете использовать OrderedDict , чтобы гарантировать порядок.
Передача списка, подобного списку, сгенерирует вывод DataFrame . Вы получите матричный вывод всех агрегаторов. Вывод будет состоять из всех уникальных функций. Те, которые не указаны для определенного столбца, будут NaN :
Mixed dtypes
Устарело, начиная с версии 1.4.0: Попытка определить, какие столбцы нельзя агрегировать, и молча удалить их из результатов, устарела и будет удалена в будущей версии. Если какая-либо часть предоставленных столбцов или операций завершится неудачно, будет вызван вызов .agg .
При представлении смешанных типов данных, которые не могут агрегироваться, .agg будет принимать только действительные агрегаты. Это похоже на то, как работает .groupby.agg .
Custom describe
С помощью .agg() можно легко создать настраиваемую функцию описания, аналогичную встроенной функции описания .
Transform API
Метод transform() возвращает объект, имеющий тот же индекс (такой же размер), что и оригинал. Этот API позволяет выполнять несколько операций одновременно, а не одну за другой. Его API очень похож на API .agg .
Мы создаем рамку,аналогичную той,что использовалась в предыдущих разделах.
Преобразуйте весь кадр. .transform() позволяет вводить функции как: функцию NumPy, имя строковой функции или пользовательскую функцию.
Здесь transform() получил единственную функцию; это эквивалентно приложению ufunc .
Передача одной функции в .transform() с помощью Series даст взамен одну Series .
Трансформация с несколькими функциями
При передаче нескольких функций будет получен столбец MultiIndexed DataFrame.На первом уровне будут имена столбцов исходного фрейма;на втором уровне-имена преобразующих функций.
Передача нескольких функций в Series приведет к созданию DataFrame.Имена результирующих столбцов будут именами преобразующих функций.
Преобразование с помощью диктанта
Передача диктанта функций позволит выполнять выборочное преобразование для каждого столбца.
Передача диктанта списков создаст MultiIndexed DataFrame с этими выборочными преобразованиями.