Calling Python from C#: an introduction to PythonNET
Recently, I had to dig into PythonNET since I had to find a way to call Python scripts from my C# projects at work. Surprisingly enough, I found very little material online on the subject and most of it was contradictory or rather confusing. PythonNET’s official documentation wasn’t particularly helpful either, so I decided to write this guide for those who will find themselves in the same situation as me in the future.
Compile PythonNET’s dll
First thing we’re going to do is compiling PythonNET’s dll. Download PythonNET’s source code from Github, open it with Visual Studio and compile the project called “Python.Runtime”. If the compilation is successful, it will generate a .dll file in pythonnet-master\pythonnet\runtime\Python.Runtime.dll
Python prerequisites
At the moment when this guide was written (September 2021), PythonNET supports only Python 3.6–3.8. Python’s last available release is the 3.9.7. Check if you have a newer version of Python installed on your machine and, if that’s the case, be absolutely sure to uninstall it to avoid any problems. You can then download Python 3.8 from here.
If you have a version like “3.8.10” or something like that it should be fine, PythonNET doesn’t seem to mind until you reach 3.9 and up from my personal experience.
You will also need to install PythonNET’s module by launching pip install pythonnet from the CMD.
Create a test application
Let’s create a console application called “TestProject” and add a reference to PythonNET’s dll (Python.Runtime.dll) in it.
Hello world
Let’s create a class named “PythonInterop”. We are going to create an Initialize method, where we’re going to set Python’s dll as an environment variable (be sure to replace my path with yours) and Initialize PythonNET’s engine. You can find Python’s dll in Python’s installation path. By default, it should be C:\Users\YourUserNameHere\AppData\Local\Programs\Python\Python38\python38.dll. for Python 3.8.0.
NOTE: If you’re planning on running PythonNET on Linux, you want to look for Python’s .so file instead! It can be quite hard to find since it isn’t always found in the same directory. You can try to run this script to get its location:
wget https://gist.githubusercontent.com/tkf/d980eee120611604c0b9b5fef5b8dae6/raw/9f074cd233f83180676b4421212ed33c257968af/find_libpython.py
/usr/bin/python3 find_libpython.py —list-all
We are also going to create a method to run our Python code with PythonNET.
Now let’s go in Program.cs and launch our hello world.
Use C# classes in Python
To import C# classes, variables and methods into Python we need to import a module called clr.
Let’s create a class named “MyClass” with a string property named “MyVar”.
In our Program.cs file, let’s run
If everything went well, we should see our variable printed in the Console.
In this way we can import any class, including C#’s System classes, and use their methods too.
Use C# types from Python
In the same way, PythonNET lets us se C#’s types from Python aswell.
Import variables with PythonNET’s methods
An alternative way to import C# variables into Python is using PythonNET’s methods.
Let’s create an overload of the RunPythonCode method in our PythonInterop class
parameter is the variable that we want to inject into Python, parameterName is the name that we want the variable to have inside of Python (usually, the same as C#).
Let’s create a Person class and test our method.
Return values from Python to C#
Let’s suppose now that we want to return a variable from Python into C# to perform some other operations on it.
To do so, let’s go back into our PythonInterop class and create a method.
returnedVariableName is the name of the Python variable that we want to get returned in C#.
Let’s try to edit the Age property’s value, to assign it to a variable and to return it to C#.
Import from external projects
You can also easily import from external libraries using the clr module. Let’s create a class library called “MyLibrary” in our solution and let’s reference it from our main Console Application method.
After that, let’s add a class called “LibClass” in our library.
Now, let’s just try to print our string from Python.
Final notes
This was meant as a short introduction to PythonNET for absolute beginners, since it was so difficult for me to find material like this online.
From now on you will probably have a decent toolset that you can use to research and pursue your goals with PythonNET on your own. Good luck!
I’m going to leave the full source of the PythonInterop class below.
Объединяя C++ и Python. Тонкости Boost.Python. Часть первая
Исходим из того, что у вас уже установлен удобный инструментарий для сборки динамически-линкуемой библиотеки на C++, а также установлен интерпретатор Python.
Также понадобится скачать библиотеку Boost, после чего собрать её, следуя инструкции для своей ОС Windows или Linux.
В двух словах в Windows все действия сводятся к двум строкам в командной строке. Распакуйте скачанный архив Boost в любое место на диске, перейдите туда в командной строке и наберите последовательно две команды:
Для сборки x64 нужно добавить аргумент address-model=64
Если у вас уже есть библиотека Boost, но вы не устанавливали Python, либо вы скачали и установили свежий интерпретатор Python и хотите собрать только Boost.Python, это делается дополнительным ключом —with-python
То есть вся строка для сборки только Boost.Python с 64-разрядной адресацией выглядит так:
Стоит заметить, что x64 сборку следует заказывать, если у вас установлен Python x64. Также и модули для него нужно будет собирать с 64-разрядной адресацией.
Ключ —with-python серьёзно сэкономит вам время, если вам из библиотеки Boost кроме функционала Boost.Python ничего не нужно.
Если у вас установлено несколько интерпретаторов, крайне рекомендую прочитать подробную документацию по сборке Boost.Python
После сборки у вас появятся в папке Boost\stage\lib собранные библиотеки Boost.Python, они нам очень скоро понадобятся.
Настраиваем проект на C++
Создаём проект для создания динамически-линкуемой библиотеки на C++, предлагаю назвать его example.
После создания проекта, требуется указать дополнительные INCLUDE каталоги Python\include и корень Boost, а также каталоги для поиска библиотек Python\libs и Boost\stage\lib
Под Windows также следует в настройках Post-build events задать переименование $(TargetPath) в модуль с расширением example.pyd в корне проекта.
Также возможно стоит скопировать собранные библиотеки Boost.Python в каталог с собираемым модулем.
Подключение модуля после запуска интерпретатора в том же каталоге сведётся к одной команде:
Не забываем также про сборку под x64 если вы собираете для 64-разрядного Python.
Обычный класс с простыми полями
Итак, давайте заведём нашем новом проекте сразу три файла:
some.h
some.cpp
wrap.cpp
В файлах some.h и some.cpp опишем некий замечательный класс Some, который обернём для Python в модуле example в файле wrap.cpp — для этого в файле wrap.cpp следует подключить <boost/python.hpp> и использовать макрос BOOST_PYTHON_MODULE( example ) <… >, также для лаконичности будет совсем не лишним использовать using namespace boost::python. В целом наш будущий модуль будет выглядеть вот так:
В файле some.h нам следует наваять объявление нашего чудо-класса. Для объяснения большинства базовых механизмов нам достаточно всего два поля:
Допустим класс содержит описание чего-то, что имеет имя и целочисленный идентификатор. Как ни странно этот несложный класс вызовет кучу сложностей, благодаря в основном стандартному классу string, перегрузкам методов, константной ссылке и статическому свойству NOT_AN_IDENTIFIER, которое мы конечно же тоже введём:
Разумеется эта константа нужна как идентификатор для объекта созданного конструктором по умолчанию, опишем также и другой конструктор, задающий оба поля:
В файле some.cpp опишем реализацию данных конструкторов, в дальнейшем реализацию описывать я не буду, но давайте конструкторы напишем вместе:
Одновременно с появлением класса Some будет появляться обёртка класса для Python в файле wrap.cpp:
Здесь используется бессовестный обман зрения и шаблон boost::python::class_, который создаёт описание класса для Python в указанном модуле с помощью Python C-API, жутко сложного и непонятно при описании методов, а потому полностью скрытого за объявлением простого метода def() на каждой строчке.
Конструктор по умолчанию и конструктор копирования создаются для объекта по умолчанию, если не указано обратное, но мы этого ещё коснёмся чуть ниже.
Уже сейчас можно собрать модуль, импортировать его из интерпретатора Python и даже создать экземпляр класса, но ни прочитать его свойства, ни вызывать методы мы у него пока не можем, пока они физически отсутствуют.
Давайте это исправим, создадим «богатейшее» API нашего чудо класса. Вот полный код нашего заголовочного файла some.h:
Раз реализация методов получилась также довольно короткой, давайте приведу и код some.cpp:
Что ж, самое время описать обёртку в файле wrap.cpp:
Первый метод Some::ID() оборачивается без каких-либо проблем:
Зато второй с результатом в виде константной ссылки на строку уже показывает, что всё не так просто:
- copy_non_const_reference — создаёт новый объект в Python, который содержит неконстантную ссылку на объект в C++, не требует обёртки для класса из C++ (пример: string не имеет обёртки, только конвертер в питоновский str)
- copy_const_reference — создаёт новый объект в Python, который содержит константную ссылку на объект в C++, не требует обёртки для класса из C++ (пример: тот же string)
- manage_new_object — создаёт новый объект в Python, используя обёртку класса из C++, по завершении содержимое удаляется
- reference_existing_object — создаёт новый объект в Python, используя обёртку класса из C++, по завершении содержимое остаётся
Метод Some::ResetID я намерено перегрузил, чтобы усложнить задачу с передачей указателя на метод в .def():
Как видите, можно указать, с каким именем в Python будет создан аргумент метода. Как известно имя аргумента в Python куда важнее чем в C++. Рекомендую указывать имена аргументов для каждой обёртки метода, принимающего параметры:
Осталось описать статическим свойством константу NOT_AN_IDENTIFIER:
Здесь используется специальная функция boost::python::make_getter, которая по свойству класса генерирует get-функцию.
Вот так примерно выглядит наша обёртка:
Если написать несложный тестовый скрипт вроде этого (Python 3.x):
Мы увидим вывод:
Питонизируем обёртку класса
Итак, класс со всеми методами обёрнут, но счастья не наступило. При попытке из командной строки Python выполнив Some(123,’asd’) мы не увидим описания полей и вообще объекта, поскольку мы не обзавелись методом __repr__, так же как и преобразование к строке, тот же print( Some(123,’asd’) ) будет ужасно неинформативен, так как мы не обзавелись методом __str__. Очевидно также, что работать со свойствами через методы на C++ на Python не имеет смысла, это в C++ мы не имеем возможности заводить property, в Python их можно и нужно завести. Однако как же мы навесим методы на готовый класс C++ предназначенные для Python?
Очень просто: вспоминаем, что в Python методы не отличаются от функций, принимающих первым параметром ссылку на self — экземпляр класса. Заводим в C++ такие функции прямо во wrap.cpp и описываем их как методы в обёртке:
Сами функции можно описать например вот так:
Со свойствами идентификатора и имени ещё проще, так как методы set и get для них уже описаны в классе:
При описании свойств однако было два тонких момента:
1. Для set-метода свойства some_id было явное приведение к типу метода, принимающего int, т.к. есть ещё одна перегрузка метода.
2. Для get-метода свойства name была использована конструкция boost::python::make_function, которая позволила повесить return_value_policy на результат метода возвращающего константную ссылку на string.
Выполняем print( Some(123,’asd’) ) и просто Some(123,’asd’) из командной строки после from example import * и видим что подозрительно похожее на встроенный питоновский dict: < ID: 123, Name: 'asd' >
Почему бы не завести свойство инициализирующее экземпляр Some от стандартного dict и обратно?
Заведём ещё пару питонистических функций и заведём свойство as_dict:
Здесь использован класс boost::python::dict, для доступа на уровне C++ к стандартному dict Python.
Также есть классы для доступа к str, list, tuple, называются они соответственно. Ведут себя классы в C++ так же как и в Python в плане операторов, вот только возвращают по большей части boost::python::object, из которого требуется ещё извлечь значение через функцию boost::python::extract.
Python Bindings: Calling C or C++ From Python
Are you a Python developer with a C or C++ library you’d like to use from Python? If so, then Python bindings allow you to call functions and pass data from Python to C or C++, letting you take advantage of the strengths of both languages. Throughout this tutorial, you’ll see an overview of some of the tools you can use to create Python bindings.
In this tutorial, you’ll learn about:
- Why you want to call C or C++ from Python
- How to pass data between C and Python
- What tools and methods can help you create Python bindings
This tutorial is aimed at intermediate Python developers. It assumes basic knowledge of Python and some understanding of functions and data types in C or C++. You can get all of the example code you’ll see in this tutorial by clicking on the link below:
Get Sample Code: Click here to get the sample code you’ll use to learn about Python Bindings in this tutorial.
Let’s dive into looking at Python bindings!
Python Bindings Overview
Before you dive into how to call C from Python, it’s good to spend some time on why. There are several situations where creating Python bindings to call a C library is a great idea:
You already have a large, tested, stable library written in C++ that you’d like to take advantage of in Python. This may be a communication library or a library to talk to a specific piece of hardware. What it does is unimportant.
You want to speed up a particular section of your Python code by converting a critical section to C. Not only does C have faster execution speed, but it also allows you to break free from the limitations of the GIL, provided you’re careful.
You want to use Python test tools to do large-scale testing of their systems.
All of the above are great reasons to learn to create Python bindings to interface with your C library.
Note: Throughout this tutorial, you’ll be creating Python bindings to both C and C++. Most of the general concepts apply to both languages, and so C will be used unless there’s a specific difference between the two languages. In general, each of the tools will support either C or C++, but not both.
Let’s get started!
Marshalling Data Types
Wait! Before you start writing Python bindings, take a look at how Python and C store data and what types of issues this will cause. First, let’s define marshalling. This concept is defined by Wikipedia as follows:
The process of transforming the memory representation of an object to a data format suitable for storage or transmission. (Source)
For your purposes, marshalling is what the Python bindings are doing when they prepare data to move it from Python to C or vice versa. Python bindings need to do marshalling because Python and C store data in different ways. C stores data in the most compact form in memory possible. If you use an uint8_t , then it will only use 8 bits of memory total.
In Python, on the other hand, everything is an object. This means that each integer uses several bytes in memory. How many will depend on which version of Python you’re running, your operating system, and other factors. This means that your Python bindings will need to convert a C integer to a Python integer for each integer passed across the boundary.
Other data types have similar relationships between the two languages. Let’s look at each in turn:
Integers store counting numbers. Python stores integers with arbitrary precision, meaning that you can store very, very, large numbers. C specifies the exact sizes of integers. You need to be aware of data sizes when you’re moving between languages to prevent Python integer values from overflowing C integer variables.
Floating-point numbers are numbers with a decimal place. Python can store much larger (and much smaller) floating-point numbers than C. This means that you’ll also have to pay attention to those values to ensure they stay in range.
Complex numbers are numbers with an imaginary part. While Python has built-in complex numbers, and C has complex numbers, there’s no built-in method for marshalling between them. To marshal complex numbers, you’ll need to build a struct or class in the C code to manage them.
Strings are sequences of characters. For being such a common data type, strings will prove to be rather tricky when you’re creating Python bindings. As with other data types, Python and C store strings in quite different formats. (Unlike the other data types, this is an area where C and C++ differ as well, which adds to the fun!) Each of the solutions you’ll examine have slightly different methods for dealing with strings.
Boolean variables can have only two values. Since they’re supported in C, marshalling them will prove to be fairly straightforward.
Besides data type conversions, there are other issues you’ll need to think about as you build your Python bindings. Let’s keep exploring them.
Understanding Mutable and Immutable Values
In addition to all of these data types, you’ll also have to be aware of how Python objects can be mutable or immutable. C has a similar concept with function parameters when talking about pass-by-value or pass-by-reference. In C, all parameters are pass-by-value. If you want to allow a function to change a variable in the caller, then you need to pass a pointer to that variable.
You might be wondering if you can get around the immutable restriction by simply passing an immutable object to C using a pointer. Unless you go to ugly and non-portable extremes, Python won’t give you a pointer to an object, so this just doesn’t work. If you want to modify a Python object in C, then you’ll need to take extra steps to achieve this. These steps will be dependent on which tools you use, as you’ll see below.
So, you can add immutability to your checklist of items to consider as you create Python bindings. Your final stop on the grand tour of creating this checklist is how to handle the different ways in which Python and C deal with memory management.
Managing Memory
C and Python manage memory differently. In C, the developer must manage all memory allocations and ensure they’re freed once and only once. Python takes care of this for you using a garbage collector.
While each of these approaches has its advantages, it does add an extra wrinkle into creating Python bindings. You’ll need to be aware of where the memory for each object was allocated and ensure that it’s only freed on the same side of the language barrier.
For example, a Python object is created when you set x = 3 . The memory for this is allocated on the Python side and needs to be garbage collected. Fortunately, with Python objects, it’s quite difficult to do anything else. Take a look at the converse in C, where you directly allocate a block of memory:
When you do this, you need to ensure that this pointer is freed in C. This may mean manually adding code to your Python bindings to do this.
That rounds out your checklist of general topics. Let’s start setting up your system so you can write some code!
Setting Up Your Environment
For this tutorial, you’re going to be using pre-existing C and C++ libraries from the Real Python GitHub repo to show a test of each tool. The intent is that you’ll be able to use these ideas for any C library. To follow along with all of the examples here, you’ll need to have the following:
- A C++ library installed and knowledge of the path for command-line invocation
- Python development tools:
- For Linux, this is the python3-dev or python3-devel package, depending on your distro.
- For Windows, there are multiple options.
The last one might be new to you, so let’s take a closer look at it.
Using the invoke Tool
invoke is the tool you’ll be using to build and test your Python bindings in this tutorial. It has a similar purpose to make but uses Python instead of Makefiles. You’ll need to install invoke in your virtual environment using pip :
To run it, you type invoke followed by the task you wish to execute:
To see which tasks are available, you use the —list option:
Note that when you look in the tasks.py file where the invoke tasks are defined, you’ll see that the name of the second task listed is build_cffi . However, the output from —list shows it as build-cffi . The minus sign ( — ) can’t be used as part of a Python name, so the file uses an underscore ( _ ) instead.
For each of the tools you’ll examine, there will be a build- and a test- task defined. For example, to run the code for CFFI , you could type invoke build-cffi test-cffi . An exception is ctypes , as there’s no build phase for ctypes . In addition, there are two special tasks added for convenience:
- invoke all runs the build and test tasks for all tools.
- invoke clean removes any generated files.
Now that you’ve got a feeling for how to run the code, let’s take a peek at the C code you’ll be wrapping before hitting the tools overview.
C or C++ Source
In each of the example sections below, you’ll be creating Python bindings for the same function in either C or C++. These sections are intended to give you a taste of what each method looks like, rather than an in-depth tutorial on that tool, so the function you’ll wrap is small. The function you’ll create Python bindings for takes an int and a float as input parameters and returns a float that’s the product of the two numbers:
The C and C++ functions are almost identical, with minor name and string differences between them. You can get a copy of all of the code by clicking on the link below:
Get Sample Code: Click here to get the sample code you’ll use to learn about Python Bindings in this tutorial.
Now you’ve got the repo cloned and your tools installed, you can build and test the tools. So let’s dive into each section below!
ctypes
You’ll start with ctypes , which is a tool in the standard library for creating Python bindings. It provides a low-level toolset for loading shared libraries and marshalling data between Python and C.
How It’s Installed
One of the big advantages of ctypes is that it’s part of the Python standard library. It was added in Python version 2.5, so it’s quite likely you already have it. You can import it just like you do with the sys or time modules.
Calling the Function
All of the code to load your C library and call the function will be in your Python program. This is great since there are no extra steps in your process. You just run your program, and everything is taken care of. To create your Python bindings in ctypes , you need to do these steps:
- Load your library.
- Wrap some of your input parameters.
- Tell ctypes the return type of your function.
You’ll look at each of these in turn.
Library Loading
ctypes provides several ways for you to load a shared library, some of which are platform-specific. For your example, you’ll create a ctypes.CDLL object directly by passing in the full path to the shared library you want:
This will work for cases when the shared library is in the same directory as your Python script, but be careful when you attempt to load libraries that are from packages other than your Python bindings. There are many details for loading libraries and finding paths in the ctypes documentation that are platform and situation-specific.
NOTE: Many platform-specific issues can arise during library loading. It’s best to make incremental changes once you get an example working.
Now that you have the library loaded into Python, you can try calling it!
Calling Your Function
Remember that the function prototype for your C function is as follows:
You need to pass in an integer and a float and can expect to get a float returned. Integers and floats have native support in both Python and in C, so you expect this case will work for reasonable values.
Once you’ve loaded the library into your Python bindings, the function will be an attribute of c_lib , which is the CDLL object you created earlier. You can try to call it just like this:
Oops! This doesn’t work. This line is commented out in the example repo because it fails. If you attempt to run with that call, then Python will complain with an error:
It looks like you need to tell ctypes about any parameters that aren’t integers. ctypes doesn’t have any knowledge of the function unless you tell it explicitly. Any parameter that’s not marked otherwise is assumed to be an integer. ctypes doesn’t know how to convert the value 2.3 that’s stored in y to an integer, so it fails.
To fix this, you’ll need to create a c_float from the number. You can do that in the line where you’re calling the function:
Now, when you run this code, it returns the product of the two numbers you passed in:
Wait a minute… 6 multiplied by 2.3 is not 48.0 !
It turns out that, much like the input parameters, ctypes assumes your function returns an int . In actuality, your function returns a float , which is getting marshalled incorrectly. Just like the input parameter, you need to tell ctypes to use a different type. The syntax here is slightly different:
That should do the trick. Let’s run the entire test-ctypes target and see what you’ve got. Remember, the first section of output is before you fixed the restype of the function to be a float:
That’s better! While the first, uncorrected version is returning the wrong value, your fixed version agrees with the C function. Both C and Python get the same result! Now that it’s working, take a look at why you may or may not want to use ctypes .
Strengths and Weaknesses
The biggest advantage ctypes has over the other tools you’ll examine here is that it’s built into the standard library. It also requires no extra steps, as all of the work is done as part of your Python program.
In addition, the concepts used are low-level, which makes exercises like the one you just did manageable. However, more complex tasks grow cumbersome with the lack of automation. In the next section, you’ll see a tool that adds some automation to the process.
CFFI is the C Foreign Function Interface for Python. It takes a more automated approach to generate Python bindings. CFFI has multiple ways in which you can build and use your Python bindings. There are two different options to select from, which gives you four possible modes:
ABI vs API: API mode uses a C compiler to generate a full Python module, whereas ABI mode loads the shared library and interacts with it directly. Without running the compiler, getting the structures and parameters correct is error-prone. The documentation heavily recommends using the API mode.
in-line vs out-of-line: The difference between these two modes is a trade-off between speed and convenience:
- In-line mode compiles the Python bindings every time your script runs. This is convenient, as you don’t need an extra build step. It does, however, slow down your program.
- Out-of-line mode requires an extra step to generate the Python bindings a single time and then uses them each time the program is run. This is much faster, but that may not matter for your application.
For this example, you’ll use the API out-of-line mode, which produces the fastest code and, in general, looks similar to other Python bindings you’ll create later in this tutorial.
How It’s Installed
Since CFFI is not a part of the standard library, you’ll need to install it on your machine. It’s recommended that you create a virtual environment for this. Fortunately, CFFI installs with pip :
This will install the package into your virtual environment. If you’ve already installed from the requirements.txt , then this should be taken care of. You can take a look at requirements.txt by accessing the repo at the link below:
Get Sample Code: Click here to get the sample code you’ll use to learn about Python Bindings in this tutorial.
Now that you have CFFI installed, it’s time to take it for a spin!
Calling the Function
Unlike ctypes , with CFFI you’re creating a full Python module. You’ll be able to import the module like any other module in the standard library. There is some extra work you’ll have to do to build your Python module. To use your CFFI Python bindings, you’ll need to take the following steps:
- Write some Python code describing the bindings.
- Run that code to generate a loadable module.
- Modify the calling code to import and use your newly created module.
That might seem like a lot of work, but you’ll walk through each of these steps and see how it works.
Write the Bindings
CFFI provides methods to read a C header file to do most of the work when generating Python bindings. In the documentation for CFFI , the code to do this is placed in a separate Python file. For this example, you’ll place that code directly into the build tool invoke , which uses Python files as input. To use CFFI , you start by creating a cffi.FFI object, which provides the three methods you need:
Once you have the FFI, you’ll use .cdef() to process the contents of the header file automatically. This creates wrapper functions for you to marshal data from Python:
Reading and processing the header file is the first step. After that, you need to use .set_source() to describe the source file that CFFI will generate:
Here’s a breakdown of the parameters you’re passing in:
«cffi_example» is the base name for the source file that will be created on your file system. CFFI will generate a .c file, compile it to a .o file, and link it to a .<system-description>.so or .<system-description>.dll file.
‘#include «cmult.h»‘ is the custom C source code that will be included in the generated source before it’s compiled. Here, you just include the .h file for which you’re generating bindings, but this can be used for some interesting customizations.
libraries=[«cmult»] tells the linker the name of your pre-existing C library. This is a list, so you can specify several libraries if required.
library_dirs=[this_dir.as_posix(),] is a list of directories that tells the linker where to look for the above list of libraries.
extra_link_args=[‘-Wl,-rpath,.’] is a set of options that generate a shared object, which will look in the current path ( . ) for other libraries it needs to load.
Build the Python Bindings
Calling .set_source() doesn’t build the Python bindings. It only sets up the metadata to describe what will be generated. To build the Python bindings, you need to call .compile() :
This wraps things up by generating the .c file, .o file, and the shared library. The invoke task you just walked through can be run on the command line to build the Python bindings:
You have your CFFI Python bindings, so it’s time to run this code!
Calling Your Function
After all of the work you did to configure and run the CFFI compiler, using the generated Python bindings looks just like using any other Python module:
You import the new module, and then you can call cmult() directly. To test it out, use the test-cffi task:
This runs your cffi_test.py program, which tests out the new Python bindings you’ve created with CFFI . That completes the section on writing and using your CFFI Python bindings.
Strengths and Weaknesses
It might seem that ctypes requires less work than the CFFI example you just saw. While this is true for this use case, CFFI scales to larger projects much better than ctypes due to automation of much of the function wrapping.
CFFI also produces quite a different user experience. ctypes allows you to load a pre-existing C library directly into your Python program. CFFI , on the other hand, creates a new Python module that can be loaded like other Python modules.
What’s more, with the out-of-line-API method you used above, the time penalty for creating the Python bindings is done once when you build it and doesn’t happen each time you run your code. For small programs, this might not be a big deal, but CFFI scales better to larger projects in this way, as well.
Like ctypes , using CFFI only allows you to interface with C libraries directly. C++ libraries require a good deal of work to use. In the next section, you’ll see a Python bindings tool that focuses on C++.
PyBind11
PyBind11 takes a quite different approach to create Python bindings. In addition to shifting the focus from C to C++, it also uses C++ to specify and build the module, allowing it to take advantage of the metaprogramming tools in C++. Like CFFI , the Python bindings generated from PyBind11 are a full Python module that can be imported and used directly.
PyBind11 is modeled after the Boost::Python library and has a similar interface. It restricts its use to C++11 and newer, however, which allows it to simplify and speed things up compared to Boost, which supports everything.
How It’s Installed
The First Steps section of the PyBind11 documentation walks you through how to download and build the test cases for PyBind11 . While this doesn’t appear to be strictly required, working through these steps will ensure you’ve got the proper C++ and Python tools set up.
Note: Most of the examples for PyBind11 use cmake , which is a fine tool for building C and C++ projects. For this demo, however, you’ll continue to use the invoke tool, which follows the instructions in the Building Manually section of the docs.
You’ll want to install this tool into your virtual environment:
PyBind11 is an all-header library, similar to much of Boost. This allows pip to install the actual C++ source for the library directly into your virtual environment.
Calling the Function
Before you dive in, please note that you’re using a different C++ source file, cppmult.cpp , instead of the C file you used for the previous examples. The function is essentially the same in both languages.
Writing the Bindings
Similar to CFFI , you need to create some code to tell the tool how to build your Python bindings. Unlike CFFI , this code will be in C++ instead of Python. Fortunately, there’s a minimal amount of code required:
Let’s look at this a piece at a time, as PyBind11 packs a lot of information into a few lines.
The first two lines include the pybind11.h file and the header file for your C++ library, cppmult.hpp . After that, you have the PYBIND11_MODULE macro. This expands into a block of C++ code that’s well described in the PyBind11 source:
This macro creates the entry point that will be invoked when the Python interpreter imports an extension module. The module name is given as the first argument and it should not be in quotes. The second macro argument defines a variable of type py::module which can be used to initialize the module. (Source)
What this means for you is that, for this example, you’re creating a module called pybind11_example and that the rest of the code will use m as the name of the py::module object. On the next line, inside the C++ function you’re defining, you create a docstring for the module. While this is optional, it’s a nice touch to make your module more Pythonic.
Finally, you have the m.def() call. This will define a function that’s exported by your new Python bindings, meaning it will be visible from Python. In this example, you’re passing three parameters:
- cpp_function is the exported name of the function that you’ll use in Python. As this example shows, it doesn’t need to match the name of the C++ function.
- &cppmult takes the address of the function to be exported.
- «A function. « is an optional docstring for the function.
Now that you have the code for the Python bindings, take a look at how you can build this into a Python module.
Build the Python Bindings
The tool you use to build the Python bindings in PyBind11 is the C++ compiler itself. You may need to modify the defaults for your compiler and operating system.
To begin, you must build the C++ library for which you’re creating bindings. For an example this small, you could build the cppmult library directly into the Python bindings library. However, for most real-world examples, you’ll have a pre-existing library you want to wrap, so you’ll build the cppmult library separately. The build is a standard call to the compiler to build a shared library:
Running this with invoke build-cppmult produces libcppmult.so :
The build for the Python bindings, on the other hand, requires some special details:
Let’s walk through this line-by-line. Line 3 contains fairly standard C++ compiler flags that indicate several details, including that you want all warnings caught and treated as errors, that you want a shared library, and that you’re using C++11.
Line 4 is the first step of the magic. It calls the pybind11 module to have it produce the proper include paths for PyBind11 . You can run this command directly on the console to see what it does:
Your output should be similar but show different paths.
In line 5 of your compilation call, you can see that you’re also adding the path to the Python dev includes . While it’s recommended that you don’t link against the Python library itself, the source needs some code from Python.h to work its magic. Fortunately, the code it uses is fairly stable across Python versions.
Line 5 also uses -I . to add the current directory to the list of include paths. This allows the #include <cppmult.hpp> line in your wrapper code to be resolved.
Line 6 specifies the name of your source file, which is pybind11_wrapper.cpp . Then, on line 7 you see some more build magic happening. This line specifies the name of the output file. Python has some particular ideas on module naming, which include the Python version, the machine architecture, and other details. Python also provides a tool to help with this called python3.7-config :
You may need to modify the command if you’re using a different version of Python. Your results will likely change if you’re using a different version of Python or are on a different operating system.
The final line of your build command, line 8, points the linker at the libcppmult library you built earlier. The rpath section tells the linker to add information to the shared library to help the operating system find libcppmult at runtime. Finally, you’ll notice that this string is formatted with the cpp_name and the extension_name . You’ll be using this function again when you build your Python bindings module with Cython in the next section.
Run this command to build your bindings:
That’s it! You’ve built your Python bindings with PyBind11 . It’s time to test it out!
Calling Your Function
Similar to the CFFI example above, once you’ve done the heavy lifting of creating the Python bindings, calling your function looks like normal Python code:
Since you used pybind11_example as the name of your module in the PYBIND11_MODULE macro, that’s the name you import. In the m.def() call you told PyBind11 to export the cppmult function as cpp_function , so that’s what you use to call it from Python.
You can test it with invoke as well:
That’s what PyBind11 looks like. Next, you’ll see when and why PyBind11 is the right tool for the job.
Strengths and Weaknesses
PyBind11 is focused on C++ instead of C, which makes it different from ctypes and CFFI . It has several features that make it quite attractive for C++ libraries:
- It supports classes.
- It handles polymorphic subclassing.
- It allows you to add dynamic attributes to objects from Python and many other tools, which would be quite difficult to do from the C-based tools you’ve examined.
That being said, there’s a fair bit of setup and configuration you need to do to get PyBind11 up and running. Getting the installation and build correct can be a bit finicky, but once that’s done, it seems fairly solid. Also, PyBind11 requires that you use at least C++11 or newer. This is unlikely to be a big restriction for most projects, but it may be a consideration for you.
Finally, the extra code you need to write to create the Python bindings is in C++ and not Python. This may or may not be an issue for you, but it is different than the other tools you’ve looked at here. In the next section, you’ll move on to Cython , which takes quite a different approach to this problem.
Cython
The approach Cython takes to creating Python bindings uses a Python-like language to define the bindings and then generates C or C++ code that can be compiled into the module. There are several methods for building Python bindings with Cython . The most common one is to use setup from distutils . For this example, you’ll stick with the invoke tool, which will allow you to play with the exact commands that are run.
How It’s Installed
Cython is a Python module that can be installed into your virtual environment from PyPI:
Again, if you’ve installed the requirements.txt file into your virtual environment, then this will already be there. You can grab a copy of requirements.txt by clicking on the link below:
Get Sample Code: Click here to get the sample code you’ll use to learn about Python Bindings in this tutorial.
That should have you ready to work with Cython !
Calling the Function
To build your Python bindings with Cython , you’ll follow similar steps to those you used for CFFI and PyBind11 . You’ll write the bindings, build them, and then run Python code to call them. Cython can support both C and C++. For this example, you’ll use the cppmult library that you used for the PyBind11 example above.
Write the Bindings
The most common form of declaring a module in Cython is to use a .pyx file:
There are two sections here:
- Lines 3 and 4 tell Cython that you’re using cppmult() from cppmult.hpp .
- Lines 6 and 7 create a wrapper function, pymult() , to call cppmult() .
The language used here is a special mix of C, C++, and Python. It will look fairly familiar to Python developers, though, as the goal is to make the process easier.
The first section with cdef extern. tells Cython that the function declarations below are also found in the cppmult.hpp file. This is useful for ensuring that your Python bindings are built against the same declarations as your C++ code. The second section looks like a regular Python function—because it is! This section creates a Python function that has access to the C++ function cppmult .
Now that you’ve got the Python bindings defined, it’s time to build them!
Build the Python Bindings
The build process for Cython has similarities to the one you used for PyBind11 . You first run Cython on the .pyx file to generate a .cpp file. Once you’ve done this, you compile it with the same function you used for PyBind11 :
You start by running cython on your .pyx file. There are a few options you use on this command:
- —cplus tells the compiler to generate a C++ file instead of a C file.
- -3 switches Cython to generate Python 3 syntax instead of Python 2.
- -o cython_wrapper.cpp specifies the name of the file to generate.
Once the C++ file is generated, you use the C++ compiler to generate the Python bindings, just as you did for PyBind11 . Note that the call to produce the extra include paths using the pybind11 tool is still in that function. It won’t hurt anything here, as your source will not need those.
Running this task in invoke produces this output:
You can see that it builds the cppmult library and then builds the cython module to wrap it. Now you have the Cython Python bindings. (Try saying that quickly…) It’s time to test it out!
Calling Your Function
The Python code to call your new Python bindings is quite similar to what you used to test the other modules:
Line 2 imports your new Python bindings module, and you call pymult() on line 7. Remember that the .pyx file provided a Python wrapper around cppmult() and renamed it to pymult . Using invoke to run your test produces the following:
You get the same result as before!
Strengths and Weaknesses
Cython is a relatively complex tool that can provide you a deep level of control when creating Python bindings for either C or C++. Though you didn’t cover it in depth here, it provides a Python-esque method for writing code that manually controls the GIL, which can significantly speed up certain types of problems.
That Python-esque language is not quite Python, however, so there’s a slight learning curve when you’re coming up to speed in figuring out which parts of C and Python fit where.
Other Solutions
While researching this tutorial, I came across several different tools and options for creating Python bindings. While I limited this overview to some of the more common options, there are several other tools I stumbled across. The list below is not comprehensive. It’s merely a sampling of other possibilities if one of the above tools doesn’t fit your project.
PyBindGen
PyBindGen generates Python bindings for C or C++ and is written in Python. It’s targeted at producing readable C or C++ code, which should simplify debugging issues. It wasn’t clear if this has been updated recently, as the documentation lists Python 3.4 as the latest tested version. There have been yearly releases for the last several years, however.
Boost.Python
Boost.Python has an interface similar to PyBind11 , which you saw above. That’s not a coincidence, as PyBind11 was based on this library! Boost.Python is written in full C++ and supports most, if not all, versions of C++ on most platforms. In contrast, PyBind11 restricts itself to modern C++.
SIP is a toolset for generating Python bindings that was developed for the PyQt project. It’s also used by the wxPython project to generate their bindings, as well. It has a code generation tool and an extra Python module that provides support functions for the generated code.
Cppyy
cppyy is an interesting tool that has a slightly different design goal than what you’ve seen so far. In the words of the package author:
“The original idea behind cppyy (going back to 2001), was to allow Python programmers that live in a C++ world access to those C++ packages, without having to touch C++ directly (or wait for the C++ developers to come around and provide bindings).” (Source)
Shiboken
Shiboken is a tool for generating Python bindings that’s developed for the PySide project associated with the Qt project. While it was designed as a tool for that project, the documentation indicates that it’s neither Qt- nor PySide-specific and is usable for other projects.
SWIG is a different tool than any of the others listed here. It’s a general tool used to create bindings to C and C++ programs for many other languages, not just Python. This ability to generate bindings for different languages can be quite useful in some projects. It, of course, comes with a cost as far as complexity is concerned.
Conclusion
Congrats! You’ve now had an overview of several different options for creating Python bindings. You’ve learned about marshalling data and issues you need to consider when creating bindings. You’ve seen what it takes to be able to call a C or C++ function from Python using the following tools:
- ctypes
- CFFI
- PyBind11
- Cython
You now know that, while ctypes allow you to load a DLL or shared library directly, the other three tools take an extra step, but still create a full Python module. As a bonus, you’ve also played a little with the invoke tool to run command-line tasks from Python. You can get all of the code you saw in this tutorial by clicking the link below:
Get Sample Code: Click here to get the sample code you’ll use to learn about Python Bindings in this tutorial.
Now pick your favorite tool and start building those Python bindings! Special thanks to Loic Domaigne for the extra technical review of this tutorial.
Get a short & sweet Python Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.
About Jim Anderson
Jim has been programming for a long time in a variety of languages. He has worked on embedded systems, built distributed build systems, done off-shore vendor management, and sat in many, many meetings.
Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:
Master Real-World Python Skills With Unlimited Access to Real Python
Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:
Master Real-World Python Skills
With Unlimited Access to Real PythonJoin us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:
What Do You Think?
What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.
Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students. Get tips for asking good questions and get answers to common questions in our support portal. Looking for a real-time conversation? Visit the Real Python Community Chat or join the next “Office Hours” Live Q&A Session. Happy Pythoning!
How do I run a Python script from C#?
This sort of question has been asked before in varying degrees, but I feel it has not been answered in a concise way and so I ask it again.
I want to run a script in Python. Let’s say it’s this:
Which gets a file location, reads it, then prints its contents. Not so complicated.
Okay, so how do I run this in C#?
This is what I have now:
When I pass the code.py location as cmd and the filename location as args it doesn’t work. I was told I should pass python.exe as the cmd , and then code.py filename as the args .
I have been looking for a while now and can only find people suggesting to use IronPython or such. But there must be a way to call a Python script from C#.
I need to run it from C#, I need to capture the output, and I can’t use IronPython or anything else. Whatever hack you have will be fine.