Python сколько памяти использует программа

resource — Resource usage information¶

This module provides basic mechanisms for measuring and controlling system resources utilized by a program.

Availability : not Emscripten, not WASI.

This module does not work or is not available on WebAssembly platforms wasm32-emscripten and wasm32-wasi . See WebAssembly platforms for more information.

Symbolic constants are used to specify particular system resources and to request usage information about either the current process or its children.

An OSError is raised on syscall failure.

exception resource. error ¶

A deprecated alias of OSError .

Changed in version 3.3: Following PEP 3151, this class was made an alias of OSError .

Resource Limits¶

Resources usage can be limited using the setrlimit() function described below. Each resource is controlled by a pair of limits: a soft limit and a hard limit. The soft limit is the current limit, and may be lowered or raised by a process over time. The soft limit can never exceed the hard limit. The hard limit can be lowered to any value greater than the soft limit, but not raised. (Only processes with the effective UID of the super-user can raise a hard limit.)

The specific resources that can be limited are system dependent. They are described in the getrlimit(2) man page. The resources listed below are supported when the underlying operating system supports them; resources which cannot be checked or controlled by the operating system are not defined in this module for those platforms.

Constant used to represent the limit for an unlimited resource.

resource. getrlimit ( resource ) ¶

Returns a tuple (soft, hard) with the current soft and hard limits of resource. Raises ValueError if an invalid resource is specified, or error if the underlying system call fails unexpectedly.

resource. setrlimit ( resource , limits ) ¶

Sets new limits of consumption of resource. The limits argument must be a tuple (soft, hard) of two integers describing the new limits. A value of RLIM_INFINITY can be used to request a limit that is unlimited.

Raises ValueError if an invalid resource is specified, if the new soft limit exceeds the hard limit, or if a process tries to raise its hard limit. Specifying a limit of RLIM_INFINITY when the hard or system limit for that resource is not unlimited will result in a ValueError . A process with the effective UID of super-user can request any valid limit value, including unlimited, but ValueError will still be raised if the requested limit exceeds the system imposed limit.

setrlimit may also raise error if the underlying system call fails.

VxWorks only supports setting RLIMIT_NOFILE .

Raises an auditing event resource.setrlimit with arguments resource , limits .

resource. prlimit ( pid , resource [ , limits ] ) ¶

Combines setrlimit() and getrlimit() in one function and supports to get and set the resources limits of an arbitrary process. If pid is 0, then the call applies to the current process. resource and limits have the same meaning as in setrlimit() , except that limits is optional.

When limits is not given the function returns the resource limit of the process pid. When limits is given the resource limit of the process is set and the former resource limit is returned.

Raises ProcessLookupError when pid can’t be found and PermissionError when the user doesn’t have CAP_SYS_RESOURCE for the process.

Raises an auditing event resource.prlimit with arguments pid , resource , limits .

New in version 3.4.

These symbols define resources whose consumption can be controlled using the setrlimit() and getrlimit() functions described below. The values of these symbols are exactly the constants used by C programs.

The Unix man page for getrlimit(2) lists the available resources. Note that not all systems use the same symbol or same value to denote the same resource. This module does not attempt to mask platform differences — symbols not defined for a platform will not be available from this module on that platform.

The maximum size (in bytes) of a core file that the current process can create. This may result in the creation of a partial core file if a larger core would be required to contain the entire process image.

The maximum amount of processor time (in seconds) that a process can use. If this limit is exceeded, a SIGXCPU signal is sent to the process. (See the signal module documentation for information about how to catch this signal and do something useful, e.g. flush open files to disk.)

The maximum size of a file which the process may create.

The maximum size (in bytes) of the process’s heap.

The maximum size (in bytes) of the call stack for the current process. This only affects the stack of the main thread in a multi-threaded process.

The maximum resident set size that should be made available to the process.

The maximum number of processes the current process may create.

The maximum number of open file descriptors for the current process.

The maximum address space which may be locked in memory.

The largest area of mapped memory which the process may occupy.

The maximum area (in bytes) of address space which may be taken by the process.

The number of bytes that can be allocated for POSIX message queues.

New in version 3.4.

The ceiling for the process’s nice level (calculated as 20 — rlim_cur).

New in version 3.4.

The ceiling of the real-time priority.

New in version 3.4.

The time limit (in microseconds) on CPU time that a process can spend under real-time scheduling without making a blocking syscall.

New in version 3.4.

The number of signals which the process may queue.

New in version 3.4.

The maximum size (in bytes) of socket buffer usage for this user. This limits the amount of network memory, and hence the amount of mbufs, that this user may hold at any time.

New in version 3.4.

The maximum size (in bytes) of the swap space that may be reserved or used by all of this user id’s processes. This limit is enforced only if bit 1 of the vm.overcommit sysctl is set. Please see tuning(7) for a complete description of this sysctl.

New in version 3.4.

The maximum number of pseudo-terminals created by this user id.

New in version 3.4.

The maximum number of kqueues this user id is allowed to create.

New in version 3.10.

Resource Usage¶

These functions are used to retrieve resource usage information:

resource. getrusage ( who ) ¶

This function returns an object that describes the resources consumed by either the current process or its children, as specified by the who parameter. The who parameter should be specified using one of the RUSAGE_* constants described below.

A simple example:

The fields of the return value each describe how a particular system resource has been used, e.g. amount of time spent running is user mode or number of times the process was swapped out of main memory. Some values are dependent on the clock tick internal, e.g. the amount of memory the process is using.

For backward compatibility, the return value is also accessible as a tuple of 16 elements.

The fields ru_utime and ru_stime of the return value are floating point values representing the amount of time spent executing in user mode and the amount of time spent executing in system mode, respectively. The remaining values are integers. Consult the getrusage(2) man page for detailed information about these values. A brief summary is presented here:

How to get current CPU and RAM usage in Python?

How can I get the current system status (current CPU, RAM, free disk space, etc.) in Python? Ideally, it would work for both Unix and Windows platforms.

There seems to be a few possible ways of extracting that from my search:

Using a library such as PSI (that currently seems not actively developed and not supported on multiple platforms) or something like pystatgrab (again no activity since 2007 it seems and no support for Windows).

Using platform specific code such as using a os.popen("ps") or similar for the *nix systems and MEMORYSTATUS in ctypes.windll.kernel32 (see this recipe on ActiveState) for the Windows platform. One could put a Python class together with all those code snippets.

It’s not that those methods are bad but is there already a well-supported, multi-platform way of doing the same thing?

21 Answers 21

The psutil library gives you information about CPU, RAM, etc., on a variety of platforms:

psutil is a module providing an interface for retrieving information on running processes and system utilization (CPU, memory) in a portable way by using Python, implementing many functionalities offered by tools like ps, top and Windows task manager.

It currently supports Linux, Windows, OSX, Sun Solaris, FreeBSD, OpenBSD and NetBSD, both 32-bit and 64-bit architectures, with Python versions from 2.6 to 3.5 (users of Python 2.4 and 2.5 may use 2.1.3 version).

Here’s other documentation that provides more concepts and interest concepts:

Use the psutil library. On Ubuntu 18.04, pip installed 5.5.0 (latest version) as of 1-30-2019. Older versions may behave somewhat differently. You can check your version of psutil by doing this in Python:

To get some memory and CPU stats:

The virtual_memory (tuple) will have the percent memory used system-wide. This seemed to be overestimated by a few percent for me on Ubuntu 18.04.

You can also get the memory used by the current Python instance:

which gives the current memory use of your Python script.

There are some more in-depth examples on the pypi page for psutil.

Only for Linux: One-liner for the RAM usage with only stdlib dependency:

One can get real time CPU and RAM monitoring by combining tqdm and psutil . It may be handy when running heavy computations / processing.

It also works in Jupyter without any code changes:

It’s convenient to put those progress bars in separate process using multiprocessing library.

This code snippet is also available as a gist.

Below codes, without external libraries worked for me. I tested at Python 2.7.9

CPU Usage

And Ram Usage, Total, Used and Free

To get a line-by-line memory and time analysis of your program, I suggest using memory_profiler and line_profiler .

Installation:

The common part is, you specify which function you want to analyse by using the respective decorators.

Example: I have several functions in my Python file main.py that I want to analyse. One of them is linearRegressionfit() . I need to use the decorator @profile that helps me profile the code with respect to both: Time & Memory.

Make the following changes to the function definition

For Time Profiling,

Run:

For Memory Profiling,

Run:

Also, the memory profiler results can also be plotted using matplotlib using

Note: Tested on

line_profiler version == 3.0.2

memory_profiler version == 0.57.0

psutil version == 5.7.0

EDIT: The results from the profilers can be parsed using the TAMPPA package. Using it, we can get line-by-line desired plots as

We chose to use usual information source for this because we could find instantaneous fluctuations in free memory and felt querying the meminfo data source was helpful. This also helped us get a few more related parameters that were pre-parsed.

Code

Output for reference (we stripped all newlines for further analysis)

MemTotal: 1014500 kB MemFree: 562680 kB MemAvailable: 646364 kB Buffers: 15144 kB Cached: 210720 kB SwapCached: 0 kB Active: 261476 kB Inactive: 128888 kB Active(anon): 167092 kB Inactive(anon): 20888 kB Active(file): 94384 kB Inactive(file): 108000 kB Unevictable: 3652 kB Mlocked: 3652 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 168160 kB Mapped: 81352 kB Shmem: 21060 kB Slab: 34492 kB SReclaimable: 18044 kB SUnreclaim: 16448 kB KernelStack: 2672 kB PageTables: 8180 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 507248 kB Committed_AS: 1038756 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 88064 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 43008 kB DirectMap2M: 1005568 kB

Measuring memory usage in Python: it’s tricky!

If you want your program to use less memory, you will need to measure memory usage. You’ll want to measure the current usage, and then you’ll need to ensure it’s using less memory once you make some improvements.

It turns out, however, that measuring memory usage isn’t as straightforward as you’d think. Even with a highly simplified model of how memory works, different measurements are useful in different situations.

In this article you’ll learn:

• A simplified but informative model of how memory works.
• Two measures of memory–resident memory and allocated memory–and how to measure them in Python.
• The tradeoffs between the two.

A (not so) simple example

Consider the following code:

This creates an array of 3GB–gibibytes, specifically–filled with ones. Naively, once this code runs we expect the process to be using a little bit more than 3GB.

One way to measure memory is using “resident memory”, which we’ll define later in the article. We can get this information using the handy psutil library, checking the resident memory of the current process:

With this particular measurement, we’re using 3083MB, or 3.08GB, and the difference from the array size is no doubt the memory used by the Python interpreter and the libraries we’ve imported. Success!

But resident memory is tricker than it seems. Let’s say I go and open some websites in my browser, leaving the Python interpreter running the background. Then I switch back to the interpreter, and run the exact same command again:

What’s going on? Why did 200MB of memory disappear?

To find the answer we’ll need to learn a bit more about how the operating system manages memory.

A simplified model of memory allocation

A running program allocates some memory, i.e. gets back an address in virtual memory from the operating system. Virtual memory is a process-specific address space, essentially numbers from 0 to 2 64 -1 , where the process can read or write bytes.

In a C program you might use APIs like malloc() or mmap() to do so; in Python you just create objects, and the Python interpreter will call malloc() or mmap() when necessary. The process can then read or write to that particular address and consecutive bytes.

We can see this in action by using the Linux utility ltrace to trace calls to malloc() . Given the following program:

We can run Python under ltrace :

Here’s what happening:

1. Python create a NumPy array.
2. Under the hood NumPy calls malloc() .
3. The result of that malloc() is an address in memory: 0x5638862a45e0 .
4. The C code used to implement NumPy can then read and write to that address and the next consecutive 169,999 addresses, each address representing one byte in virtual memory.

Where are these 170,000 bytes stored?

• They can be stored in RAM; this is the default.
• They can be stored on the computer’s hard drive or disk, in which case they’re said to be stored in swap.
• Some of the bytes might be stored in RAM and some in swap.
• Finally, they can be stored nowhere at all.

For now, we’ll ignore that confusing last case, and just focus on the first three situations.

Resident memory is RAM usage

RAM is fast, and your hard drive is slow… but RAM is also expensive, so typically you’ll have far more hard drive space than RAM. For example, the computer I’m writing this on has about 500GB of hard drive storage, but only 16GB RAM.

Ideally, all of your program’s memory would be stored in fast RAM, but the various processes running on your computer might have allocated more memory than is available in RAM. If that happens, the operating system will move–or “swap”–some data from RAM to hard drive. In particular, it will try to swap data that isn’t actively being used.

And now we’re ready to define our first measure of memory usage: resident memory. Resident memory is how much of the process’ allocated memory is resident–or stored–in RAM.

In our first example we started out with all 3GB the allocated array stored in RAM.

Then when I opened a bunch of browser tabs, loading those websites used quite a lot of RAM, and so the operating system decided to swap some data from RAM to disk. And part of the memory that got swapped was from our array. As a result, the resident memory for our Python process went down: the data was all still accessible, but some of it had been moved to disk.

An alternative: allocated memory

It would be useful to measure allocated memory, to always get 3GB back regardless of whether the operating system put the data in RAM or swapped it to disk. This would give us consistent results, and also tell us how much memory the program actually asked for.

In Python (if you’re on Linux or macOS), you can measure allocated memory using the Fil memory profiler, which specifically measures peak allocated memory. The Sciagraph profiler is another alternative: unlike Fil, it also does performance profiling, and it’s lower overhead by using sampling.

Here’s the output of Fil for our example allocation of 3GB:

The tradeoffs between resident memory and allocated memory

As a method of measuring memory, resident memory has some problems:

• Other processes can distort the results by using up limited RAM and encouraging swapping, as they did in our 3GB example above.
• Because resident memory is capped at available physical RAM, you never actually learn how much memory the program asks for, once you hit that ceiling. If you have 16GB of RAM you won’t be able to distinguish between a program that needs 17GB memory and a program that needs 30GB of memory: they will both report the same resident memory.

Allocated memory, on the other hand, is not affected by other processes, and tells you what the program actually requested.

Of course, resident memory does have some advantages over allocated memory:

• That swapped memory may well never be used: imagine creating an array, forgetting to delete the reference, and then never actually using it again for the rest of the program. If it gets swapped, that’s fine, and arguably we shouldn’t be measuring it.
• More broadly, because resident memory measures actually used RAM from the operating system perspective, it can capture edge cases that aren’t visible to the allocated memory tracking.

Let’s look at an example of one such edge case.

Phantom memory!

In all our examples so far we’ve been allocating arrays that are full of ones. If we’re measuring allocated memory, what the array is filled with makes no difference: we can switch to creating arrays full of zeroes, and still get the exact same result.

But on Linux, resident memory tells us an interesting story:

Once again, we’ve allocated a 3GB array, this time full of zeroes. We measure resident memory and–the array isn’t counted, resident memory is just 29M. Where did the array go?

It turns out that Linux doesn’t bother storing all those zeroes in RAM. Instead, it will add chunks of zeroes to RAM only when the data is actually accessed–until then no RAM is used. Until that happens, allocated memory will report 3GB, but resident memory will notice that those 3GB aren’t actually using any resources.

Allocated memory is a good starting point

It’s worth keeping in mind that this is still a quite simplified model of memory usage. It doesn’t cover the file cache, or memory fragmentation in the allocator, or other available metrics–some useful, some less so–you can get from Linux.

That being said, for many applications allocated memory is probably sufficient as a first-pass measure to help you optimize your program’s memory use. Allocated memory tells you what your application actually asked for, and that’s usually what you’re going to have to optimize.

Learn even more techniques for reducing memory usage—read the rest of the Larger-than-memory datasets guide for Python.

Find performance and memory bottlenecks in your data processing code with the Sciagraph profiler

Slow-running jobs waste your time during development, impede your users, and increase your compute costs. Speed up your code and you’ll iterate faster, have happier users, and stick to your budget—but first you need to identify the cause of the problem.

Find performance bottlenecks and memory hogs in your data science Python jobs with the Sciagraph profiler. Profile in development and production, with multiprocessing support and more.

Learn practical Python software engineering skills you can use at your job

Sign up for my newsletter, and join over 6700 Python developers and data scientists learning practical tools and techniques, from Python performance to Docker packaging, with a free new article in your inbox every week.

Про большой расход памяти в Python

Если вы работаете на современном компьютере и не пишите приложения схожие с GTA 5, то возможно вы никогда и не задумывались, сколько памяти расходует ваша программа. Даже если вы data engineer, и вам приходится работать с большим количеством данных, возможно, вы встречались с memory error. В дальнейшим мы рассмотрим сколько расходуют памяти примитивные объекты в python.

И прежде чем лезть в дебри, укажем момент работы с памятью в Python, который должен знать даже программист, который никогда не работает с крупными проектами или большими данными.

Работа в Python с памятью происходит достаточно просто и понятно, используя систему счетчиков ссылок, и память, занимавшую этим объектом, освобождается, когда счетчик равен нулю.

Рассмотрим размер основных примитивных типов данных: integer(int), float, tuple, string(str), list, dict.

основные объекты

Каков размер самого простого и часто используемого объекта int? Если вы перешли на Python с какого-либо С подобного языка, то ответите, что не более 8 байт. Сравним с python:

Я использую Python версии 3.7.0 64x.

В 64 битном С, int занимает 4 байта, а это гораздо меньше, чем в Python.

Числа с плавающей запятой в Python:

А что насчёт строковых значений?

Пустая строка в Python в 64 битной среде, занимает 49 байт!

Затем потребление памяти увеличивается за счет увеличения полезного размера значения.

Рассмотрим еще несколько не менее важных объектов:

В 64-битном С размер пустого std::list() равен 16 байт и это в 4 раза меньше, чем в Python.

Например, пустой словарь в С занимает 48 байт.

Ну что? Мы разобрали занимаемый объем памяти самых популярных объектов в Python. Но, наверное, никто из читателей после этого сравнения не перейдет на С, я тоже не перейду. Для того, чтобы качественно писать код, с точки зрения оптимизации, мы должны иметь представление о требованиях создаваемых нами объектов. Посмотрим, как происходит выделение памяти под капотом…

внутреннее управление памятью

Я упоминал выше, что когда счетчик ссылок обнуляется, память освобождается, звучит это просто, понятно и логично, но здесь есть несколько подводных камней.

Обратимся к истории…

Возьмём любую версию python, ниже 2.1.

Если заглянуть под капот, то мы обнаружим, что если ОС выделила Python программе память, то эта память никогда не вернется в ОС. Интерпретатор Python оставляет эту память себе, до следующего использования, это ускоряет работу программы, так как ОС не требуется постоянное выделение памяти для процесса. Но если в программе есть место, где потребление памяти резко возрастает на небольшой промежуток времени и дальше программа требует значительно меньше памяти, то Python всю память, которая требуется для этого пика будет сохранять за собой и не отдаст ее ОС, что приводит к уменьшению производительности системы в целом.

Распределитель памяти Python, называемый pymalloc, был написан Владимиром Марангозова и изначально был экспериментальной функцией Python 2.1 и 2.2, прежде чем стать включенной по умолчанию в 2.3.

В python программах используется много различных мелких объектов, которые то создаются, то уничтожаются, для этого вызываются функции malloc() (для выделения) и free() (для освобождения). Я уже упоминал, что выделение памяти ОС несколько затяжной процесс, поэтому pymalloc выделяет память куском в 256 Кб и называется это арена. Сама арена делится на пулы размер каждого 4Кб, и пулы уже делятся на блоки фиксированного размера под наши объекты.

Если мы создаем объект, распределитель проверяет, есть ли пулы, с блоками нужного нам размера. Связанный массив usedpools хранит пулы каждого размера. И каждый пул имеет связанный список доступных блоков. Если есть нужный нам блок, то мы берем его из списка пула, эта операция очень быстрая.

Если нет пулов, разбитых на нужные нам блоки, то нужно найти свободный пул, свободные пулы хранятся в связанном списке freepools. Если в списке есть пул, мы берем его, если нет, то придется выделить новый пул, если в Арене еще есть место, то мы отрежем новый пул используя arenabase. Если в Арене нет места, придется выделить новую Арену, вызвав malloc(). Теперь, когда у нас есть нужные блоки, забираем его с помощью usedpools.

Когда программа удаляет объект, освобождается блок. Блок помещается в свободный список пула и если пул был полностью выделен, то он добавляется к usedpools (список пулов, имеющих свободные блоки), а если пул теперь полностью свободен, то он добавляется к freepools.

В данной статье мы рассмотрели, как устроено распределение памяти в Python. Как вы могли заметить, Python очень прожорливый на память язык.