How to find memory usage with memory_profiler Python?

Question

At the end of this question I found some code. I will put it here for convenience:

import memory_profiler as mp

def fun(n):
    tmp = []
    for i in range(n):
        tmp.extend(list(range(i*i)))
    return "XXXXX"

start_mem = mp.memory_usage(max_usage=True)
res = mp.memory_usage(proc=(fun, [100]), max_usage=True, retval=True) 
print('start mem', start_mem)
print('max mem', res[0][0])
print('used mem', res[0][0]-start_mem)
print('fun output', res[1])

But this doesn't work because res is not a double array, it is float number. Moreover, I don't understand how to check memory usage for many functions. I mean, would something like this work?

import memory_profiler as mp

def fun1(n):
    return "XXXXX"
def fun2(n):
    return "YYYYY"

methods = [
    'fun1(n)',
    'fun2(n)',
    ]

start_mem = mp.memory_usage(max_usage=True)
res = mp.memory_usage(proc=(methods[0], [100]), max_usage=True, retval=True) 
print('start mem', start_mem)
print('max mem', res[0][0])
print('used mem', res[0][0]-start_mem)
print('fun output', res[1])

score 3 · Answer 1 · edited May 27 '21 at 21:17

3

Installing memory_profiler:

pip3 install -U memory_profiler

Sometimes it is useful to have full memory usage reports as a function of time (not line-by-line) of external processes (be it Python scripts or not). In this case, the executable mprof might be useful

To get the line by line profiling, add decorator ‘@profile’ in front of the def function.

from memory_profiler import profile
@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

After this save the .py file and run the below code in terminal.

mprof run perf_test_v1.py

This will show you the memory usage by every line of code. To Plot the memory usage vs time through the below code.

mprof plot

The available commands for mprof are:

mprof run: running an executable, recording memory usage
mprof plot: plotting one the recorded memory usage (by default, the last one)
mprof list: listing all recorded memory usage files in a user-friendly way.
mprof clean: removing all recorded memory usage files.
mprof rm: removing specific recorded memory usage files

This would help you identify the bottlenecks in your codes and helps in optimizing the steps.

To get time-based memory usage

Execute the code passing the option -m memory_profiler to the python interpreter to load the memory_profiler module and print to stdout the line-by-line analysis. If the file name was example.py, this would result in:

$ python -m memory_profiler example.py

edited May 27 '21 at 21:17

halfer

19,471
17
87
173

answered May 22 '21 at 15:30

Alok Raj

1,000
9
17

Is it possible to use mprof without terminal? – dotagenius May 22 '21 at 15:39
There are two ways you can run it :- – Alok Raj May 22 '21 at 15:47
Sometimes it is useful to have full memory usage reports as a function of time (not line-by-line) of external processes (be it Python scripts or not). In this case, the executable mprof might be useful. – Alok Raj May 22 '21 at 15:48
Execute the code passing the option -m memory_profiler to the python interpreter to load the memory_profiler module and print to stdout the line-by-line analysis. If the file name was example.py, this would result in: $ python -m memory_profiler example.py – Alok Raj May 22 '21 at 15:48
Added in the answer itself, please check it once – Alok Raj May 22 '21 at 15:52
It is still terminal but I need something like this: `x = memory_profiler.memory_usage(fun1(10))` – dotagenius May 22 '21 at 16:13
There are other libraries which can help then like Guppy, Pympler etc.Try those they are also good – Alok Raj May 22 '21 at 16:30

How to find memory usage with memory_profiler Python?

1 Answers1