Introduction

Python code profiling is an important technique that helps to understand the code performance and identify potential bottlenecks. In this article, I would like to share my experience of profiling package usage and cover some of the most useful of them.

Types of profiling

The are multiple points of view on how you can profile the Python code, everything depends on your goals:

• Performance (or time) Optimization: by measuring the execution time of individual functions and lines of code, you can pinpoint the parts of their application that are slowing it down and make changes to improve its overall performance. For example, you can identify the slowest blocks of code and add multi-threading for them, it's important to understand the exact amount of resources in the testing and production environment and chose the right way of parallelization. In addition to it, Numpy, Pandas, Tensorflow, Pytorch libraries, for example, allow you to write the same code in multiple ways and not necessarily of the same performance. You can investigate deeper slow code parts and rewrite the same logic with more efficient functions or by using the more efficient read/write methods, etc.

• Memory Management: Python profiling packages can be used to identify memory leaks and other issues related to memory management. In Python, memory management is handled automatically by the interpreter, but this can lead to performance issues if memory is not released when it is no longer needed. For example, you can track how much memory is allocated to the execution of your script line by line and add the garbage collector calls to the proper places.

• Code Refactoring: specific Python profiling packages can also be used to identify areas of the code that could benefit from refactoring. Refactoring involves restructuring code to make it more efficient, easier to understand, and easier to maintain. It's not so precisely defined as Latency or Memory consumption, but usually good practice to rely on Cyclomatic complexity and Maintainability Index.

• Code coverage: code coverage is a measure of how much of your code is executed when running tests. It's very important because it helps to ensure that your tests are comprehensive and cover all possible code execution scenarios.

Libraries

Let's dive deeper into the most convenient libraries of each Profiling and toy examples of their practical application. (You can use the links from the library names to go directly to the GitHub of the package or documentation of the module).

cProfile

cProfile module is a built-in Python library that provides deterministic profiling of Python programs. The usage of the module is pretty straightforward and doesn't require any external package installation:

import cProfile

def test_function():
    ...

cProfile.run('test_function()')

In this example, we define a function test_function() to profile, and then use cProfile.run() to run the function and generate a profiling report, which will include information about the number of times each function was called, how long it took to run, and how much time was spent in each of its subfunctions (by default report is displayed in the console).

A more convenient way of using cProfile is the combination with pstats module. It provides tools for working with profiling data generated by cProfile, including sorting, filtering, and formatting the data. Here's an example of how to use cProfile with the pstats module:

import cProfile
import pstats

def test_function():
    ...

cProfile.run('test_function()', 'profile_data')
stats = pstats.Stats('profile_data')
stats.sort_stats('cumulative')
stats.print_stats()

In the example above we use cProfile to profile the test_function() function and save the profiling data to a file named profile_data. We then use the pstats module to read the profiling data from the file and sort it by the cumulative time spent in each function. Finally, we print the profiling report to the console.

Pyinstrument

Pyinstrument is an external Python library for profiling code execution time, it can be installed using pip, the Python package manager: pip install pyinstrument .

To use Pyinstrument, we need to create an instance of the pyinstrument.Profiler class and call its start() and stop() methods to profile the code between them:

import pyinstrument

def test_function():
    ...

profiler = pyinstrument.Profiler()
profiler.start()
test_function()
profiler.stop()

print(profiler.output_text())

Pyinstrument also provides a context manager that can be used to profile code within a with block. Here's an example of how to use the context manager:

import pyinstrument

def test_function():
    ...

with pyinstrument.Profiler() as profiler:
    test_function()

print(profiler.output_text())

But it's not all the benefits of this library, otherwise, there will be no sense to mention it here :) The main powerful advantages are:

• Supports profiling of asynchronous code with the asyncio and curio libraries;

• Provides a graphical user interface for viewing profiling reports;

• Can profile code running on remote machines using ssh;

• Can save profiling reports to disk in various formats: text, html, json;

• Supports both CPU and memory profiling.

timeit

To take a step back from external powerful libraries, let's remember about the internal Python module to measure the small code blocks. The timeit module is designed to be used from the command-line interface or from within a Python program. Let's take a look firstly how timeit can be used inside Python script:

import timeit

code_snippet = '''
x = 0
for i in range(10):
    x += i ** 2
    print(i)
'''

execution_time = timeit.timeit(code_snippet, number=1000)
print(f'The execution time is {execution_time} seconds')

The number parameter specifies the number of times the code snippet should be executed to get an accurate measurement of the execution time. You can also use the timeit module from the command line interface by running the following command:

python -m timeit -s "import py_module" "py_module.py_function()"

This command will execute py_function() from py_module and display the execution time. The benefit of such an approach is: you don't need to change the exact code for time measurement as the profiling instruction is being given externally.

Py-spy

Py-Spy is a profiling and tracing tool for Python that can be used to capture the performance data of Python programs. It's a separate package which can be installed with pip install py-spy . Once installed, you can easily run Py-Spy from the command-line interface by running the following command (in this sense it's similar to timeit):

py-spy record -p <PID> -o <output_file>

In this command, PID is the process ID of the Python program you want to profile, and output_file is the name of the output file where the profiling data will be saved.

Py-Spy also provides a feature to profile a running Python program without specifying the PID. To do this, you can use the --python option followed by the command you want to profile:

py-spy record -- python python_script.py

The key advantages that are worth mentioning of this library are:

• Can be used to profile running Python programs without restarting them;

• Provides an easy-to-use command-line interface;

• Supports exporting profiling data in multiple formats including flamegraph, json, and raw data.

FunctionTrace

This is one of the most beautiful (from the visual output perspective) Python profiling libraries, to see the beauty you can quickly check the live demo on the official website. It takes some time to get used to the visualisations if you have never worked with the such a format but in the end, it's a very convenient and interactive way of investigating your Python code. To use FunctionTrace, you can install it using pip by running the usual pip command: pip install functiontrace - this is the Python client of the library. The whole library also includes the server part which can be installed via cargo command: cargo install functiontrace-server . More instructions about it can be found on the main page.

You can modify your code to start tracing at some point during its runtime:

import functiontrace

def my_function():
    print("Hello, World!")

functiontrace.trace(my_function)

as well as trace the whole Python application via functiontrace :

functiontrace py_program.py arguments_for_py_program

As mentioned in the tutorial: the FunctionTrace will emit a trace file (in the form of functiontrace.time.json) to the current directory, though the location can be controlled via the --output-dir switch when running functiontrace.

To upload and view the recorded functiontrace file you can go to the Firefox Profiler and upload the file for visualisation. This is the most convenient way to share the reports with your teammates for collaborative investigation or reports.

Here are the most remarkable features of the FunctionTrace library:

• The library supports multithread and multiprocess applications;

• Profiling is non-sampled and low-overhead: developers guarantee <10% overhead from profiling even on complex applications;

• Very convenient uploading and sharing of profile reports.

Yet Another Python Profiler (yappi)

Yappi library works by instrumenting the Python code, which means it adds extra code to your program to collect performance data. It then generates reports based on the collected data that can help you identify the slowest parts of your code. The library is written in C and it's very fast. Yappi can be installed as a pip package: pip install yappi . It requires from you to select the section of the code that you want to profile and 3 simple lines:

import yappi

# start
yappi.set_clock_type("cpu")
yappi.start()

# YOUR CODE

# stop
yappi.stop()

# report
yappi.get_func_stats().print_all()
yappi.get_thread_stats().print_all()

An important advantage of the library is the support of multithreaded code and you can profile each separate thread of your code:

yappi.start()

threads = []

# YOUR MULTITHREADED CODE

yappi.stop()

# retrieve thread stats by their thread id (given by yappi)
threads = yappi.get_thread_stats()
for thread in threads:
    print(
        "Function stats for (%s) (%d)" % (thread.name, thread.id)
    )  # it is the Thread.__class__.__name__
    yappi.get_func_stats(ctx_id=thread.id).print_all()

Starting from version 1.2 Yappi contains the method of coroutine profiling. With coroutine-profiling, you should be able to profile the correct wall/cpu time and call count of your coroutine.

Radon

Radon is a Python library that can be used to perform code metrics and complexity analysis on Python code. The library currently supports 4 types of metrics to measure the Python code complexity:

• Cyclomatic Complexity (i.e. McCabe’s Complexity)

• raw metrics: SLOC, comment lines, blank lines, &c.

• Halstead metrics (all of them)

• the Maintainability Index (a Visual Studio metric)

Radon uses the Python Abstract Syntax Tree (AST) to analyze code, which allows it to provide accurate results that take into account the actual structure of the code. Radon can be installed as a pip package: pip install radon. If you use random frequently for multiple files, you can specify the exceptions in the configuration file radon.cfg to exclude test files from the analysis or set up limits on cyclomatic complexity. The example of the cyclomatic complexity analysis, for example, is only one line:

radon cc your_code.py -a -nc

cc - cyclomatic complexity;

-a - calculation of the average complexity at the end;

-nc tells radon to print only results with a complexity rank of C or worse.

Coverage.py

Coverage.py is a Python library used for measuring the code coverage of Python programs. It is an essential tool for Python developers who want to ensure that their code is well-tested and has sufficient test coverage. Coverage measurement is typically used to gauge the effectiveness of tests. It can show which parts of your code are being exercised by tests, and which are not. The library can be installed as a pip package: pip install coverage.

The easiest way is to use the command line tool to run the coverage of the file or module:

coverage run --source=dir1,dir2 my_program.py arg1 arg2
# OR
coverage run --source=dir1,dir2 -m packagename.modulename arg1 arg2

For a more detailed description of the arguments, please read the documentation here.

Conclusion

To sum up, profiling is a crucial method for detecting performance obstacles and enhancing Python programs to achieve faster speed, better memory usage, and better code quality. Open source has numerous libraries, each with its own advantages for a particular application. I hope that the libraries discussed in this article will satisfy all or most of your requirements.