Python line profiler and f2py

This example is prepared by Qingkai Kong (qingkai.kong@gmail.com) from Berkeley Seismological Lab for the lightning talk at The Hacker Within at BIDS on April 6th 2016. You can find the code on Qingkai's Github.

The purpose of this script is to show how I usually do to speedup my python script. And hope this is useful to you.

Line_profiler¶

Many times, you find your python script is slow, but you just don't know which part drags the whole performance down. To get an idea, I usually use the line_profiler from Robert Kern, this is really good if you want to identify which line uses more time than you expceted, and how often each line executed. Here's a blog talking about profile python Marco Bonzanini
You can use line_profiler either in command line or in ipython notebook.

1 Command line¶

In a typical workflow, one only cares about line timings of a few functions because wading through the results of timing every single line of code would be overwhelming. However, LineProfiler does need to be explicitly told what functions to profile. The easiest way to get started is to use the kernprof script.
Steps:

1. in your script, you decorate the functions you want to profile with @profile. For example:

   @profile
   def function_to_profile(a, b, c):
    ...
   

2. Run the following command in the terminal:

   kernprof -v -l profile_test.py
   

2 Run in Ipython notebook¶

To run line_profiler in the notebook, you need load the extension first, and then use the magic commands %lprun to profile the script.

In [1]:

%load_ext line_profiler

In [2]:

def example_function(myRange):
    # directly convert range to string list
    str_list = []
    for i in myRange:
        str_list.append(str(i))
        
def example_function2(myRange):
    # use list comprehension to convert range to string list
    str_list = [str(i) for i in myRange] 
        

In [3]:

%lprun -f example_function example_function(range(1000000))

In [4]:

%lprun -f example_function2 example_function2(range(1000000))

Using f2py¶

f2py - Fortran to Python interface generator, is used to call Fortran 77/90/95 external subroutines and Fortran 90/95 module subroutines as well as C functions. It is part of the Numpy now. You can find more details here.
Let's grab the example from the above link, and compare a python version and a fortran version for speed.

Python version¶

In [5]:

import numpy as np

def fib(A):
    '''
    CALCULATE FIRST N FIBONACCI NUMBERS
    '''
    n = len(A)
    
    for i in range(n):
        if i == 0:
            A[i] = 0.
        elif i == 1:
            A[i] = 1.
        else:
            A[i] = A[i-1] + A[i-2]
            
    return A

In [6]:

dat_in = np.zeros(10) 
dat_out = fib(dat_in)
dat_out

Out[6]:

array([  0.,   1.,   1.,   2.,   3.,   5.,   8.,  13.,  21.,  34.])

In [7]:

dat_in = np.zeros(1000) 

In [8]:

%lprun -f fib fib(dat_in)

Fortran version¶

let's first write a simple fortran subroutine

In [9]:

!ls

Profile python script and f2py.ipynb example.py

README.md

In [10]:

%%writefile fib1.f
C FILE: FIB1.F
      SUBROUTINE FIB(A,N)
C
C     CALCULATE FIRST N FIBONACCI NUMBERS
C
      INTEGER N
      REAL*8 A(N)
      DO I=1,N
         IF (I.EQ.1) THEN
            A(I) = 0.0D0
         ELSEIF (I.EQ.2) THEN
            A(I) = 1.0D0
         ELSE 
            A(I) = A(I-1) + A(I-2)
         ENDIF
      ENDDO
      END
C END FILE FIB1.F

Writing fib1.f

In [11]:

!f2py -c fib1.f -m fib1

In [12]:

!ls

Profile python script and f2py.ipynb fib1.f

README.md                            fib1.so

example.py                           fib1.so.dSYM

You can see it created an python interface fib1.so. Now you can import the function into python.

In [13]:

import fib1
import numpy as np

In [14]:

print fib1.fib.__doc__

fib(a,[n])

Wrapper for ``fib``.

Parameters
----------
a : input rank-1 array('d') with bounds (n)

Other Parameters
----------------
n : input int, optional
    Default: len(a)

In [15]:

a = np.zeros(9)

In [16]:

fib1.fib(a)

In [17]:

a

Out[17]:

array([  0.,   1.,   1.,   2.,   3.,   5.,   8.,  13.,  21.])

Let's compare the time of both function, we can see for this exmaple, the fortran version is about 100 times faster than the python version.

In [18]:

a = np.zeros(1000)

In [19]:

%timeit fib1.fib(a)

100000 loops, best of 3: 2.66 µs per loop

In [20]:

%timeit fib(a)

1000 loops, best of 3: 363 µs per loop

My workflow¶

When I start some script, I usually use the following workflow to speedup:

1. write python script
2. profile it line by line
3. finding some stupid thing that can be easily fix (I found this many times)
4. parallel the part that used a lot of time
5. use a f2py to take advantage of the fotran speed