Wednesday, June 22, 2016

Python Module: joblib - make parallelism easy!

This week, I found a nice python module to do quick parallel computing - joblib. I used to do parallel computing using python Multiprocessing module. But for a quick dirty way to parallel for loop, joblib is a very nice tool! Here's an example. You can download the example from Qingkai's Github.

Serial Version

In [1]:
def square_int(i):
    return i * i
In [2]:
results = []
for i in range(10):
    results.append(square_int(i))
print results
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Parallel Version

In [3]:
from joblib import Parallel, delayed

results = []
results = Parallel(n_jobs= -1, backend="threading")\
    (delayed(square_int)(i) for i in range(10))

print results
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
We can see that square_int is the function that you want to run in the for loop. The main restriction is that myfun must be a toplevel function. The backend parameter can be either "threading" or "multiprocessing". If you choose "multiprocessing", under the hood, the Parallel object create a multiprocessing pool that forks separate Python worker processes to execute tasks concurrently on separate CPUs.
If you know that the function you are calling is based on a compiled extension that releases the Python Global Interpreter Lock (GIL) during most of its computation then it might be more efficient to use threads instead of Python processes as concurrent workers.

4 comments:

  1. Serial Version execution time is less than to Parallel Version.
    Serial Version execution time:
    real 0m0.287s
    user 0m0.579s
    sys 0m6.112s

    Parallel Version execution time:

    real 0m0.460s
    user 0m0.614s
    sys 0m6.084s

    ReplyDelete
  2. @Daoud This is just an example to show how it works. The running time for parellel execution is more because the time to spawn thread/process is more than the function execution time. You should optimize using Parellel only when your function execution time is large, like >1s.

    ReplyDelete
  3. The information which you have provided in this blog is really useful to everyone. Thanks for sharing.
    Python Online Training in Hyderabad

    Python Institute in Hyderabad

    Python Course in Hyderabad

    ReplyDelete
  4. Five weeks ago my boyfriend broke up with me. It all started when i went to summer camp i was trying to contact him but it was not going through. So when I came back from camp I saw him with a young lady kissing in his bed room, I was frustrated and it gave me a sleepless night. I thought he will come back to apologies but he didn't come for almost three week i was really hurt but i thank Dr.Azuka for all he did i met Dr.Azuka during my search at the internet i decided to contact him on his email dr.azukasolutionhome@gmail.com he brought my boyfriend back to me just within 48 hours i am really happy. What’s app contact : +44 7520 636249‬

    ReplyDelete