Saturday, December 3, 2016

Python: Parallel method in a class

We talked about a simple way to parallel your python code by using joblib in a former blog. Today, I want to use it to parallel a method in a class, but I encountered some problem. In this week's blog, I will show you how we can solve the problem by using the joblib (You can use python's multiprocessing as well). You can download the script at Qingkai's Github.  

Error! If we parallel directly

from joblib import Parallel, delayed
# let's re-write the exact function before into a class
class square_class:
    def square_int(self, i):
        return i * i
     
    def run(self, num):
        results = []
        results = Parallel(n_jobs= -1, backend="threading")\
            (delayed(self.square_int)(i) for i in num)
        print(results)
square_int = square_class()
square_int.run(num = range(10))
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-54-63d78c56401c> in <module>()
      1 square_int = square_class()
----> 2 square_int.run(num = range(10))


<ipython-input-53-c2a13f8e0fa7> in run(self, num)
      6     def run(self, num):
      7         results = []
----> 8         results = Parallel(n_jobs= -1, backend="threading")            (delayed(self.square_int)(i) for i in num)
      9         print(results)


/Users/qingkaikong/miniconda2/lib/python2.7/site-packages/joblib/parallel.pyc in __call__(self, iterable)
    798             # was dispatched. In particular this covers the edge
    799             # case of Parallel used with an exhausted iterator.
--> 800             while self.dispatch_one_batch(iterator):
    801                 self._iterating = True
    802             else:


/Users/qingkaikong/miniconda2/lib/python2.7/site-packages/joblib/parallel.pyc in dispatch_one_batch(self, iterator)
    651             batch_size = self.batch_size
    652         with self._lock:
--> 653             tasks = BatchedCalls(itertools.islice(iterator, batch_size))
    654             if not tasks:
    655                 # No more tasks available in the iterator: tell caller to stop.


/Users/qingkaikong/miniconda2/lib/python2.7/site-packages/joblib/parallel.pyc in __init__(self, iterator_slice)
     66 
     67     def __init__(self, iterator_slice):
---> 68         self.items = list(iterator_slice)
     69         self._size = len(self.items)
     70 


<ipython-input-53-c2a13f8e0fa7> in <genexpr>((i,))
      6     def run(self, num):
      7         results = []
----> 8         results = Parallel(n_jobs= -1, backend="threading")            (delayed(self.square_int)(i) for i in num)
      9         print(results)


/Users/qingkaikong/miniconda2/lib/python2.7/site-packages/joblib/parallel.pyc in delayed(function, check_pickle)
    156     # using with multiprocessing:
    157     if check_pickle:
--> 158         pickle.dumps(function)
    159 
    160     def delayed_function(*args, **kwargs):


/Users/qingkaikong/miniconda2/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)


TypeError: can't pickle instancemethod objects

Solution to the problem

Why we got the above error? 
joblib or Multiprocessing is based on pickling to pass functions around to achieve parallelization. In order to pickle the object, this object must capable of being referred to in the global context for the unpickle to be able to access it. The function we want to parallel above is not in global context, therefore, causing the error. 
Therefore, one solution I found online is to create a function outside the class to unpack the self from the arguments and calls the function again. Here's the example: 
def unwrap_self(arg, **kwarg):
    return square_class.square_int(*arg, **kwarg)

class square_class:
    def square_int(self, i):
        return i * i
     
    def run(self, num):
        results = []
        results = Parallel(n_jobs= -1, backend="threading")\
            (delayed(unwrap_self)(i) for i in zip([self]*len(num), num))
        print(results)
square_int = square_class()
square_int.run(num = range(10))
[0, 1, 4, 9, 16]
We can see from above, by simply write a function outside of the class, we can parallel the method in class as before. You may wonder what we passed to the function, if we print out the argument we passed to unwrap_self function, you will have a better understanding: 
num = range(10)
# note, I use a string "self" here to print it out
print(zip(["self"]*len(num), num))
[('self', 0), ('self', 1), ('self', 2), ('self', 3), ('self', 4), ('self', 5), ('self', 6), ('self', 7), ('self', 8), ('self', 9)]

26 comments:

  1. This article is very much helpful and i hope this will be an useful information for the needed one. Keep on updating these kinds of informative things...
    PHP training in chennai

    ReplyDelete
  2. Great post! I am see the programming coding and step by step execute the outputs.I am gather this coding more information. It's helpful for me my friend. Also great blog here with all of the valuable information you have.
    Python Training in Chennai

    ReplyDelete
  3. Nice blog, here I had an opportunity to learn something new in my field. I have an expectation about your future post so please keep updates...Thanks..



    Python Online Training

    ReplyDelete
  4. Thank you so much for sharing this worth able content with us. The concept taken here will be useful for my future programs and i will surely implement them in my study. Keep blogging article like this.

    python training in bangalore|

    ReplyDelete
  5. Existing without the answers to the difficulties you’ve sorted out through this guide is a critical case, as well as the kind which could have badly affected my entire career if I had not discovered your website.

    python training in bangalore|

    ReplyDelete
  6. You could just make square_int @staticmethod or @classmethod. And it'll be working like a charm.

    ReplyDelete
  7. Python has adopted as a language of choice for almost all the domain in IT including the most trending technologies such as Artificial Intelligence, Machine Learning, Data Science, Internet of Things (IoT), Cloud Computing technologies such as AWS, OpenStack, VMware, Google Cloud, etc.., Big Data Analytics, DevOps and Python is prepared language in traditional IT domain such as Web Application Development, Infrastructure Automation ,Software Testing, Mobile Testing.

    ReplyDelete
  8. Nice tutorial. Thanks for sharing the valuable information. it’s really helpful. Who want to learn this blog most helpful. Keep sharing on updated tutorials…
    Click here:
    Angularjs training in chennai

    Click here:
    angularjs training in bangalore

    Click here:
    angularjs training in online

    Click here:
    angularjs training in Annanagar

    ReplyDelete
  9. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Devops training in tambaram|Devops training in velachery|Devops training in annanagar|Devops training in sholinganallur

    ReplyDelete
  10. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    DevOps online Training|DevOps Training in USA
    Devops Training in Chennai

    Devops Training in Bangalore

    ReplyDelete
  11. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.


    rpa training in Chennai | rpa training in velachery

    rpa training in tambaram | rpa training in sholinganallur

    ReplyDelete
  12. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.


    rpa training in Chennai | rpa training in velachery

    rpa training in tambaram | rpa training in sholinganallur

    ReplyDelete
  13. I found this informative and interesting blog so i think so its very useful and knowledge able.I would like to thank you for the efforts you have made in writing this article.
    python training in tambaram
    python training in annanagar
    python training in OMR
    python training in chennai

    ReplyDelete
  14. I am so proud of you and your efforts and work make me realize that anything can be done with patience and sincerity. Well I am here to say that your work has inspired me without a doubt.
    java training in jayanagar | java training in electronic city

    java training in chennai | java training in USA

    ReplyDelete
  15. Best AngularJS Training in Bangalore offered by myTectra. India's No.1 AngularJS Training Institute. Classroom, Online and Corporate training in AngularJS.
    angularjs training in bangalore

    ReplyDelete
  16. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    rpa training in velachery| rpa training in tambaram |rpa training in sholinganallur | rpa training in annanagar| rpa training in kalyannagar

    ReplyDelete
  17. Is there a way to run an instance method that modifies the instance variables of a class using joblib?

    ReplyDelete
  18. Thanks for the good words! Really appreciated. Great post. I’ve been commenting a lot on a few blogs recently, but I hadn’t thought about my approach until you brought it up. 

    Data Science training in kalyan nagar | Data Science training in OMR
    Data Science training in chennai | Data science training in velachery
    Data science training in tambaram | Data science training in jaya nagar

    ReplyDelete
  19. Thanks for such a great article here. I was searching for something like this for quite a long time and at last I’ve found it on your blog. It was definitely interesting for me to read  about their market situation nowadays.
    java training in chennai | java training in bangalore

    java interview questions and answers | core java interview questions and answers

    ReplyDelete
  20. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    AWS Training in Toronto | Amazon Web Services Training in Toronto , Canada

    AWS Training in New york city | Amazon Web Services in New York City


    AWS Training in London | Amazon Web Services Training in London , UK

    ReplyDelete