Welcome, guest | Sign In | My Account | Store | Cart

Twisted FAQs clearly state that "deferreds do not magically convert blocking code into non-blocking code". So, how do you magically convert blocking code into non-blocking code?

This recipe is the solution!

Python, 47 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
  ## version 1.1, changed according to the suggestions in the comments

  from twisted.internet import reactor, defer
  from twisted.python import threadable; threadable.init(1)
  import sys, time

  ## the wrong way

  def callInThread(func, *args):
      """Takes a blocking function an converts it into a deferred-valued 
      function running in a separate thread.
      """
      de = defer.Deferred()
      de.addCallback(func)
      reactor.callInThread(de.callback, *args)
      return de
  
  deferred = callInThread.__get__ # decorator associated to callInThread

  # the right way
   
  from twisted.internet.threads import deferToThread
  deferred = deferToThread.__get__

  ## example code

  def print_(result):
    print result
  
  def running():
      "Prints a few dots on stdout while the reactor is running."
      sys.stdout.write("."); sys.stdout.flush()
      reactor.callLater(.1, running)

  @deferred
  def sleep(sec):
    "A blocking function magically converted in a non-blocking one."
    print 'start sleep %s' % sec
    time.sleep(sec)
    print '\nend sleep %s' % sec
    return "ok"
  
  if __name__ == "__main__":
     sleep(2).addBoth(print_)
     reactor.callLater(.1, running)
     reactor.callLater(3, reactor.stop)
     reactor.run() 

How to make blocking code non-blocking is the obvious question for everybody using Twisted, but the Twisted documentation does not make easy to find the solution :-(

The trick is to run the blocking function in a separate thread. Here all the magic is performed by the decorator, deferred, which converts sleep, a blocking function, into a deferred function i.e. a non-blocking function that returns a deferred object. The code for callInThread is clear, and the .__get__ trick converts callInThread in a one-argument function returning a closure, i.e. an object suitable to be used as a decorator. I have seen this trick in Alex Martelli's lectures at PyCon 2005. If you are confused by it, you should read Raymond Hettinger essay on descriptors (http://users.rcn.com/python/download/Descriptor.htm).

In short: every time you have a blocking function in your code, wrap it with the deferred decorator and live happy!

2 comments

Michele Simionato (author) 18 years, 8 months ago  # | flag

deferToThread. Valentino Volonghi points out to me that there is already a function in Twisted doing the same job as my 'callInThread', and doing it better: twisted.internet.threads.deferToThread.

So just import it and define the decorator as 'deferred=deferToThread.__get__'

Valentino also points out that threads are not scalable, but I bet you already knew that ;)

Itamar Shtull-Trauring 18 years, 8 months ago  # | flag

You don't want to use this code. Callbacks you add to the Deferred will run in the thread, but only sometimes. This makes thread safety really hard, possibly impossible and means lots of obscure hard to track down bugs.

Instead, use twisted.internet.threads.deferToThread, where callbacks added to result will run in the Twisted main thread.

Created by Michele Simionato on Wed, 17 Aug 2005 (PSF)
Python recipes (4591)
Michele Simionato's recipes (12)

Required Modules

  • (none specified)

Other Information and Tasks