2009/11/7 Antoine Pitrou <solipsis@[...].net> 

> 
>  [...]
>  So, to sum it up, the way the current GIL manages to have good latencies is
>  by
>  issueing an unreasonable number of system calls on a contended lock, and
>  potentially killing throughput performance (this depends on the OS too,
>  because
>  numbers under Linux are not so catastrophic).
> 
Ah, I remember reading this in the analysis that was published now!

I made another benchmark using one of my script I ported to python 3 (and
simplified a bit). I only test the total execution time. Tests done on
Windows XP SP3. Processor is an Intel Core 2 Quad Q9300 (4 cores).

You can get the script from:
http://gaiacrtn.free.fr/py/benchmark-kwd-newgil/purekeyword-py26-3k.py
Script + test doc (940KB):
http://gaiacrtn.free.fr/py/benchmark-kwd-newgil/benchmark-kwd-newgil.tar.bz2

The threaded loop is:
for match in self.punctuation_pattern.finditer( document ):
    word = document[last_start_index:match.start()]
    if len(word) >  1 and len(word) < MAX_KEYWORD_LENGTH:
        words[word] = words.get(word, 0) + 1
    last_start_index = match.end()
    if word:
        word_count += 1

Here are the results:

-j0 (main thread only)
2.5.2 : 17.991s, 17.947s, 17.780s
2.6.2 : 19.071s, 19.023s, 19.054s
3.1.1 : 46.384s, 46.321s, 46.425s
newgil: 47.483s, 47.605s, 47.512s

-j4 (4 consumer threads, main thread producing/waiting)
2.5.2 : 31.105s, 30.568s
2.6.2 : 31.550s, 30.599s
3.1.1 : 85.114s, 85.185s
newgil: 48.428s, 49.217s

It shows that, on my platform for this specific benchmark:

   -  newgil manage to leverage a significant amount of parallelism (1.7)
   where python 3.1 does not (3.1 is 80% slower)
   - newgil as a small impact on non multi-threaded execution (~1-2%) [may
   be worth investigating]
   - 3.1 is more than 2 times slower than python 2.6 on this benchmark
   - 2.6 is "only" 65% slower when run with multiple threads compared to the
   80% slower of 3.1.

Newgil is a vast improvement as it manages to leverage the short time the
GIL is released by finditer [if I understood correctly in 3.x regex release
the GIL].

What's worry me is the single threaded performance degradation between 2.6
and 3.1 on this test. Could the added GIL release/acquire on each finditer
call explain this?