[C++-sig] Embedding Python, threading and scalability
by Qiu, Wenning other posts by this author
Jun 27 2003 4:10PM messages near this date
[C++-sig] [ANN] SciPy '03 -- The 2nd Annual Python for Scientific Computing Workshop
|
[C++-sig] Re: Embedding Python, threading and scalability
------_=_NextPart_002_01C33CC6.A056795E
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
I am researching issues related to emdedding Python in C++ for a =
project.=20
My project will be running on an SMP box and requires scalability. =
However, my test shows that Python threading has very poor performance =
in terms of scaling. In fact it doesn't scale at all.=20
I wrote a simple test program to complete given number of iterations of =
a simple loop. The total number of iterations can be divided evenly =
among a number of threads. My test shows that as the number of threads =
grows, the CPU usage grows and the response time gets longer. For =
example, to complete the same amount of work, one thread takes 10 =
seconds, 2 threads take 20 seconds and 3 threads take 30 seconds.
The fundamental reason for lacking scalability is that Python uses a =
global interpreter lock for thread safety. That global lock must be held =
by a thread before it can safely access Python objects.=20
I thought I might be able to make embedded Python scalable by embedding =
multiple interpreters and have them run independently in different =
threads. However "Python/C API Reference Manual" chapter 8 says that =
"The global interpreter lock is also shared by all threads, regardless =
of to which interpreter they belong". Therefore with current =
implementation, even multiple interpreters do not provide scalability.
Has anyone on this list run into the same problem that I have, or does =
anyone know of any plan of totally insulating multiple embedded Python =
interpreters? =20
I've attached my test script to this message. I am using Python 2.2.3. =
<<mytest.py> >=20
------_=_NextPart_002_01C33CC6.A056795E
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.0.6249.1">
<TITLE> Embedding Python, threading and scalability</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<BR>
<P> <FONT SIZE=3D2 FACE=3D"Arial">I am researching issues related to =
emdedding Python in C++ for a project. </FONT>
</P>
<P> <FONT SIZE=3D2 FACE=3D"Arial">My project will be running on an SMP =
box and requires scalability. However, my test shows that Python =
threading has very poor performance in terms of scaling. In fact =
it doesn't scale at all. </FONT> </P>
<P> <FONT SIZE=3D2 FACE=3D"Arial">I wrote a simple test program to =
complete given number of iterations of a simple loop. The total number =
of iterations can be divided evenly among a number of threads. My test =
shows that as the number of threads grows, the CPU usage grows and the =
response time gets longer. For example, to complete the same amount of =
work, one thread takes 10 seconds, 2 threads take 20 seconds and 3 =
threads take 30 seconds.</FONT> </P>
<P> <FONT SIZE=3D2 FACE=3D"Arial">The fundamental reason for lacking =
scalability is that Python uses a global interpreter lock for thread =
safety. That global lock must be held by a thread before it can safely =
access Python objects. </FONT> </P>
<P> <FONT SIZE=3D2 FACE=3D"Arial">I thought I might be able to make =
embedded Python scalable by embedding multiple interpreters and have =
them run independently in different threads. However "Python/C API =
Reference Manual" chapter 8 says that "The global interpreter =
lock is also shared by all threads, regardless of to which interpreter =
they belong". Therefore with current implementation, even multiple =
interpreters do not provide scalability.</FONT> </P>
<P> <FONT SIZE=3D2 FACE=3D"Arial">Has anyone on this list run into the =
same problem that I have, or does anyone know of any plan of totally =
insulating multiple embedded Python interpreters? </FONT> </P>
<P> <FONT SIZE=3D2 FACE=3D"Arial">I've attached my test script to this =
message. I am using Python 2.2.3.<FONT FACE=3D"Arial" SIZE=3D2 =
COLOR=3D"#000000"> <<mytest.py>> </FONT></FONT>
</P>
</BODY>
</HTML>
------_=_NextPart_002_01C33CC6.A056795E--
Attachments:
mytest.py
Thread:
Qiu, Wenning
David Abrahams
|
