Alexandre Ferrieux wrote:
 >  Hello again,
 > 
 >  (I'm not posting this to SF because it has now become so slow it's 
unusable)
 > 
 >  I think I can now produce a global analysis of the pipe-leak issue.
 > 
 >  (1) This is a core issue, Tclx is a red herring here, it is only one
 >  of many ways of exhibiting the problem;
 > 
 >  (2) A core-only way is:
 > 
 >    # this leaks 2M per run in Windows
 >    # assuming "block.tcl" contains just "after 5000"
 >    set pr [open "|tclsh block.tcl" r];after 500;fconfigure $pr
 >  -blocking 0;read $pr 1;close $pr


Yes, that looks accurate.


 >  (3) The crux of the problem is the (now possiby obsolete)
 >  PipeReaderThread being blocked on a ReadFile at [close]-time, implying
 >  the use of the thermonuclear TerminateThread, which is known to have a
 >  4k kernel leak in Windows.


Yes, exactly.  That 4K was from a global pool that just kept getting 
smaller over time.  Only rebooting would bring it back.


 >  (4) But in today's Windows (XPSP2), it leaks 2M, not 4k !!!
 > 
 >  I don't know at which point in time the x500 amplification of the 
leak occured.
 >  Notice that the situation producing it is rather unusual (it assumes a
 >  long-running child which we close prematurely from the output side,
 >  and only on  Windows where those tools (like vmstat) are not common).
 > 
 >  Now there are IMO two parallel paths to follow simultaneously:
 > 
 >    (P1) investigate to understand the large leak
 > 
 >    (P2) (longer term) modernize the core to use waitable handles
 >  (assuming they are really !), thus avoiding the need of ancillary
 >  threads.
 > 
 >  I would love to hear comments on this. I feel a bit shaky on Windows
 >  to undertake (P2); and possibly a person who is literate in Windows
 >  patch history migh home in on (P1) much fasterthan I would...

About (P1), I barely remember when I was looking at the problem back in 
2000.  I tried a memory trace debugger, but didn't have good luck with 
it.  I had to imply the effect, but now if you can see it with a simple 
PSapi tool, something is different now.

In all, it really just comes down to the use of the very evil 
TerminateThread().  It would be so much easier to fix if something 
simple like a call to CloseHandle() would interrupt ReadFile() or some 
other measure, but I don't know if such an attempt could pop it loose. 
I'm sure I tried CloseHandle() and a re-wait on the exit status in place 
of the TerminateThread(), but it would be worth a second try.

I feel (P2) is the right way to go.  One of the first things you'll 
notice is that you don't need two additional threads per process.  What 
you will need is a generalized notification api that uses thread pools 
to act as the event notification mechanism.  Each thread can wait on 64 
handles.  It can be any waitable handle type, not just pipes (sockets, 
console, mailslots, etc..).  The first handle should be a stop event, so 
that leaves 63 usable slots per thread.  To avoid resource starvation, 
the list should be rotated after a wait is signaled.  The new 
notification api should also allow the replacement of NT-only completion 
ports instead of events when appropriate at run-time.

I came close to starting that a number of years ago for my IOCPSOCK 
project, but never did.

As I'm extremely out-of-practice with programming these days I can't be 
much help with the implementation, unfortunately.  I'd need a copy of 
Visual C++ if someone wants to donate... I like its debugger.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
Tcl-Core mailing list
Tcl-Core@[...].net
https://lists.sourceforge.net/lists/listinfo/tcl-core