Toby Dickenson wrote:
> >Not malicious, consider this:
> >
> >for brain in context.some_catalog(an_index='fish'):
> >   object = brain.getObject()
> >   object.doSomeMaintenence()
> >
> >...now if an_index goes walkies, you've just loaded all the obejcts you've
> >cataloged into memory by mistake.
>  
>  Yes. IMO this script *needs*:
>  
>  1. either batching, as is traditional for a search results page. The search 
>  might be huge, but memory useage will be controlled if the batch size is 
>  reasonable:

This isn't a search results page, it's an object maintenance script. I find thsi 
pattern extremely useful.

>  2. or a garbage collection tickler as part of the loop. See ZCatalog's 
>  "reindex all" implementation for an example.

Can this tickling be done in a TTW script?

>  Probably not, but I dont do much ttw work.

Touché. Lots of people do, and they're the most likely to topple their servers 
this way. I've had just such a thing going on this morning, on 2.6.0, from 
someone accidentally doing objectValues or some such on a BTreeFolder that 
contains 6,000 odd big chunky objects.


> >how would raising a MemoryError (as happens when you run of RAM for real)
> >affect these?
>  
>  Not a bad idea. This could be done easily and efficiently in Connection.py

:-)

>  Of course it still doesnt help with the case of memory usage by non-persistent 
>  objects.

Well, a partial fix is better than no fix, and persistent objects are the things 
I'm worrying about.

> >>I can suggest a change to your indexing loop that will make this happen
> >>automatically. Would that help? ; -)
> >
> >What would the change be?
>  
>  sys.exit(1)

Hmmm... somewhat less re-rentrant than I'd hoped ;-)

Anyway, what can I do to help make the raising of a MemoryError a reality?

That said, I'd prefer to see code that dumps an old object out of the cache to 
bring the new one in rather than a MemoryError.

Sure, this could lead to processor loading, but that's better than 
machine-dying-due-to-running-out-of-RAM or confused end users being presented 
with MemoryErrors when the machine has plenty of memory left.

cheers,

Chris


_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@[...].org
http://lists.zope.org/mailman/listinfo/zodb-dev