[Mulgara-dev] RMI Out of Memory Exception in Mulgara

[Life is hard, and then you die]

On Fri, Jul 24, 2009 at 11:34:29AM -0500, Paul Gearon wrote:
> On Thu, Jul 23, 2009 at 12:23 PM, Richard Cave<rcave at plos.org> wrote:
> >
> > We encountered a RMI Out of Memory exception today (exception and dump
> > below).  Mulgara is set to use 1Gb memory:
> >
> > JAVA_OPTS="-Xmx1000m -XX:MaxPermSize=256m -Djava.rmi.dgc.leaseValue=30000"
> >
> > Do we need to increase the memory more?  Is the RMI heap space tied directly
> > to Mulgara or can we increase memory for RMI separately?
> 
> Increasing memory more will certainly help the current issue, but at
> the expense of performance.

Agreed. I don't know why it's even as high as 1GB - for a long time
400MB was more than sufficient, so this warrants more investigation.

> Unfortunately for this situation, RMI is handled as a normal library,
> meaning that it uses heap just like everything else. I note that the
> Topaz code makes heavy use of ehcache, which is also using RMI. I
> don't know for sure, but I'm suspicious that this library may be what
> is tipping the balance here. We haven't seen this type of OOM in
> systems without the Topaz resolvers, so this bears investigation.
[snip]

This is pure random speculation, and hence is likely to be wrong. Just
because RMI shows up in the OOM stack traces is no indication
whatsoever that RMI is involved in the memory issue - in general stack
traces and thread dumps are pretty useless in trying to track down
memory leaks because of the many allocations that happen in Java even
for simple operations (i.e.  everybody always allocating memory all
over the place, and hence it's quite random as to which thread will
happen to attempt to allocate the last remaining few bytes, and hence
more often than not that thread has nothing to do the memory leak).

Instead, get some actual heap dumps; if possible, monitor the heap
usage (actual heap usage, via jconsole, jvmstat, visualgc, etc, not
the process memory usage as indicated by ps, top, etc as that is
basically useless here), and then generate heap dumps when you see the
heap usage grow significantly.


  Cheers,

  Ronald