[Mulgara-dev] [Ambra-dev] mulgara crash with java heap errors

[Life is hard, and then you die]

On Thu, Aug 28, 2008 at 10:55:10AM -0700, Russell Uman wrote:
> 
> one day after reporting strange heap OOM errors on our dev server, we 
> seem to have had a mulgara crash on our production server.
> 
> this is still on topaz 0.9-rc1 - we haven't changed anything in weeks.
> 
> from mulgara.log, many of these:
> 
> 2008-08-28 10:49:34,459 ERROR MulticastRMICacheManagerPeerProvider> 
> Unable to lookup remote cache peer for 
> //192.168.66.18:8111/RepresentationCache. Cause was not due to an 
> IOException or NotBoundException which will occur in normal operation: 
> Java heap space [pool-1-thread-92 
> net.sf.ehcache.distribution.MulticastRMICacheManagerPeerProvider]
> 2008-08-28 10:49:34,460 ERROR MulticastRMICacheManagerPeerProvider> 
> Unable to lookup remote cache peer for 
> //192.168.66.18:8111/permission-revokes. Cause was not due to an 
> IOException or NotBoundException which will occur in normal operation: 
> Java heap space [pool-1-thread-92
> net.sf.ehcache.distribution.MulticastRMICacheManagerPeerProvider]

Are you running jdk 1.5 or 1.6? In any case can you 

 - get the 'ps' output for the mulgara process (i.e. how much memory
   it's using)

 - try and get a heap dump via jmap:
     1.5:  jmap -heap:format=b <pid>
     1.6:  jmap -deump:live,format=b,file=/tmp/heap.bin <pid>

   (under 1.5 things may crap out or hang - under 1.6 jmap works
   reliably)

 - save the mulgara logs so we can take a closer look
 - shut down mulgara, make a copy of the db, then restart

(anybody else have any better ideas?)


  Cheers,

  Ronald