[Mulgara-general] Mulgara size limits

Chuck Borromeo cborromeo3 at yahoo.com
Wed Jul 9 20:58:58 UTC 2008


Hi Alex,
  I pulled the latest version of Mulgara from svn and built it.  I ran it using Java 1.6 (as per Ronald's suggestion).  Now the files load. It seems to take a long time to load these files.  It takes about 40 minutes to load a 97 MB RDF/XML file.  I'll try to put together a more formal measurement of the loading times.

I'll probably keep taxing the system.  I've loaded about 152 GB of data into the server1 instance on our development server.  I still have another 1 GB of RDF/XML files to load.  Please let me know if there are any theoretical limits regarding size.  I'm running the Mulgara server using the 64 bit Java switch on a 64 bit Mac OS X server.

Thanks for your time and efforts regarding this issue.

Chuck


--- On Tue, 7/8/08, Alex Hall <alexhall at revelytix.com> wrote:

> From: Alex Hall <alexhall at revelytix.com>
> Subject: Re: [Mulgara-general] Mulgara size limits
> To: cborromeo3 at yahoo.com, "Mulgara General" <mulgara-general at mulgara.org>
> Date: Tuesday, July 8, 2008, 4:36 PM
> Chuck Borromeo wrote:
> > Hi Everyone,
> >   I just wanted to check to see if anyone has figured
> out the issue I was encountering when I tried to load
> RDF/XML files larger than 98 MB.
> 
> I've been looking at this off and on since last week,
> but with little 
> results to show.  I can tell the offending class that is
> filling the 
> heap, and I can tell where it is being allocated, but I
> haven't been 
> able to track down where the references to the class are
> not being 
> released.  To add a degree of difficulty, I'm
> developing on a 32-bit 
> Windows laptop so I can't duplicate the conditions
> causing the 
> out-of-memory error.
> 
> However, I do have a couple of suggestions for you to try:
> 
> 1.  Paul recently committed some changes that affect how
> objects are 
> allocated.  These were meant to simplify our code and make
> it easier to 
> track object allocation, but there's an outside chance
> they might 
> address your memory problem.  Try building and running the
> latest 
> revision from SVN (directions at the Mulgara Download page)
> and see if 
> you still get out-of-memory errors.
> 
> 2.  I would also try running Mulgara with assertions
> enabled.  Do this 
> by adding the "-ea" flag to the Java VM
> arguments.  Scan the log file or 
> console for AssertionError stack traces, and let us know if
> you find 
> anything.
> 
> In both cases, I would also gather the heap profile, just
> to be sure 
> that the AVLNode class is still the culprit.
> 
> Beyond these two suggestions, I can't tell how the
> AVLNode references 
> are not being released.  The new Mulgara XA2 storage layer,
> planned for 
> next year, will eliminate the use of the AVLNode class
> altogether, so 
> this should not be an issue under the next-generation
> architecture. 
> Given my limited time and resources, I just can't
> justify putting much 
> more effort into trying to track this down when I'm
> unable to reproduce 
> the problem myself.  If either of the previous suggestions
> produces new 
> results, I'll be happy to take a look at them, but
> absent any new 
> developments I'm afraid there's not much more I can
> do at the moment.
> 
> Regards,
> Alex


      



More information about the Mulgara-general mailing list