[Mulgara-general] Mulgara size limits
Chuck Borromeo
cborromeo3 at yahoo.com
Wed Jul 9 20:58:58 UTC 2008
Hi Alex,
I pulled the latest version of Mulgara from svn and built it. I ran it using Java 1.6 (as per Ronald's suggestion). Now the files load. It seems to take a long time to load these files. It takes about 40 minutes to load a 97 MB RDF/XML file. I'll try to put together a more formal measurement of the loading times.
I'll probably keep taxing the system. I've loaded about 152 GB of data into the server1 instance on our development server. I still have another 1 GB of RDF/XML files to load. Please let me know if there are any theoretical limits regarding size. I'm running the Mulgara server using the 64 bit Java switch on a 64 bit Mac OS X server.
Thanks for your time and efforts regarding this issue.
Chuck
--- On Tue, 7/8/08, Alex Hall <alexhall at revelytix.com> wrote:
> From: Alex Hall <alexhall at revelytix.com>
> Subject: Re: [Mulgara-general] Mulgara size limits
> To: cborromeo3 at yahoo.com, "Mulgara General" <mulgara-general at mulgara.org>
> Date: Tuesday, July 8, 2008, 4:36 PM
> Chuck Borromeo wrote:
> > Hi Everyone,
> > I just wanted to check to see if anyone has figured
> out the issue I was encountering when I tried to load
> RDF/XML files larger than 98 MB.
>
> I've been looking at this off and on since last week,
> but with little
> results to show. I can tell the offending class that is
> filling the
> heap, and I can tell where it is being allocated, but I
> haven't been
> able to track down where the references to the class are
> not being
> released. To add a degree of difficulty, I'm
> developing on a 32-bit
> Windows laptop so I can't duplicate the conditions
> causing the
> out-of-memory error.
>
> However, I do have a couple of suggestions for you to try:
>
> 1. Paul recently committed some changes that affect how
> objects are
> allocated. These were meant to simplify our code and make
> it easier to
> track object allocation, but there's an outside chance
> they might
> address your memory problem. Try building and running the
> latest
> revision from SVN (directions at the Mulgara Download page)
> and see if
> you still get out-of-memory errors.
>
> 2. I would also try running Mulgara with assertions
> enabled. Do this
> by adding the "-ea" flag to the Java VM
> arguments. Scan the log file or
> console for AssertionError stack traces, and let us know if
> you find
> anything.
>
> In both cases, I would also gather the heap profile, just
> to be sure
> that the AVLNode class is still the culprit.
>
> Beyond these two suggestions, I can't tell how the
> AVLNode references
> are not being released. The new Mulgara XA2 storage layer,
> planned for
> next year, will eliminate the use of the AVLNode class
> altogether, so
> this should not be an issue under the next-generation
> architecture.
> Given my limited time and resources, I just can't
> justify putting much
> more effort into trying to track this down when I'm
> unable to reproduce
> the problem myself. If either of the previous suggestions
> produces new
> results, I'll be happy to take a look at them, but
> absent any new
> developments I'm afraid there's not much more I can
> do at the moment.
>
> Regards,
> Alex
More information about the Mulgara-general
mailing list