[Mulgara-general] Memory error during rdf load

[Life is hard, and then you die]

On Thu, Aug 14, 2008 at 11:35:52AM -0700, Bill OConnor wrote:
> 
> There is no big hurry. DMOZ only has an outdated RDF formatted data set so
> N3 is not an easy option.

You could try converting using the RdfToTriplesStylesheet at
http://www.semanticplanet.com/library?pagename=Main.RdfToTriplesStylesheet
(download link at the very bottom)

Though I have no idea how well it'll handle an RDF/XML file of your
size.

[snip]
> Would it help to break the file down into more manageable chunks? (say 500
> meg?)

Quite likely, if that's possible.

> What are the limitations are far as the number of triples are concerned?
> (100k, millions etc)

It depends on the structure of your data and how long you're willing
to wait :-)  In the past I have loaded nearly 100M triples in 24h or
so.


  Cheers,

  Ronald


> On Thu, Aug 14, 2008 at 10:10 AM, Paul Gearon <gearon at ieee.org> wrote:
> 
> > Hi Bill,
> >
> > Well, the obvious answer is to go to 64 bit, but if you're really
> > against that then you can try a couple of things.
> >
> > First off, does DMOZ have an N3 format? That has different memory
> > requirements while loading, so it may do better.
> >
> > Another thing to try would be to set the heap size for Mulgara. That's
> > done with the -Xms (initial size) and -Xmx (maximum size) options on
> > the Java command line.
> >
> > Finally, I'm working on some tweaks to indexing that will make loading
> > a little more efficient. However, I won't be done with this for a few
> > days yet, and it won't be in a formal release for nearly 2 weeks. But
> > if you can hang on, then this should help 32 bit installations.
> >
> > Regards,
> > Paul Gearon
> >
> >
> > On Wed, Aug 13, 2008 at 12:59 PM, Bill OConnor <wtoconnor at gmail.com>
> > wrote:
> > > Hello,
> > >
> > > Any more ideas about how I might fix some of these problems?
> > >
> > > On Mon, Aug 11, 2008 at 11:00 AM, Bill OConnor <wtoconnor at gmail.com>
> > wrote:
> > >>
> > >> Hello,
> > >>
> > >> Thanks for responding.
> > >>
> > >> I am running a 32 bit version of Ubuntu on a AMD 64 bit processor. I
> > would
> > >> assume the JVM
> > >> that is running is 32 bit.  The 64 bit operating systems have given me
> > >> compatibility problems
> > >> in the past so I generally try to avoid it. The system has 1 GB physical
> > >> and 3 GB swap.