[Mulgara-general] size of XML-literals

Paul Gearon gearon at ieee.org
Fri Jun 15 22:11:46 UTC 2007


You just need the correct datatype on your literal.  If you just want to use
an XSD datatype, then there's hexBinary and base64Binary.  Otherwise, we
have user-defined datatypes as well, but I don't know that you'd gain much.

I just looked it up.  We can't store 100MB.  Once upon a time we could.
Sorry about that!  (Actually, all we'd have to do would be to change a
single number, and we'd be able to again.  But we don't believe there's a
lot of point to it)

We can store data of up to 16777288 bytes (16MB + 72 bytes), and we can only
store 128 of them.  To give you the exact details (add 72 bytes to each of
the following sizes):

128 slots for data between 8MB and 16MB.
256 slots for data from 4MB to 8MB.
512 slots for data of 2MB to 4MB
1024 slots for data of 1MB to 2MB
2048 slots for data of 512kb to 1MB
1024 slots for data of 256kb to 512kb
.
.
.
67108864 slots for data of 16bytes (+72) to 32bytes (+72)
134217728 slots for data of 72bytes to 16bytes+72bytes

and finally......
128102389400760774 slots for data of less than 72 bytes

Gotta love that last one.  :-)

We use UTF-8, so most URIs fit into 72 bytes.

These limits are because we're using 32bit file access for this data, and
all the data of a particular size ends up in one file.  It wouldn't be a big
deal to change it to 64 bit.  We could also use multiple files.  Neither
have been necessary before now.  It can be updated, but if you need really
large data objects like this, then perhaps you should be storing URLs to the
data instead?

Also, the list can be longer, just by changing the value of:
 org.mulgara.store.stringpool.xa.XAStringPoolImpl.NR_BLOCK_FILES

It's currently set to 20.  The maximum it can be is 27.  This would extend
the list up to "1 slot for data of 1GB to 2GB".  (which is pretty
pointless).


Regards,
Paul

On 6/15/07, thomas <thomas at stray.net> wrote:
>
> 100 mb is a lot. that spurs another question: is it a) legal in rdf-land
> and/or b) possible with mulgara to put binary data like images in the
> object position?
>
> ciao
> thomas
>
>
>
> --On 15. Juni 2007 16:22:43 -0500 Paul Gearon <gearon at ieee.org> wrote:
>
> > I can't recall how XML gets stored, but I think it's just as a string
> > (like
> > an untyped literal).  I believe we handle 100MB, so that should be fine.
> > :-)
> >
> > Regards,
> > Paul
> >
> > On 6/15/07, thomas <thomas at stray.net> wrote:
> >>
> >> hi,
> >>
> >> i'd like to add XML-literals as objects to the store (among other
> reasons
> >> because i want to benefit from the fulltext search). is there a limit
> in
> >> size of those XML-fragments that's supported or reasonable? would 100
> kb
> >> be
> >> okay? would 1 mb be way out of the limits?
> >>
> >> thanks,
> >> thomas
> >>
> >>
> >> mailto:thomas at stray.net
> >> http://stray.net
> >>
> >> _______________________________________________
> >> Mulgara-general mailing list
> >> Mulgara-general at mulgara.org
> >> http://mulgara.org/mailman/listinfo/mulgara-general
> >>
>
>
>
> mailto:thomas at stray.net
> http://stray.net
>
>
>
>
> : accumulated wisdom
> . early optimization is the root of many evil [donald e. knuth]
> . if you've got a hammer every problem looks like a nail
> . the difference between theory and practice is always greater
>   in practice than it is in theory
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mulgara.org/pipermail/mulgara-general/attachments/20070615/d3a35c8e/attachment.htm>


More information about the Mulgara-general mailing list