[Mulgara-general] Performance of the commit operation

[Chris Wilper]

Hi Paul and Andre,

Just to give some background: Fedora still goes through the Trippi
library to work with Mulgara.  Unless Andre has configured it
differently on his own, it is using Mulgara embedded mode.

We'd like to change this in the future so that it's more direct -- a
Fedora write transaction (which involves Mulgara updates, rdbms
updates, and filesystem writes) should ultimately be using a Mulgara
transaction for that portion of the work.  The Trippi layer buffers the
writes (adds and deletes of a big batch of triples), then sends them
to Mulgara when the buffer is full or it's being explicitly flushed.

The main reason for this buffering has always been to increase throughput.
Back when Mulgara was Kowari, we observed that there's a significant
cost to having small transactions vs. big ones, if you're just trying to
get things through as quickly as possible.

I mentioned we want to move away from Trippi in the future for
transaction reasons.  Note that I don't expect this to solve the
throughput issue.  When you have a lot of small (say, 20 to 40
add/deletes) in one transaction, we've observed that it's quite
a bit slower than putting a few thousand of these together in
one transaction.

- Chris

On Feb 5, 2008 1:09 PM, Paul Gearon <gearon at ieee.org> wrote:

> Errr, this is a very open ended question, so it's hard to answer.
>
> Yes, the timing your describing does sound very slow, and it should be
> possible to fix it, I'm just not sure where your problems are going to
> lie.
>
> For a start, I don't actually know how Fedora uses Mulgara.  I don't
> know if they've embedded the server, connecting to it with RMI, SOAP,
> or what.  I also don't know how transactions are handled - and this
> can have a big impact.
>
> Out of curiosity, are you able to run a server independently, and
> store data quickly enough, or is this slow as well?  If it's fast with
> Mulgara, then it's an issue with the Fedora interface (and I'll work
> with Fedora to see what we can do to help there).  Otherwise, then
> I'll try to help from the Mulgara end.
>
> Paul
>
> On Feb 5, 2008 11:33 AM, Schenk, Andre <Andre.Schenk at fiz-karlsruhe.de>
> wrote:
> > Hi all,
> >
> > we are implementing a framework based on Fedora (www.fedora.org). The
> newest Fedora version supports Mulgara so we just had to change some
> properties in the configuration to take advantage of it.
> >
> > But with Mulgara we have discovered a massive loss in write speed (about
> 5 times slower than before). When playing with the Mulgara ItqlBean we found
> that the commit operation needs a relatively constant time of about 0,5s -
> 1s on our dual Opteron server machine, even on an empty triple store.
> >
> > Now my question is: Are there any configuration properties which could
> give some extra speed? Is this by any chance a known problem which may be
> fixed in a future version?
> >
> > Thanks for your help!
> >
> > Best regards,
> >
> > André
> >
> >
> > -------------------------------------------------------
> >
> > Fachinformationszentrum Karlsruhe, Gesellschaft für
> wissenschaftlich-technische Information mbH.
> > Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim
> HRB 101892.
> > Geschäftsführerin: Sabine Brünger-Weilandt.
> > Vorsitzender des Aufsichtsrats: MinR Hermann Riehl.
> >
> >
> > _______________________________________________
> > Mulgara-general mailing list
> > Mulgara-general at mulgara.org
> > http://mulgara.org/mailman/listinfo/mulgara-general
> >
> _______________________________________________
> Mulgara-general mailing list
> Mulgara-general at mulgara.org
> http://mulgara.org/mailman/listinfo/mulgara-general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mulgara.org/pipermail/mulgara-general/attachments/20080205/a4312d34/attachment.htm>