[Mulgara-general] Recommendations for batch loading

Paul Gearon gearon at ieee.org
Thu Sep 14 16:14:04 UTC 2006


Hi Eric,

Make sure that you have autocommit set to off during your load, as
this will slow you down significantly.

That said, I confess that I've had some slow loads recently as well.
I suspect that something has been introduced which is slowing us down,
but I don't know what yet.

My biggest problem in this regard is that I have so many Mulgara tasks
which are high priority that important things like this are being left
behind.  (General call for help here).

The two things to track down the problem are:
- Profiling.  This probably requires knowledge of the code base to be
really useful.
- Testing old Kowari builds to find out when the problem was
introduced.  This will be tedious, but easy.

Kowari used to load significantly larger data sets without breaking a
sweat, so it's very important we understand when these slow loads were
introduced.  Unfortunately, it's not an easy thing to test for when
modifying code, as small loads are fast no matter what overhead you
have.

Regards,
Paul Gearon

On 9/14/06, Eric Jain <Eric.Jain at isb-sib.ch> wrote:
> I'm just giving Mulgara another try -- this time on a machine with 64-bit
> support.
>
> Loading a 100MB file with 1.5M triples worked fine, the import was complete
> after 3.5 min.
>
> Now I'm trying to load an 8GB file. The import has been running for almost
> a day now, and what worries me is that Mulgara's data directory has only
> grown to about 14GB in this time. Moreover the CPU load is rather low, even
> though the machine has only one (hyperthreaded) CPU:
>
>    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
> 21944 ejain     16   0 16.1g 1.5g 1.4g S    7 78.8  96:42.30 java
>
>
> The procedure was initiated through the web-based iTQL shell (using the
> LOAD command), maybe there is a better way? I'm also using default settings
> pretty much everywhere.
> _______________________________________________
> Mulgara-general mailing list
> Mulgara-general at mulgara.org
> http://mulgara.org/mailman/listinfo/mulgara-general
>



More information about the Mulgara-general mailing list