[Mulgara-dev] trouble with large models

[Paul Gearon]

Hi David,

As you can see, I've been really bad on Mulgara issues lately. Work pulled
me off Semantic Web stuff, and I've been putting my open source time into
trying to duplicate a lot of Mulgara using Clojure.

That said, I don't plan to let Mulgara die, so I apologise for not being
more responsive.

I can't say that another debug level will necessarily show you adequate
information. It will depend on the specific code as to whether or not it
will be adequately logged.

One recent change that I've made is to put the source code on Github. It
doesn't have the log history (that's still available, but not on Github)
but I think that's a better way to coordinate with anyone who wants to
contribute. If you do a pull request I'll be sure to bring in anything you
have.

As for your specific issue.... I haven't seen that behavior myself.
Flushing buffers makes me wonder about the heap/RAM tradeoff (more memory
in the heap makes less available for memory mapping files). If you haven't
looked at that, then try making your heap a little smaller and see if that
changes any observed behavior. Otherwise, do you have a stack trace of when
the flushes happen?

Paul

On Sun, Jul 20, 2014 at 2:10 PM, David Smith <DMS at viewpointusa.com> wrote:

> I'm currently running into trouble with a very large Mulgara model  (with
> string pool and nodes files larger than 4GB) under windows (server 2008 r2)
>
> The application is currently running under Mulgara 2.1.8, but I am see
> similar problems with 2.1.13 (even after fixing a bug seeing below)
>
> The main symptom is very slow performance... especially startup, but even
> after it has started up it will pause on occasion, with the same
> symptoms... using mapped files.
> During startup, mulgara is issuing continuous "FlushBuffersFiles" calls
> ... going through all the xa.g_XXXX_tb  and xa11.sp_avl files.
> I don't see this when using "smaller" models.
> My guess is that I'm looking for another int vs long issue.
>
> The system will run, but with a different level of degraded performance,
> if I set the IO to explicit.
>
> If anyone has any suggestions on where to look in the code or what log4j
> categories to turn on, I would appreciate it!
> I can try and track it down from there.
>
> Dave Smith
> Viewpoint Systems, Inc.
>
> ==
>
> I did find, and fix a bug in 2.1.13 related to large files... this bug
> manifested as an IO exception : trying to seek to a negative offset....
>
> In src\jar\util\java\org\mulgara\util\io\LMappedBufferedFile.java,
>  In truncate and mapFile
>  lines  152, 212, 215, the second parameter to the fc.map call needs to be
> performed as a long operation
>  my fix is to use a long for the pageSize local variable (lines 113, 185)
> I'll submit more formally after I track down my issue above.
>
> ==
>
>
> As an FYI:
>
> Directory of D:\Mulgara\server1\xaStatementStore
>
> 07/20/2014  01:27 PM    <DIR>          .
> 07/20/2014  01:27 PM    <DIR>          ..
> 07/20/2014  01:27 PM         8,388,608          xa.g
> 07/20/2014  01:27 PM                 0          xa.g.lock
> 07/20/2014  01:27 PM        83,812,352          xa.g_3012
> 07/20/2014  01:27 PM         8,388,608          xa.g_3012_fl
> 07/20/2014  01:27 PM       134,217,728          xa.g_3012_fl_ph
> 07/20/2014  01:27 PM     6,509,559,808          xa.g_3012_tb
> 07/20/2014  01:27 PM         8,388,608           xa.g_3012_tb_fl
> 07/20/2014  01:27 PM       134,217,728          xa.g_3012_tb_fl_ph
> 07/20/2014  01:27 PM       150,855,680          xa.g_3120
> 07/20/2014  01:27 PM         8,388,608          xa.g_3120_fl
> 07/20/2014  01:27 PM       134,217,728          xa.g_3120_fl_ph
> 07/20/2014  01:27 PM    12,314,476,544  xa.g_3120_tb
> 07/20/2014  01:27 PM         8,388,608          xa.g_3120_tb_fl
> 07/20/2014  01:27 PM       134,217,728          xa.g_3120_tb_fl_ph
> 07/20/2014  01:27 PM       142,475,264          xa.g_3201
> 07/20/2014  01:27 PM         8,388,608          xa.g_3201_fl
> 07/20/2014  01:27 PM       134,217,728          xa.g_3201_fl_ph
> 07/20/2014  01:27 PM    12,096,372,736  xa.g_3201_tb
> 07/20/2014  01:27 PM         8,388,608          xa.g_3201_tb_fl
> 07/20/2014  01:27 PM       134,217,728          xa.g_3201_tb_fl_ph
>               20 File(s) 32,161,579,008 bytes
>
>  Directory of D:\Mulgara\server1\xaStringPool
>
> 07/20/2014  01:27 PM    <DIR>          .
> 07/20/2014  01:27 PM    <DIR>          ..
> 07/20/2014  01:27 PM         8,388,608          xa11.sp
> 07/20/2014  01:27 PM                 0          xa11.sp.lock
> 07/20/2014  01:27 PM     4,760,084,480          xa11.sp_avl
> 07/20/2014  01:27 PM         8,388,608          xa11.sp_avl_fl
> 07/20/2014  01:27 PM       268,435,456          xa11.sp_avl_fl_ph
> 07/16/2014  06:14 PM     3,623,195,416          xa11.sp_nd
>                6 File(s)  8,668,492,568 bytes
>
> This E-Mail and any attachments thereto may contain confidential,
> privileged and ITAR controlled information. It is intended solely for the
> recipient(s) indicated. Any review, use, or distribution by anyone other
> than the intended recipient(s) is strictly prohibited. If you have received
> this E-Mail in error or are not the intended recipient, please notify the
> sender and delete all copies immediately.
> _______________________________________________
> Mulgara-dev mailing list
> Mulgara-dev at mulgara.org
> http://lists.mulgara.org/mailman/listinfo/mulgara-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mulgara.org/pipermail/mulgara-dev/attachments/20140721/d27a5188/attachment.html>