[Mulgara-dev] Size of server1 directory

[Paul Gearon]

Yes, the problem is due to unclosed Answer objects.

You say that you're using SOAP?  So that means all your results are
coming back to you as XML?  I thought we had closed everything after
serialization.  I must be wrong.  :-(

Paul

On 7/30/07, David Moll <DMoll at myaperio.com> wrote:
> Paul,
>
> I don't believe we have a method of manually closing Answers, as all of
> our interaction with Mulgara is through the SOAP interface.
>
> I have uploaded a new directory listing from this morning:
>
> ftp://viewpointusa.com/outgoing/Mulgara/dirlist_after_restart.txt
>
> So I can compare the directory size from before a restart (19 GB) and
> after a restart.  When we restart we wipe everything in Mulgara's
> server1 directory and restore from a backup.  Before running any queries
> the server1 directory is around 7.6 GB after restoring from the backup.
> This morning it was around 10 GB.
>
> The largest file size differences are in  /server1/xaStatementStore.
>
> It is difficult to do a real diff, so I'm just eyeballing file size
> differences.  I'll just list the biggest differences, all the
> differences are apparent when comparing directorylist.txt to
> dirlist_after_restart.txt (both in the above ftp folder).
>
> Before restart:
> 07/25/2007  11:25 AM     1,283,457,024 xa.g_0123_tb
>
> After restart:
> 07/30/2007  11:27 AM     1,031,798,784 xa.g_0123_tb
>
> Before restart:
> 07/25/2007  11:59 AM       360,366,080 xa.g_1203
> 07/25/2007  01:17 PM        29,360,128 xa.g_1203_fl
> 07/20/2007  02:40 AM        16,777,216 xa.g_1203_fl_ph
> 07/25/2007  12:16 PM     3,682,598,912 xa.g_1203_tb
> 07/25/2007  01:17 PM         3,145,728 xa.g_1203_tb_fl
> 07/15/2007  06:22 PM         8,388,608 xa.g_1203_tb_fl_ph
>
> After restart:
> 07/30/2007  11:27 AM        92,192,768 xa.g_1203
> 07/30/2007  11:51 AM        11,534,336 xa.g_1203_fl
> 07/27/2007  04:36 PM         8,388,608 xa.g_1203_fl_ph
> 07/30/2007  11:43 AM     1,535,115,264 xa.g_1203_tb
> 07/30/2007  11:51 AM         1,048,576 xa.g_1203_tb_fl
> 07/27/2007  04:36 PM         8,388,608 xa.g_1203_tb_fl_ph
>
> Total /server1/xaStatementStore:
> Before:  38 File(s)  8,761,556,704 bytes
> After:  38 File(s) 17,182,793,440 bytes
>
> ~Dave
>
> -----Original Message-----
> From: gearon at gmail.com [mailto:gearon at gmail.com] On Behalf Of Paul
> Gearon
> Sent: Monday, July 30, 2007 10:54 AM
> To: David Moll
> Cc: mulgara-dev at mulgara.org
> Subject: Re: [Mulgara-dev] Size of server1 directory
>
> Hi David,
>
> If shutting down clears some of this away then I have 2 ideas.
>
> The first (and most obvious) is that you have some open Answers
> somewhere.  Is it possible that you are doing queries, and not closing
> the results, while you do more writes?  A related problem is just
> having long-running transactions.
>
> There is another possibility, but I'd need to spend some time with the
> code to confirm this... We have a set of files that map internal block
> IDs to the phase they were allocated in.  It works fine, but takes up
> far more space than it should.  I'm wondering if these are the
> culprit?  While the phases they refer to are no longer valid when you
> restart, I didn't think these files were truncated.  (Meaning that it
> is probably not the issue you are seeing - but as I said, I'd have to
> check).
>
> Can you see exactly which files are the ones that grow, and are then
> truncated?  Each file name indicates the purpose of the file, and this
> would help work out the issue.
>
> Regards,
> Paul Gearon
>
>