[Mulgara-dev] Size of server1 directory

[David Moll]

Paul,

I don't believe we have a method of manually closing Answers, as all of
our interaction with Mulgara is through the SOAP interface.

I have uploaded a new directory listing from this morning:

ftp://viewpointusa.com/outgoing/Mulgara/dirlist_after_restart.txt

So I can compare the directory size from before a restart (19 GB) and
after a restart.  When we restart we wipe everything in Mulgara's
server1 directory and restore from a backup.  Before running any queries
the server1 directory is around 7.6 GB after restoring from the backup.
This morning it was around 10 GB.

The largest file size differences are in  /server1/xaStatementStore.

It is difficult to do a real diff, so I'm just eyeballing file size
differences.  I'll just list the biggest differences, all the
differences are apparent when comparing directorylist.txt to
dirlist_after_restart.txt (both in the above ftp folder).

Before restart:
07/25/2007  11:25 AM     1,283,457,024 xa.g_0123_tb

After restart:
07/30/2007  11:27 AM     1,031,798,784 xa.g_0123_tb

Before restart:
07/25/2007  11:59 AM       360,366,080 xa.g_1203
07/25/2007  01:17 PM        29,360,128 xa.g_1203_fl
07/20/2007  02:40 AM        16,777,216 xa.g_1203_fl_ph
07/25/2007  12:16 PM     3,682,598,912 xa.g_1203_tb
07/25/2007  01:17 PM         3,145,728 xa.g_1203_tb_fl
07/15/2007  06:22 PM         8,388,608 xa.g_1203_tb_fl_ph

After restart:
07/30/2007  11:27 AM        92,192,768 xa.g_1203
07/30/2007  11:51 AM        11,534,336 xa.g_1203_fl
07/27/2007  04:36 PM         8,388,608 xa.g_1203_fl_ph
07/30/2007  11:43 AM     1,535,115,264 xa.g_1203_tb
07/30/2007  11:51 AM         1,048,576 xa.g_1203_tb_fl
07/27/2007  04:36 PM         8,388,608 xa.g_1203_tb_fl_ph

Total /server1/xaStatementStore:
Before:  38 File(s)  8,761,556,704 bytes
After:  38 File(s) 17,182,793,440 bytes

~Dave

-----Original Message-----
From: gearon at gmail.com [mailto:gearon at gmail.com] On Behalf Of Paul
Gearon
Sent: Monday, July 30, 2007 10:54 AM
To: David Moll
Cc: mulgara-dev at mulgara.org
Subject: Re: [Mulgara-dev] Size of server1 directory

Hi David,

If shutting down clears some of this away then I have 2 ideas.

The first (and most obvious) is that you have some open Answers
somewhere.  Is it possible that you are doing queries, and not closing
the results, while you do more writes?  A related problem is just
having long-running transactions.

There is another possibility, but I'd need to spend some time with the
code to confirm this... We have a set of files that map internal block
IDs to the phase they were allocated in.  It works fine, but takes up
far more space than it should.  I'm wondering if these are the
culprit?  While the phases they refer to are no longer valid when you
restart, I didn't think these files were truncated.  (Meaning that it
is probably not the issue you are seeing - but as I said, I'd have to
check).

Can you see exactly which files are the ones that grow, and are then
truncated?  Each file name indicates the purpose of the file, and this
would help work out the issue.

Regards,
Paul Gearon