[Mulgara-dev] disk usage
Tate Jones
tate at bluedog.com.au
Sun Jan 28 07:36:39 UTC 2007
Perform a backup and then a restore to an empty DB. Is the DB the
same size?
Do you have many blank nodes? Is so, do you delete these?
Checkout the backup file. It will tell you what is happening in the
triple store. Are there many blank nodes as subject that are not
referenced by any objects?
On 27/01/2007, at 2:10 PM, Life is hard, and then you die wrote:
>
> Hi,
>
> we have a question about disk usage. We have a database with around
> 1.4 million triples currently, and the disk usage looks as follows:
>
> 4.0K lucene
> 1.7M xaNodePool
> 6.0G xaStatementStore
> 169M xaStringPool
>
> (A detailed listing of file sizes is at the end). While the string
> pool looks fine, the statement store looks a bit large.
>
> Now, we have two calculations for the statement store, one from Paul
> and one from Andrae. Given N statements, Paul said basically the disk
> usage should be around N / 192 * 8292 * 6; Andrae said something like
> 12 * 32 * N. This comes to about 363MB and 538MB, respectively, i.e.
> same ballbark (I'm not interested in exact numbers). But both are an
> order of magnitude less than what we're seeing, though.
>
> Paul and Andrae mentioned that space is not reclaimed on deletes, but
> instead goes back into a pool. We don't have the exact numbers, but in
> our case about 1/5 of the triples got inserted (~240000), then about
> 6500 were removed, and then the rest of the triples were inserted.
> There have been some small sets of deletes since then, but nothing
> beyond a few thousand triples. So in total the deletes are < 1% of the
> inserts. Plus anything in the free pool from the deletes should've
> pretty much been used up by the following inserts. But even if not,
> this doesn't look it could account for the discrepency.
>
> So, I'm a bit curious: anybody have any idea why the large disk usage?
> Has anybody else seen this (ever 4K per statement)?
>
>
> Cheers,
>
> Ronald
>
>
> P.S. here's a detailed listing of the files by size:
>
> Files in xaStatementStore, sorted by size:
>
> 1744830464 xa.g_3120_tb
> 1744830464 xa.g_1203_tb
> 847249408 xa.g_3201_tb
> 838860800 xa.g_2013_tb
> 251658240 xa.g_0123_tb
> 243269632 xa.g_3012_tb
> 192757760 xa.g_3120
> 192757760 xa.g_1203
> 100573184 xa.g_3201
> 100573184 xa.g_2013
> 33529856 xa.g_3012
> 33529856 xa.g_0123
> 27623424 xa.g_1203_fl
> 27557888 xa.g_3120_fl
> 14680064 xa.g_2013_fl
> 13959168 xa.g_3201_fl
> 8388608 xa.g_3201_tb_fl_ph
> 8388608 xa.g_3201_fl_ph
> 8388608 xa.g_3120_tb_fl_ph
> 8388608 xa.g_3120_fl_ph
> 8388608 xa.g_3012_tb_fl_ph
> 8388608 xa.g_3012_fl_ph
> 8388608 xa.g_2013_tb_fl_ph
> 8388608 xa.g_2013_fl_ph
> 8388608 xa.g_1203_tb_fl_ph
> 8388608 xa.g_1203_fl_ph
> 8388608 xa.g_0123_tb_fl_ph
> 8388608 xa.g_0123_fl_ph
> 4194304 xa.g_0123_fl
> 4128768 xa.g_3012_fl
> 2949120 xa.g_1203_tb_fl
> 2916352 xa.g_3120_tb_fl
> 1409024 xa.g_3201_tb_fl
> 1409024 xa.g_2013_tb_fl
> 360448 xa.g_3012_tb_fl
> 360448 xa.g_0123_tb_fl
> 1088 xa.g
>
> Files in xaStringPool, sorted by size:
>
> 125714432 xa.sp_avl
> 33554432 xa.sp_nd
> 11927552 xa.sp_avl_fl
> 8388608 xa.sp_avl_fl_ph
> 8388608 xa.sp_08_fl_ph
> 8388608 xa.sp_08
> 8388608 xa.sp_07_fl_ph
> 8388608 xa.sp_07
> 8388608 xa.sp_06_fl_ph
> 8388608 xa.sp_06
> 8388608 xa.sp_05_fl_ph
> 8388608 xa.sp_05
> 8388608 xa.sp_04_fl_ph
> 8388608 xa.sp_04
> 8388608 xa.sp_03_fl_ph
> 8388608 xa.sp_03
> 8388608 xa.sp_02_fl_ph
> 8388608 xa.sp_02
> 8388608 xa.sp_01_fl_ph
> 8388608 xa.sp_01
> 8388608 xa.sp_00_fl_ph
> 8388608 xa.sp_00
> 65536 xa.sp_19_fl
> 65536 xa.sp_18_fl
> 65536 xa.sp_17_fl
> 65536 xa.sp_16_fl
> 65536 xa.sp_15_fl
> 65536 xa.sp_14_fl
> 65536 xa.sp_13_fl
> 65536 xa.sp_12_fl
> 65536 xa.sp_11_fl
> 65536 xa.sp_10_fl
> 65536 xa.sp_09_fl
> 65536 xa.sp_08_fl
> 65536 xa.sp_07_fl
> 65536 xa.sp_06_fl
> 65536 xa.sp_05_fl
> 65536 xa.sp_04_fl
> 65536 xa.sp_03_fl
> 65536 xa.sp_02_fl
> 65536 xa.sp_01_fl
> 65536 xa.sp_00_fl
> 1408 xa.sp
> 0 xa.sp.lock
> 0 xa.sp_19_fl_ph
> 0 xa.sp_18_fl_ph
> 0 xa.sp_17_fl_ph
> 0 xa.sp_16_fl_ph
> 0 xa.sp_15_fl_ph
> 0 xa.sp_14_fl_ph
> 0 xa.sp_13_fl_ph
> 0 xa.sp_12_fl_ph
> 0 xa.sp_11_fl_ph
> 0 xa.sp_10_fl_ph
> 0 xa.sp_09_fl_ph
> 0 xa.sp_19
> 0 xa.sp_18
> 0 xa.sp_17
> 0 xa.sp_16
> 0 xa.sp_15
> 0 xa.sp_14
> 0 xa.sp_13
> 0 xa.sp_12
> 0 xa.sp_11
> 0 xa.sp_10
> 0 xa.sp_09
>
> _______________________________________________
> Mulgara-dev mailing list
> Mulgara-dev at mulgara.org
> http://mulgara.org/mailman/listinfo/mulgara-dev
Tate Jones
tate at bluedog.com.au
More information about the Mulgara-dev
mailing list