[Mulgara-dev] disk usage

Tate Jones tate at bluedog.com.au
Sun Jan 28 07:36:39 UTC 2007


Perform a backup and then a restore to an empty DB.  Is the DB the  
same size?

Do you have many blank nodes? Is so, do you delete these?

Checkout the backup file.  It will tell you what is happening in the  
triple store.  Are there many blank nodes as subject that are not  
referenced by any objects?

On 27/01/2007, at 2:10 PM, Life is hard, and then you die wrote:

>
>   Hi,
>
> we have a question about disk usage. We have a database with around
> 1.4 million triples currently, and the disk usage looks as follows:
>
>   4.0K   lucene
>   1.7M   xaNodePool
>   6.0G   xaStatementStore
>   169M   xaStringPool
>
> (A detailed listing of file sizes is at the end). While the string
> pool looks fine, the statement store looks a bit large.
>
> Now, we have two calculations for the statement store, one from Paul
> and one from Andrae. Given N statements, Paul said basically the disk
> usage should be around N / 192 * 8292 * 6; Andrae said something like
> 12 * 32 * N. This comes to about 363MB and 538MB, respectively, i.e.
> same ballbark (I'm not interested in exact numbers). But both are an
> order of magnitude less than what we're seeing, though.
>
> Paul and Andrae mentioned that space is not reclaimed on deletes, but
> instead goes back into a pool. We don't have the exact numbers, but in
> our case about 1/5 of the triples got inserted (~240000), then about
> 6500 were removed, and then the rest of the triples were inserted.
> There have been some small sets of deletes since then, but nothing
> beyond a few thousand triples. So in total the deletes are < 1% of the
> inserts. Plus anything in the free pool from the deletes should've
> pretty much been used up by the following inserts. But even if not,
> this doesn't look it could account for the discrepency.
>
> So, I'm a bit curious: anybody have any idea why the large disk usage?
> Has anybody else seen this (ever 4K per statement)?
>
>
>   Cheers,
>
>   Ronald
>
>
> P.S. here's a detailed listing of the files by size:
>
>   Files in xaStatementStore, sorted by size:
>
>     1744830464  xa.g_3120_tb
>     1744830464  xa.g_1203_tb
>      847249408  xa.g_3201_tb
>      838860800  xa.g_2013_tb
>      251658240  xa.g_0123_tb
>      243269632  xa.g_3012_tb
>      192757760  xa.g_3120
>      192757760  xa.g_1203
>      100573184  xa.g_3201
>      100573184  xa.g_2013
>       33529856  xa.g_3012
>       33529856  xa.g_0123
>       27623424  xa.g_1203_fl
>       27557888  xa.g_3120_fl
>       14680064  xa.g_2013_fl
>       13959168  xa.g_3201_fl
>        8388608  xa.g_3201_tb_fl_ph
>        8388608  xa.g_3201_fl_ph
>        8388608  xa.g_3120_tb_fl_ph
>        8388608  xa.g_3120_fl_ph
>        8388608  xa.g_3012_tb_fl_ph
>        8388608  xa.g_3012_fl_ph
>        8388608  xa.g_2013_tb_fl_ph
>        8388608  xa.g_2013_fl_ph
>        8388608  xa.g_1203_tb_fl_ph
>        8388608  xa.g_1203_fl_ph
>        8388608  xa.g_0123_tb_fl_ph
>        8388608  xa.g_0123_fl_ph
>        4194304  xa.g_0123_fl
>        4128768  xa.g_3012_fl
>        2949120  xa.g_1203_tb_fl
>        2916352  xa.g_3120_tb_fl
>        1409024  xa.g_3201_tb_fl
>        1409024  xa.g_2013_tb_fl
>         360448  xa.g_3012_tb_fl
>         360448  xa.g_0123_tb_fl
>           1088  xa.g
>
>   Files in xaStringPool, sorted by size:
>
>     125714432  xa.sp_avl
>      33554432  xa.sp_nd
>      11927552  xa.sp_avl_fl
>       8388608  xa.sp_avl_fl_ph
>       8388608  xa.sp_08_fl_ph
>       8388608  xa.sp_08
>       8388608  xa.sp_07_fl_ph
>       8388608  xa.sp_07
>       8388608  xa.sp_06_fl_ph
>       8388608  xa.sp_06
>       8388608  xa.sp_05_fl_ph
>       8388608  xa.sp_05
>       8388608  xa.sp_04_fl_ph
>       8388608  xa.sp_04
>       8388608  xa.sp_03_fl_ph
>       8388608  xa.sp_03
>       8388608  xa.sp_02_fl_ph
>       8388608  xa.sp_02
>       8388608  xa.sp_01_fl_ph
>       8388608  xa.sp_01
>       8388608  xa.sp_00_fl_ph
>       8388608  xa.sp_00
>         65536  xa.sp_19_fl
>         65536  xa.sp_18_fl
>         65536  xa.sp_17_fl
>         65536  xa.sp_16_fl
>         65536  xa.sp_15_fl
>         65536  xa.sp_14_fl
>         65536  xa.sp_13_fl
>         65536  xa.sp_12_fl
>         65536  xa.sp_11_fl
>         65536  xa.sp_10_fl
>         65536  xa.sp_09_fl
>         65536  xa.sp_08_fl
>         65536  xa.sp_07_fl
>         65536  xa.sp_06_fl
>         65536  xa.sp_05_fl
>         65536  xa.sp_04_fl
>         65536  xa.sp_03_fl
>         65536  xa.sp_02_fl
>         65536  xa.sp_01_fl
>         65536  xa.sp_00_fl
>          1408  xa.sp
>             0  xa.sp.lock
>             0  xa.sp_19_fl_ph
>             0  xa.sp_18_fl_ph
>             0  xa.sp_17_fl_ph
>             0  xa.sp_16_fl_ph
>             0  xa.sp_15_fl_ph
>             0  xa.sp_14_fl_ph
>             0  xa.sp_13_fl_ph
>             0  xa.sp_12_fl_ph
>             0  xa.sp_11_fl_ph
>             0  xa.sp_10_fl_ph
>             0  xa.sp_09_fl_ph
>             0  xa.sp_19
>             0  xa.sp_18
>             0  xa.sp_17
>             0  xa.sp_16
>             0  xa.sp_15
>             0  xa.sp_14
>             0  xa.sp_13
>             0  xa.sp_12
>             0  xa.sp_11
>             0  xa.sp_10
>             0  xa.sp_09
>
> _______________________________________________
> Mulgara-dev mailing list
> Mulgara-dev at mulgara.org
> http://mulgara.org/mailman/listinfo/mulgara-dev

Tate Jones
tate at bluedog.com.au





More information about the Mulgara-dev mailing list