[Mulgara-dev] TransactionalAnswer not closed

Life is hard, and then you die ronald at innovation.ch
Sat Feb 10 11:00:12 UTC 2007


On Tue, Feb 06, 2007 at 09:48:27AM -0600, Paul Gearon wrote:
> 
> On Feb 5, 2007, at 2:55 AM, Life is hard, and then you die wrote:
> 
> >On Mon, Feb 05, 2007 at 12:14:11AM -0600, Paul Gearon wrote:
> >>Just how big are your sub-answers?  Thousands of lines?
> >
> >Actually, come to think of it, in our case the subanswers themselves
> >aren't that big: 2, 3, maybe 10 lines. We had/have a problem with a
> >query that returns around 30'000 lines, each line consisting of 3
> >subanswers, and each subanswer between 1 and 5 lines. This causes
> >mulgara to run out of heap space (the limit directive didn't help).
> >We ended up having to split the query apart, running it as 3 separate
> >queries with no subqueries and then merging the results ourselves.
> >
> >So this may be a different problem.
> 
> You're just full of good news this last couple of days, aren't you?  :-)

Sorry...

> 3 subanswers * 10 lines per subanswer is definitely not going to  
> cause a memory issue..... unless you're returning massive blobs in  
> base64binary literals.  I should find out what your problem is here.
> 
> The problem is even I run the same queries, I don't have THAT large a  
> data set to test with.  Maybe I could work with you to find this  
> problem?

Sure. I think we can arrange something.

> In the meantime... who throws the exception (the client or the server  
> - I'm guessing the server), and where is the code when it runs out of  
> space?

The server. (otherwise we would have already fixed it)

As to the stack trace, again it's an OOM exception and it's therefore
pretty much useless.

However, I think I just found the "problem". I was looking more
closely at the server log, specifically at the subqueries, and noticed
they were not being constrained correctly.

Ok, here some background. The query being executed looked liked
(slightly simplified):

    select $user
        subquery(select $email from
                 <...> where $pref <topaz:preference> $prefn and
                 $prefn <topaz:prefName> 'email' and
                 $prefn <topaz:prefValue> $email)
        subquery(select $foo from
                 <...> where $pref <topaz:preference> $prefn and
                 $prefn <topaz:prefName> 'foo' and
                 $prefn <topaz:prefValue> $foo)
        subquery(select $bar from
                 <...> where $pref <topaz:preference> $prefn and
                 $prefn <topaz:prefName> 'bar' and
                 $prefn <topaz:prefValue> $bar)
     from <...>
       where $user <topaz:hasPreferences> $pref;

The problem is that the subqueries executed looked like (the first
one):

    SELECT $email FROM ... WHERE ( and [$pref ... $prefn $_from] [$prefn ... "email" $_from] [$prefn ... $email $_from] [$user <mulgara:is> info:doi/10.1371/account/10093 $_from])

I.e. there no constraint on $pref from the outer query! This vaguely
rung a bell, so I changed query to look like

    select $user $pref
        subquery(select $email from
        ...

i.e. added $pref to the select clause (even though we're not
interested in it) and bingo, the inner queries now look like

    SELECT $email FROM ... WHERE ( and [info:doi/10.1371/preferences10536/topaz-plosone ... $prefn $_from] [$prefn ... "email" $_from] [$prefn ... $email $_from] [$user <mulgara:is> info:doi/10.1371/account/10536 $_from] [$pref <mulgara:is> info:doi/10.1371/preferences10536/topaz-plosone $_from])

Now the query runs beautifully and quite fast.

I guess this is sorta documented:

  Subqueries nest inside select by binding variables in the subquery
  where clause to the outer select clause.

But it seems, from a user perspective, a bit annoying - I suppose
there's a good reason why all variables that are "passed" to the
subquery have to be listed in the select clause?


  Cheers,

  Ronald


P.S. for the modified query to actually work without throwing a
     heap-error I needed to apply the ItqlInterpreterBean patch too
     which closes the subanswers correctly.




More information about the Mulgara-dev mailing list