[Mulgara-dev] Emailing: protocolServlet POST answercleanup.patch

David Smith DMS at viewpointusa.com
Tue Jan 19 22:58:54 UTC 2010


Thanks
These were the breadcrumbs i needed.
I'll take a crack at it.

Dave s



-----Original Message-----
From: Paul Gearon <gearon at ieee.org>
Sent: Tuesday, January 19, 2010 5:31 PM
To: Mulgara Developers <mulgara-dev at mulgara.org>
Subject: Re: [Mulgara-dev] Emailing: protocolServlet POST answercleanup.patch

On Tue, Jan 19, 2010 at 3:07 PM, David Smith <DMS at viewpointusa.com> wrote:
> Thanks for answering.
>
> Yes, we're switching from SOAP to REST.
> Yes, the system has been working correctly under SOAP.
>
> We were seeing performance/scaling issues, with slow progress through
> our actual work,
> with very small actual demand on the server. Something is/was blocking
> somewhere.

There are a few bottlenecks, yes. I did some work on this last year,
and I'll be back onto it again next week.

> After working through some .NET configuration issues with calling REST
> services, we started working with the JETTY configuration settings.
> These did not seem to help us with the calling through the SOAP
> interface.
>
> Then we tested with the REST interface and did see better performance.
> Moved everything and in testing we're seeing the logs flooded (really a
> lot) with these messages.

As I mentioned earlier, subanswers need to be closed as well. I've
been through the code that closes them when you use RMI, and I can see
that it's not happening when streaming results to XML. It looks like a
reasonably simple fix, but I don't have adequate tests to make sure
I'll get them all. I *can* ensure that you'll get fewer of these
errors though.

> Mulgara seems to bog down (significantly --
> this time very busy but still not responding well) and not knowing what
> else to chase suspected the Answer close issue (and/or the actually log
> generation). Maybe something to test is commenting out the log() entry
> in the TransactedAnswer finalize.

Given the number of these that you're probably seeing, then you'll
certainly see a slowdown, yes.

Incidentally, TransactedAnswer wraps the actual answer that needs to
be closed. You may be seeing warnings coming out of those Answers as
well. I'm not all that familiar with TransactedAnswer, as it only came
into being recently with the JTA architecture, and I haven't had need
to give it more than a cursory inspection.

> We switched to REST/POST because current the system used one internal
> interface to get to mulgara (no matter the actual query). I may go
> through and separate out the pure SELECTs and switch to GET... but have
> not so far.

>From memory, GET and POST have a similar implementation for SELECT, so
it may not help much.

POST is actually supposed to accept very large queries in the body
(instead of parameters in the URL), but this implementation is
something I've been putting off. I occasionally get asked for
something simple to start doing development on, so I've been keeping
this one for the next time I'm asked. That's the only reason it hasn't
been done yet.

> The only real problem so far was one case where the SOAP call included
> multiple SELECTs and took advantage of the single response <ANSWER> with
> multiple <QUERY><SOLUTION>... sections.

Hmmm, I forgot about that for TQL/REST. Do you still need it? I think
I can get that done for you.

> Ok, so it's worth keeping the session logic.
> The earlier patch was to let us use the REST/POST with queries with
> subqueries.
> The patch below was my attempt to make sure the answers were being
> closed.
> Can you point me at an example of correctly closing Answers with
> subqueries involved. I tried looking and didn't see them.

It's in AnswerPageImpl. This is code that is *only* used when
serializing data across RMI. The REST interface operates directly on
Answers (without serializing), so it never happens in REST.

To explain how it works.... when you query over RMI, the client side
is given a RemoteAnswerWrapperAnswer. This is a class that wraps a
RemoteAnswer, and presents the Answer interface. When the client first
calls next() on their RemoteAnswerWrapperAnswer, it calls
RemoteAnswer.nextPage() and returns the first row from that page.
Subsequent calls to next() will all come from the page until the page
is empty.

On the server side, the Answer from a query is wrapped in an
AnswerWrapperRemoteAnswer. This wraps the Answer and presents a
RemoteAnswer interface. (No, I didn't invent this stuff).

When RemoteAnswer.nextPage() is called, an AnswerPageImpl is created
to suck in the next page worth of data. Since this is the class that
is reading out of the Answer, it is also the one that closes the
subanswers that it iterates through.

Back on the client side, the subanswers that you get have all been
serialized into a type of Answer called ArrayAnswer. Since these
things are completely in RAM, there are no resources to reclaim, so
their close() methods don't do anything (except to note if you close
them more than once). That's why an RMI client never has to worry
about closing these things.

The only place that these sub Answers can be closed in REST is where
they're being iterated through, which happens in the XML generation.
The overall Answer is closed outside of the class that creates the
XML, so I should either bring this inside the class and close all
answers in there (for consistency), or be expedient and close just the
subanswers.

Paul
_______________________________________________
Mulgara-dev mailing list
Mulgara-dev at mulgara.org
http://mulgara.org/mailman/listinfo/mulgara-dev

This E-Mail may contain confidential and privileged information. It is intended solely for the recipient(s) indicated. Any review, use, or distribution by anyone other than the intended recipient(s) is strictly prohibited. If you have received this E-Mail in error or are not the intended recipient, please notify the sender and delete all copies immediately.



More information about the Mulgara-dev mailing list