[Mulgara-dev] [Mulgara-svn] r1060 - branches/mgr-121-lockrecovery/src/jar/resolver/java/org/mulgara/resolver

[Paul Gearon]

Ronald's comments below for the fix he checked into a branch last
night reminds me of use cases that might help understand some of the
original assumptions in Mulgara. This might be useful in finding
future bugs, understanding the reasoning behind the current structure,
or just writing client code that drives the server a little more
effectively.

It was originally envisioned that a client would establish a
connection (session) and issue queries and commands on this connection
in a serial fashion. When I say "serial" I'm also referring to
processing Answers. One of the problems that has been coming up is to
start writing to a connection while still processing an Answer. In
fact, once upon a time, this was not even possible. Now it's just
buggy.  :-)

The VERY first implementation of Query returned a serialized Answer
object. Unfortunately, large Answers could take a LONG time to return.
So then someone had the bright idea of changing Answers to be remote
objects. Obviously that wouldn't fly due to network latency for every
call to next() or getObject() (though IIRC we got it up and running
like that to make sure we'd configured it right).

The solution chosen was a hybrid system, where we sent blocks of rows
over the network, and calls to next() and getObject() were accessing
local data instead of accessing remote data via RMI. Any Answers that
were small enough skip this and just get sent over in a single block.
The limit of "small enough" is totally arbitrarily chosen to be 100
rows, though you can override it to another value by setting the
mulgara.rmi.marshallsizelimit system property.

The issue with the "blocked data" approach was that any code that read
this data and spat it out to a console would pause at the end of every
block while it waited for the next block to come in. This is where the
motivation came from to have the next block being transferred in the
background while the most recent block is being processed. This
removed pretty much all the pauses, and seemed to work well.

Unfortunately, it was not envisioned that someone would start writing
back to the same session while processing Answers that were being
transferred with a background thread. We made (the obviously
incorrect) presumption that an Answer would be read before a new
operation were performed. With that presumption, it is fine to have a
different thread do the fetching, since there is only one thread
operating on the session at a time. But if a transaction continues in
the meantime, then this can cause the issues Topaz has been seeing.

There are lots of approaches to fix this.:

- The simplest is to remove the background thread. That has
undesirable performance characteristics (if you have a lot of data to
be transferred, you really want to be transferring it until it's all
in, and not have to wait until someone asks for something in the next
block before you bother to get around to fetching it). It also allows
buggy clients to continue doing the wrong thing.
- A client could have a arbitrator running in a thread per connection,
where all commands/queries get queued to this thread, and returned to
the caller as they get back (this would let a client use a connection
with as many threads as they wanted, and not break single-thread
assumptions on the server). This would enforce serialization of calls
from the client end.
- Another solution is to lock the server to only handle one thread at
a time. This seems to be what Ronald has done below (is this right?).
- The server could be re-architected to handle multiple threads
safely, locking portions as needed. We're moving that way, but it's
long way out.
- The RMI interface could be thrown away and a REST API brought in.
This excludes the possibility of a client issuing concurrent
operations on a single connection. I believe we need to move this was
for a number of reasons, but it's also a long way out.

Hope this was interesting/useful.

Paul

On Mon, Jul 7, 2008 at 7:54 AM,  <ronald at mulgara.org> wrote:
> Log:
> Fixed lacking synchronization and deadlocks.
>
> Specifically, this changes the transaction, transaction-factory, and the
> xa-resource instances to all use the same, shared, mutex for a given Session
> (as opposed to each instance having its own mutex). This avoids all the
> deadlock problems once and for all. Additionally, all the methods on the
> transaction classes are also protected by the mutex (see below). And lastly,
> ensured proper synchronization between timeout threads and the rest by making
> the last-active variables volatile and by putting access to the current
> write-transaction variable in a synchronized block.
>
> Some background on this. While the contract for (Database)Session does not
> allow the application to use a given instance concurrently in multiple
> threads, we end up having concurrent access to it from multiple sources:
>  * RemoteAnswerWrapperAnswer uses a prefetch thread to pre-fetch a page of
>   results; this may run concurrently behind the applications back with
>   whatever operation the application is doing on the same Session, including
>   both transaction-control and data operations. This means a
>   MulgaraTransaction.execute() may occur concurrently with any other operation
>   (transaction-control or data) on the transaction, the transaction-factory,
>   or the xa-resource.
>  * Timeouts also run in a separate thread, so a heuristicRollback may run
>   concurrently with any operation on the transaction, the transaction-factory,
>   or the xa-resource.
>  * The Session may get closed by the RMI 'unreferenced' hook in
>   SessionWrapperRemoteSession, which also runs in a separate thread. While
>   somewhat rarer, a long running operation may still be in progress when this
>   occurs (e.g. the user gave up and terminated the client), or an operation
>   may be waiting for the write-lock and acquire at the same time as the close
>   happens.
>  * A buggy client may violate the contract.
> This therefore requires all entry points to be fully synchronized.
>
> One "drawback" of using the same mutex for all parties for a given Session is
> reduced parallelism of operations on a given Session. This basically only
> affects the pre-fetching of pages of query results which will now be serialized
> on entry to the transaction execute() (an application is not allowed to issue
> multiple operations concurrently (on a given Session). However, the resulting
> complexity of trying to support this is not worth it IMNSHO (witness the
> numerous bugs in this area so far).