Avoid a client/server combination?

of Kiss Coda Wiki

The Long Story Short

There used to be specific types of problems that occurred when the Coda server and client were on the same machine. Most of the issues described below are no longer present with current Coda clients and servers.

It actually works well.

This is a technical post. Unless you are interested in all the details, skip this page.

The Short Story Long

Coda can operate in connected, write-disconnected (weakly connected), and disconnected modes.

If you’re running both a client and a server on the same machine, it is to be expected that they end up either weakly connected (write-disconnected), or completely disconnected at times. Strong connectivity cannot be guaranteed, not when there is a network between machines, and not even when both processes are running on the same machine.

See more on how Strong/weak connectivity estimates are produced.

Anyway, this works most of the time when the client and server are on different machines, because we can assume that almost all of the measured delay can be attributed to the network, as the server already subtracted it’s internal processing time. However, when both the client and the server are on the same machine, we get additional scheduling/context switching, and resource contention.

Both the client and the server have a big chunk of mmapped data that they need to access (and modify) – the RVM data. The client tends to work optimistically and flushes dirty data as little as possible, while the server is very conservative and flushes pretty much as soon as anything important happens.

The way that fflush/fsync is implemented in the Linux kernel, the process that calls sync is blocked until all dirty data for the same device is flushed to disk.

So the client ends up dirtying a lot of memory, both RVM and the data associated with the container file, but doesn’t flush anything to disk. Then it sends an RPC to the server who fetches the data, which is most likely for the most part still in memory since the client didn’t flush. But now the machine is hit with a double whammy, not only is it writing the server process' dirtied data, but also the client process' dirty stuff.

Because we’re single threaded (threading happens in userspace), the server process is blocked until all the data has hit the disk. In the mean time the client is eager for a response, earlier when it was fetching data the server was real quick to respond, the RTT estimate probably ended up being near zero. So it get’s impatient and assumes the request got dropped and retransmits. At the same time, the poor server is still stuck waiting for the disk, and can’t even dash off a quick ack telling the client that it did get the request and is working on it.

So the client tries another couple of times and either gives up and disconnects (not so likely with recent RPC2), or by the time it gets an answer from the server decides that the previous latency and bandwidth estimates were far too optimistic and switched to write-disconnected mode.

Now all of this is most likely to happen when we’re performing a STORE operation. And the fallback for a store during disconnection is a very special case. Most other operations actually return -ETIMEDOUT to userspace, but since stores happen on a close and most applications don’t retry when close returns a failed, the store is logged in the CML. This is done by internally retrying the operation, but we end up logging the operation with a different operation identifier compared to the timed out store operation.

Not really a problem if we really timed out, but in this case the server probably ended up completing the operation, we just missed out on the final reply. And when we reintegrate we automatically hit a conflict because the locally cached version vector is different from the server’s version and there is a different store identifier associated with the operation. So we assume we got an update/update conflict, i.e. another client wrote to the file while we were disconnected.

References


At its former host wikidev.net this page has been accessed 3930 times.


Last modified: Fri Jun 27 14:02:07 UTC 2014