Transaction trouble with Cache

Consider the following problem:

We have a web app which can send messages between the users that are connected to it. A user can send a message to any number of users.

The implementation is simple:

When a user sends a message

The message is saved in the database
Cache value is inserted for each recipient. The saved value is a time stamp which indicates when the user last received a message.

Users that are suppose to see the message:

The client is polling the server and asks if the server has new messages. It does that with an initial time which is loaded when the page is loaded and saved in JavaScript.

If there is something new the client asks the server for the new messages (according to saved time) and the server returns the messages from the database and the last received time from the cache.

Client time stamp is updated with the received time and continues polling

Pretty simple…

Unfortunately the implementation has a critical bug which causes clients to lose messages.

Consider this code:

using (var t = OpenTransaction())
{
	SaveMessageToDb(message);

	foreach(var user in message.Recipients)
	{
		Cache[user.Code] = message.SendDate;
	}

	t.Commit();
}

Can you spot the bug?

Let say we have 1000 users that will receive the message. After saving the message to the db the code updates the cache for each client. Each client polls the server at a constant interval. Let say that user AAA polls the server – his cache key is updated but the loop that updates the cache is not yet over. The user send a request to get the new messages but because the transaction wasn't committed so the new messages will not be returned from a query – the loop that updates the cache is still going. The client then updates the last received time – and so the user will never get the message.

I must say that this was a very frustrating bug…

The solution was composed with changes in both the client and the server:

Client

We changed the client script so that if the server claims that there are new messages but the clients receives an empty list of messages then the received time is not updated and another request is sent to the server until new messages are received.

Server

What we need to do here is to make the cache join the transaction. In this case we couldn't because the transaction was a local oracle transaction – not a distributed transaction. If it was we could use System.Transactions to implement a transactional cache.

In this case we used an event which is triggered when the transaction is commited (in TransactionScope there is the AfterCommit event, in our case we had to implement this ourselves). It looks something like this:

using (var t = OpenTransaction())
{
	SaveMessageToDb(message);

	t.AfterCommit = (sender,e) => 
	{
		foreach(var user in message.Recipients)
		{
			Cache[user.Code] = message.SendDate;
		}
	}

	t.Commit();
}

Way not simply update the cache outside the transaction?

Because we don't actually know when the transaction is committed. Maybe another method calls this method with a transaction of her own and the commit is actually done there (the OpenTransaction method actually returns a current transaction if one exists already).

Important

Our solution was infect a little more complex because we wanted other modules that are in the transaction to be able to get the updated value from the cache even if the transaction wasn't committed yet:

using (var t = OpenTransaction())
{
	SaveMessageToDb(message);

	t.AfterCommit = (sender,e) => 
	{
		foreach(var user in message.Recipients)
		{
			Cache[user.Code] = message.SendDate;
		}
	}

	DoSomthingThatUsesCache();

	t.Commit();
}

So we created a class which wraps the cache, and contains a local dictionary which contains updated values, and access them when needed. If a value is updated in a transaction then wrapper registered to the event and updates the real cache when the transaction is committed. We called the class TransactiveCache:

using (var t = OpenTransaction())
{
	SaveMessageToDb(message);

	foreach(var user in message.Recipients)
	{
		TransactiveCache[user.Code] = message.SendDate;
	}

	DoSomthingThatUsesCache();

	t.Commit();
}