Windows Azure Most Wanted - MSMQ Azure

As we all know, Windows Azure provides cloud computing services on "Platform as a Service" basis. This includes usual storage, network and computing capacities as well as a few things on top of that:

• Windows Azure Queues;
• Azure Table Storage;
• Azure Dev Fabric and terrific Visual Studio 2010 integration;
• SQL Azure.

It seems like there is everything in place to build truly scalable applications on top of the Windows Azure. In fact, that's how we do this at Lokad (BTW, it feels really interesting to watch in real time how Azure provisions 20+ worker roles to absorb the load and then lets them go).

Yet, there is one really important missing piece that would've simplified large-scale enterprise development a lot. It would've lowered the entry barrier for the development of simple cloud applications as well.

I'm talking about something like MSMQ Azure (Microsoft Message Queuing), just like we have SQL Azure.

Microsoft Message Queuing or MSMQ is a Message Queue implementation developed by Microsoft and deployed in its Windows Server operating systems since Windows NT 4 and Windows 95.

Wait, but does not Windows Azure already have messaging in form of Azure Queues? Well, not quite.

MSMQ server provides non-elastic message queues that can be transactional and durable. In a sense that's similar to any RDBMS engine (non-elastic, transactional and durable). And that's how SQL Azure is.

Azure Queues, on the other side, are designed for the cloud elasticity. This means, that any number of messages and message queues could exist for your application. You don't even have to bother about the servers, capacities or scaling - infrastructure will handle these details. As a trade-off Azure Queues have to implement lightweight transactional model in form of message visibility timeouts. This guarantees, that the message is guaranteed to be processed at least once. But it does not guarantee, that the message will be processed only once (forcing us to do some sort of locking in sensitive scenarios). We additionally lose System.Transactions support.

So Azure Queues have scalability that would work for a huge enterprise, but have a number of pain points, that such an enterprise would have to deal with:

• Implementing transaction scopes (at least in form of the volatile enlistments);
• Implementing some sort of locks (these normally fall back to your good old MS SQL Server);
• Implementing BLOB persistence for the messages above the Azure Queue message limit (which gets hit really fast).

At Lokad, we've been working around these problems in OSS projects of Lokad.Cloud (Azure Object/Cloud Mapper and distributed execution framework) and Lokad Cloud ESB (Azure Service Bus Implementation), more of which I'll talk about later. Yet, a lot of things would've been more simple if Azure MSMQ was to be provided along the similar lines as SQL Azure.

MSMQ server is not scalable on it's own, but how many solutions do really need to process hundreds of transactions per second? Probably 95% of applications would live happily with a single MSMQ Azure instance, just like SQL Azure is enough. And if the application gets stressed to the limits of a single MSMQ Azure instance, then the business behind it is probably profitable enough to have some development resources to invest into partitioning or fine-tuning performance bottlenecks.

Additionally, Azure MSMQ would open direct transition path for OSS enterprise service bus implementation projects like NServiceBus, MassTransit and Rhino ESB towards the cloud (at the moment disparities between the durable transactional message transports and cloud queues keep them away from the cloud despite all the interest in the community).

And with the proper Cloud ESB available within Windows Azure one could build really scalable, flexible and efficient enterprise solutions with the rich toolset and experience of CQRS and DDD.

If we look at the problem of porting existing code that uses MSMQ, it feels like a decent and expensive challenge to port it to Azure Queues transport. If we are talking about an enterprise solution in production, I doubt it will be an easy job to justify to your CTO migration to the cloud in such scenario.

In the next blog post we'll talk about Azure Queues with concurrency locks vs. MSMQ.

What do you think? Do you agree with the proposition (you can vote for it on uservoice) or are you happy with the Azure Queues?

Comments

[Jonathan Oliver]

Rinat,

You've no doubt been reading my blog as I have been discussing just such a set of problems. In general, we need to create a small amount of messaging middleware that can facilitate messaging de-duplication for us when operating in a cloud environment.

http://jonathan-oliver.blogspot.com/2010/04/messaging-at-least-once-delivery.html
http://jonathan-oliver.blogspot.com/2010/04/idempotency-patterns.html

One interesting possibility is to leverage a traditional RDBMS implementation (which also operates in the cloud) to provide blocking behavior thus preventing a messaging from being processed multiple times:
http://jonathan-oliver.blogspot.com/2010/05/cassandrariakdynamo-optimistic.html

[Rinat Abdullin]

Reply to you ended up in another blog post))

[Paul.Sorvik]

Good reasons for a MSMQ Azure feature, please Vote for it as this entry suggests at the end!