Scalability and Concurrency Challenge

It is interesting, how concepts of separated features and messaging become more important as the distributed applications evolve. I keep encountering these again and again in the solutions at Lokad and in new reads showing up:

• Articles about cloud computing;
• Effectus codebase by Ayende;
• Command-Query Responsibility Segregation by Udi;
• MindTouch Dream principles of building lightweight decoupled services;
• Composite Application theory.

So for the last few weeks I’ve been discovering things like Advanced Message Queue Protocol (reading its specs is a pleasure of its own) and its implementations (Apache Qpid and RabbitMQ). There had also been a different approach to the REST services and concurrency handling as in the inspiring MindTouch Dream framework.

Obviously, the toolset is not your native Microsoft .NET toolset, but:

• MindTouch Dream works natively on Mono and .NET;
• Apache QPID and RabbitMQ have .NET clients;
• It takes just a few lines of code to write .NET Dream Service to act as a gateway between AMQP and REST;
• For some reason I like AMQP more than MSMQ;
• In a current world it takes less than an hour to procure a cheap running server out in the cloud.

It is also interesting to see how REST services and AMQP middleware services have their parallels with the classical composite applications:

• AMQP == EventBroker
• Services == IoC Container environment (roughly)

So basically, there are quite a lot of concepts and technological approaches that suddenly start making a lot of sense to me.

At the moment I’m interested in giving a try to build some learning prototype application using these technologies, just for the sake of practicing and learning pros and cons better.

Does anybody know of a common or specific problems out in the field, that require potential scalability and async processing? Something, where scalability, concurrency and sync processing create a lot of challenges.

Update: Jonathan has really nice article about Getting Started with DDDD and CQS (there are a lot of references there as well).

Comments

[Jonathan]

I would pay close attention to anything Udi Dahan and Greg Young say. There are a number of recordings of Greg Young giving a presentation on CQRS and message-based systems:

http://jonathan-oliver.blogspot.com/2009/03/dddd-and-cqs-getting-started.html

[Rinat Abdullin]

@Jonathan, thanks a lot for the article. I've updated my post with link to your article. There certainly is a lot of information to digest.

[Jonathan Oliver]

In looking at "MindTouch Dream", it seems to be using a lot of RPC and request/response-style thinking. You'll want to be looking at doing what Udi Dahan calls "Autonomous Components" instead. You'll have to dig out the definition from Udi's posts or you can email me and ask. But they're an extremely powerful building block when building business services.

I attended Udi's Advanced SOA course and it was literally a night-to-day transition:
http://www.udidahan.com/training/

[Rinat Abdullin]

Autonomous component, as I understood it from the CQRS, is just an automation engine responsible for processing certain messages, while working with the state (sync here) and publishing messages. A few Dlls hosted in the win service would do the trick. If there is a perf bottleneck you can sometimes scale up with adding a few more ACs.

In that sense, MindTouch Dream + AMQP server seems to be a simple solution. Wiring all messages through the MQ will force requests to be message-based in nature, while Dream just provides coroutine framework, manageable service host and a REST API.

In a sense I'm probably just reinventing NServiceBus with a different set of tools for the sake of learning the technology properly.

What do you think?

[Colin Jack]

Sounds like an interesting topic and REST/messaging would seem sensible choices to me, also Dream/RabbitMQ would be good frameworks to choose for such an example.

As for the problem domain, dunno. All I'd say is if it were me I think I'd choose something relatively simple or something we all understand (placing orders, hotels, booking flights).

[Rinat Abdullin]

@Colin,

Yes, I'm thinking along the lines of Fojhin DDD sample