Thoughts on Event Streams in Elastic Environments
Saturday, December 3, 2011 at 14:27 Tweet A few days ago I've posted a thinking-out-loud post on Store and forward for Event Streams. Quite a lot has changed since then:
- Jeremie and Yves shared their experience with event streams on Twitter
- Nice discussion started on DDD/CQRS Google Group
- I've assembled a quick snippet to actually implement Store-and-forward (binding between multiple TCP sockets).
Let me reemphasize the purpose of this building block that I'm trying to research and learn about. It is an attempt to optimize our current technology not only on CAP scale (or CLAP for picky ones), but also from the point of maintenance and practical production experience in hybrid deployments (local, non-Azure cloud, Azure cloud etc). Plus it has to be simple and with low friction.
Anything less - would just make our projects more risky and expensive (to the point of even hiring fourth developer in Ufa office of Lokad to work on our internal portfolio CQRS/Cloud projects).
Another important factor for favoring redundant building blocks (as opposed to centralized servers) is the elastic scalability. You see, even in the cloud you might need to boost processing capability of your system by 2-500 times (or more). Fixed topology is likely to become the bottleneck here. Ideally, scaling out the system will also scale out the infrastructure as well (to a certain limit). This would be similar to how it is done with P2P networks for both data sharing and distributed computations (but with much simpler approach).
Master-Slave configuration
One of the biggest advantages of this event streaming daemon (nicknamed 'spirit' due to its angelic nature) is that these few lines of code allow to plug in various redundancy and reliability options.
- Immediate redundancy - when we plug in Master node to two (or N) slave nodes. Write is considered to be complete only after all nodes have committed. This reduces throughput but ensures data safety.
- Eventual redundancy - when we subscribe multiple downstream nodes to listen to master and record streams. This keeps throughput almost the same, but introduces small window of opportunity (a few ms) during which data is persisted only on master.
Shotgun Scenario
What would happen when the master goes down (completely, which is common in the cloud)? That's actually, where the fun starts. The distributed infrastructure has to detect this and fall back to the slave node, making it a new master.
Then, if the former master comes back, it has to realize that it is a slave now and start catching up.
Sounds complicated, but I have a feeling that all this self-balancing and self-healing could as well be described by a few simple rules; very similar to how nature builds extremely reliable systems out of basic building blocks (biological life on Earth so far experienced no recorded downtime even during the Ice Ages).
Approach could be similar in essence to the very concept of event sourcing (or git) - describe this topology and configuration via event stream that is shared and replicated between the nodes, with conflict resolution (all the voting and negotiations etc) happening within the "event merging" logic of ES. Sounds weird, but there is something promising in the idea.
Any thoughts?
Update: this article continues in Learning Distributed Systems: Sockets and Event Store
Reader Comments (4)
Rinat, thank you. Are the links you have on your Event Sourcing bliki to Greg Young's stuff on cqrsinfo no longer valid? None of those links seem to go to the content that you intended. Look forward to more posts and to Lokad.CQRS v3.
I've found in my research with Correspondence that two things must be true for a system to self-heal in the way that you describe. First, events must be unique based only upon their natural key. That allows the two peers to identify duplicates. Second, events must be partially ordered. That makes it impossible to play them back out of order when order matters, but imposes no additional burden when it doesn't.
I have experienced this self-healing first-hand with Correspondence. I developed an application locally, and then deployed it to AppHarbor. After the deployment, my test data could still be found in the system. One server "failed" and the other seamlessly took over.
When these two conditions are met, it doesn't even matter if both servers overlap for a time. The event histories cannot get corrupted.
Kerry, I'll update references shortly. The old domain has been lost to the squatters. Just use http://cqrs.wordpress.com/ as a reference
Michael, there is also a 3 requirement to handle data corruption, that I found to be helpful in production: event envelopes should have sha1 (or any equivalent) signatures of the data. This helps to detect any transfer or disk corruption problems early on.