CQRS - Validation and Business Rules

There had been a couple of questions on the place of validation within the Command-Query Responsibility Segregation principles:

• Validation: should it be UI-based only?
• Where does validation belong?
• How do validation and business rules fit into the CQRS world?

Let's try to figure this out along the lines of what we already know about CQRS and validation.

Validation

Validation generally ensures that the input data is valid at the moment of entry, within the limited scope of the UI or data adapter. This means that the data passing validation does not necessarily comply with all the rules and requirements of the organization, but at least it makes a lot of sense.

So if we are creating a customer in the UI (if for some reason, all this is included in a single command), then the validation for this operation will make sure that:

• NewCustomer has first name and last name
• NewCustomer has billing address
• NewCustomer has email that fits the standards (and is probably unique).
• etc

In the CQRS world this validation could run in form of various UI-driven rules and additional request-response checks against the available query data sources (i.e.: in-memory collection of all the emails within the system). Result of such validation is about preventing user from sending invalid command to the server.

For example, for ASP.NET MVC, validation code might look like:

[HttpPost]
public ActionResult Start(CreateNewCustomerForm form)
{
   ModelState.Validate(form, CreateNewCustomerRule);

   if (ModelState.IsValid)
   {
     var requestId = Guid.NewGuid();
     MessageBus.Send(MapFormToCommand(form, requestId));

     return RedirectToAction("wait", new {id = requestId});
   }

   return View(form);
}

The desktop alternative is similar:

var result = _createNewCustomerView
 .GetData(Rules.CreateNewCustomerRule);

if (result.IsSuccess)
{
  MessageBus.Send(MapModelToCommand(customer));
  _customerView.AddToList(customer);
}

You can check out .NET Validation application block for some inspiration on writing fluent and reusable validation rules for the UI. Keep in mind, though, that this block was developed within the xLim 3 and does not completely fit into the CQRS paradigm at the moment.

Once the user has made its way through, result of this validation is a command, that has really high probability of being correct. Ratio of success of 99.9% and above is something that we are aiming at. Although the remaining fraction of percent is important, we don't have to ensure absolute consistency at this moment of time. This small allowance let's us keep the client-side validation rather simple and scalable.

This validation process is scalable for two reasons:

• almost all the calculations are performed on the client-side.
• validation against the cache (queries) are inherently scalable due to the potentially distributed nature of the queries (which happens thanks to the fact, that commands can be 99.9% correct at the moment of UI Validation).

Valid command is sent to the server, which passes it into the messaging system, eventually reaching the command handler. After the server gets this command, it should also run the validation locally. First of all, we don't trust anybody. Second, now we run same validation rules, but within the transaction, to ensure 100% validity.

So at this moment we already see, that validation is not just UI-based. It could be applied to the command, as a logical entity, and thus should be executed either at server or client.

If the validation fails due to some simple reason (customer name is missing or does not follow the rules), then we could probably discard the command completely. We don't expect such failing commands to arrive (since we already do client-side validation), and are probably facing some hacking attempt. By discarding invalid commands we tighten the security without any penalties for the client-side experience.

Business Rules

In addition to the validity of the input data, organization would have additional business rules that define correctness and outcome of an operation. These additional rules might depend on the current context. Examples of these are:

• NewCustomer from USA must have address that makes sense for the postal services
• NewCustomer can't use email from the disposable email service.
• NewCustomer from Russia should be granted a welcome bonus of 50RUB on the account.

Checking these things at the UI level would've been too complex and would've required coupling to the additional services. So this is where separation between the validation and business rules in CQRS comes into play - we run this business logic at the server side. It does not merely answer the question "is the command correct", but rather "the command is correct, what does it do?"

For example, if the customer uses an email from the disposable email service, we could suspend the registration process and send him a validation email with the emphasis on no-spam policies in the company. American customers might get a long-running check against the US Postal address verification service. Outcome of a failed checked might be the extra validation step in the welcome email.

We could even have some plugged workflows defined by the DSL or visual designers.

All this logic might fit into the handler that is responsible for processing the command; or it might be fired by subsequent handlers. Anyway this process of business rules obviously requires more CPU power, than a simple validation and could become a bottleneck in the classical architecture. Luckily with the command-query segregation approach we don't have the problems:

• Commands are queued and could be processed by the server at it's own pace;
• Commands could be executed against different servers, based on the aggregate key (sharding). This means, that the aggregates are also distributed;
• CPU-scaling could be dynamic (a characteristic of the self-tuning applications) and adjust to the actual processing load;
• Messaging guarantees that the command will get it's chance to be processed even if the database is deadlocked or the entire server is down.

Corner Cases

Additionally with the validation and expected business scenarios, we would need to explicitly handle that tiny percent of commands (less than 0.1% probability) that were correct at the moment of creation, but happened to become invalid at the moment of handling. This handling is really important, as it provides a safety net against potentially stressful situations that could create losses for the business.

We handle such situations within the business context.

For example, customer's email was unique within the email cache, but is no longer unique at the moment of adding the row to the actual database. Then the desired business outcome might be sending a message to that email, saying:

There was another customer registration on this email, while we were processing your request. We can't proceed, but here are the details. You can create a new customer (link) or update the details of your existing registration (link).

Another possible and more complex situation is when we add items to the shopping cart. At the moment of ordering, we've got everything in the inventory. What do we do, if the order command arrives after somebody else has bought same items from the stock, and they were the last ones? Probability of this is quite low (since it usually takes a few moments to process the order), but we should handle it explicitly at the server side.

There are a few options. First one - send message (notification or email) to the customer, apologizing for the problem and giving up. Unfortunately this would result in a potential customer walking away. Let's try to avoid this.

Smarter approach works within the business context and rules. What do sales people do, when they don't have something? They quickly order the item from the supplier and apologize for the delay. We could do exactly the same thing here - extend the shipping time estimate, say sorry to the customer and give him a discount for the next purchase, if he agrees to wait a little bit for the current order to arrive. Not only we increase the probability of keeping customer happy, we might still make some profit out of this uneasy situation after all.

Under normal conditions this workflow would happen at extremely rare cases. However if we get sudden visibility, run some unexpectedly effective promotions or simply sell really popular item (like iPad), then we might run into the situation where:

• There are a lot of orders within the first 30 minutes after the the item goes on sale.
• We accept all these orders, but servers are stressed and it takes some time for them to process (or it takes some time for the cloud fabric to detect the load and deploy the additional servers).
• Within this interval a few more orders are created.

Anyway nothing bad will happen here, since we will ship all the items from the stock, while immediately handling customers that were a few seconds late (potentially keeping them and making some profit). Also, during all the time, the client UI will stay rather responsive, since it does not face any CPU-heavy processing or database-level deadlocks.

This scenario is slightly better, than the situation when the entire web site (built with a classical SOA architecture) goes down under the stress, isn't it?

Summary

As we can see, in the approach based on CQRS we logically separate Validation from the Business Rules.

Validation (in a task-based UI) ensures that a command is valid within some limited context. Validation is executed on the client side, updating all the UI with the details of any problems found (so that the user could fix them). Valid commands are sent to the server, which runs validation again, followed by the actual business logic.

This article is a part of xLim 4 body of knowledge (CQRS section). You can subscribe to the feed to stay tuned to all the updates.

What do you think about all this? Do you use validation and business rules in your solutions? Have you faced similar challenges in your projects?

Comments

[Greg Young]

I think its important to note that although cool to do this there are a few things to remember.

1) That "still have to verify later" thing for the .1% is expensive to do development wise (requires more knowledge)
2) Very few systems < 1% need to focus heavily on scalability of the write side. It can generally be scaled vertically or diagonally at a reasonable cost
3) This results in a large amount of duplication of logic in the client.

[matt kocaj]

I agree that there is a lot of room to expand on the separation of concerns for validation and business rules. Here are some of my thoughts after reading:
- I think the 0.1% issues you and Greg (above) referenced are concurrency related and not a business rules (domain) concern. These issues are specific to the implementation of the data layer and the persistence engine (database). Concurrency should be separated from the domain logic (business rules) and the issues described handled there and not with validation etc.
- From my understanding of DDD and software architecture in general i believe that Business Rules or Domain Logic will exist in many parts of your application in reality but in theory we would prefer to keep them as centralised as possible. I think that the domain should live with the domain objects and well designed domain validation may or may not be so tightly coupled with the domain logic as it seems you are suggesting it should be ("the command is correct, what does it do?").

Anyway.. that's my view. Comments on my DDD validation experiment?

[Dmitri Nesteruk]

LOL @ Russian customers using pigeons. On a more serious note, I'm really enjoying your blog. Keep up the good work!

[Rinat Abdullin]

@matt

replied to your SO comment. If you need to scale your application without paying immense costs, then you need to keep this in mind everywhere, even in the domain logic. It is extremely easy to create code that will "scalable" only by buying prohibitively expensive hardware.

[Carl Hörberg]

Interresting! How do you handle your requestsIds? Store them in the DB or keep them in some in-memory store?

[Rinat Abdullin]

Carl,

For the web polling purposes requestIds (with the actual responses) could be published to some cheap and inherently scalable media (like Azure table storage). Between the moment of requesting and publication they travel in the body of message. That's the classical CQRS approach.

[Hendry]

Thanks for writing this.
Don't the customers want to know immediately when they entering wrong postcode or entering unaccepted email address? So they can fix their postcode typo or try again using another email address straight away? (i.e., not having to walk away from the customer-service kiosk, go home, and find the validation-message in his email inbox 3 days later, and drives back to the store to fix the typo, and expecting to get another validation message via his email). This has always been so far the recommendation for business-rule checking in cqrs' command-bus pattern.
Do you have any thought about how to provide business-rule error message to the user when you design your application using this command-bus pattern?
Thanks

[Rinat Abdullin]

Hendry,

"3 days later" validation message happens only in the extremely rare cases, when the same email is registered within a few seconds (minutes) concurrently. In all the other situations, we can get the pre-validation immediately (by running sync queries against the server).

Re providing all the other business rule results with the command bus. Basically the trick here is not the technology, but rather redesigning and rethinking your UI in such a way, that principles of "validate immediately, get results slightly later" start working. Sometimes this ends up in the area of pure psychology. Also, normally the delay in business rule processing is just within a few seconds. So if the result is really important (5% of the cases probably), one could display "waiting for the result" blocking UI.

[Hendry Luk]

I didnt mean the system sends the message in 3 days. But it might take 3 days before the customer actually find it in their email inbox (not all customers are computer addicts :P).
And I'm not talking about system where the customer just access directly via browser. The system you're developing can be the one used in a department store, or when a customer buys grocieris in a local supermarket.
If the transaction is rejected due to some kind of business rule violation (e.g. wrong postcode, wrong credit-card details, maximum limit on certain products, blacklisted proof of identification, etc), you can't just silently reject this transcation and send a notification via customer's email so he can read it conveniently when he gets home, asking him to rectify the transaction... that he reads 3 days after he has finished the loaf of bread he bought from the supermarket.

[Rinat Abdullin]

Hendry,

If it is applicable to the situation (from the business context standpoint), you could simply implement the request-response interaction here, where the department terminal sends out the billing command and then blocks the UI, waiting for the result (showing "your order is being processed"). Note, that the server (enterprise-wide or company-wide) still does not return anything except the command acceptance code. Result might be retrieved by pub/sub or simply polling the query storage every few seconds. Server will be responsible for calling all long running services and carrying out the transaction.

We could also try to factor in costs and risks here, on a case by case basis, waiting for the results for all commands above certain sum of money (or using some other criteria, like product lists) and just sending out the command for all the rest.

Alternatively, based on the business case, CQRS might be not the most efficient architecture, as opposed to the plain old RPC.

[Hendry Luk]

Thanks Rinat.
Is that why you store the request ID in the client side? So the client can poll/subscribe for the exception/acknowledgement for that specific request ID?

[Rinat Abdullin]

Hendry,

In essence - yes. If client generates the ID (Guid, CombGuid, HiLo etc), it could proceed as if the command was processed (which is 99.99% cases) or introduce a slight delay (i.e.: "thank you, your order has been created, please click here to proceed" in web apps) before showing actual query results for the new ID.

Note, that this is not a pub/sub or query polling (both of which should be avoided on the client-side unless really necessary).

[w3cvalidation]

Nice information, I really appreciate the way you presented.Thanks for sharing..