Software Design Blog

Journey of Rinat Abdullin

Introducing Lokad Data Platform

Past few weeks were extremely busy for Lokad in many areas. I guess, that’s how you feel when a start-up starts lifting off.

One of my personal priorities was focused on a new Lokad project called Data Platform. If you are interested in business details, there is a blog post about it with a nice infographic.

Data Platform is essentially is going to be:

  • methodology and guidance on bringing together “big” data in organization and making it easily consumable across the same organization. All this - at a fraction of the cost and complexity usually related to doing the same with outrageously expensive “enterprise” setup;
  • open source reference project demonstrating how to aggregate and process relatively complex domain data either in a local data-center or in windows azure cloud;
  • should you need this - consulting, teaching and support from Lokad and partners on both technological setup and on details of details of dealing with the business intelligence;
  • as you can guess, a lot of details and concepts will explained in greater detail in due time within this blog and podcast.

From the technical perspective there is absolutely nothing new or innovative in Data Platform. A lot of high-tech companies have been doing “big data” and “cloud computing” for years, starting from high-frequency trading and up to the large hadron collider. If you seen Greg’s Event Store, Lokad.Cloud and Lokad.CQRS projects, you already know what to expect.

From the business perspective, situation is totally different. All this publicly available knowledge is as good for a vast majority of our customers as if it were developed and used on the dark side of the moon. It’s too far and too hard to get people who can handle it.

As you can guess, IT enthusiasts would not normally feel excited about managing data coming from cash registers in a retail company. This slows down technological progress a lot in such companies and creates an opportunity for sales-oriented software and consulting companies. These companies jump right in, selling mediocre but expensive stuff and not really solving the problems.

Such situation is the reason why 20GB of sales history is actually considered to be big data in these companies.

That’s what we are trying to break with DataPlatform. It’s too painful for us to help customers with the business intelligence (our core competency), when we can’t even get the data out of “expensive stuff” - it either breaks, slows everything down or requires literally months of effort to extract data.

With Data Platform we want to show how it is possible in some specific cases to replace “expensive stuff” setup with something much simpler and cheaper, while getting even better performance and reliability. For instance sometimes a 1000000 EUR cluster can be replaced with a few virtual machines on Windows Azure for reliable storage of terrabytes of data, while having decent throughput and dead-simple way to consume such data.

The idea behing DataPlatform is to make it extremely simple and cheap to give it a try (at least as a way to finally stop discarding valuable business history data). Think about dead-simple event store (even simpler than the one used in the latest Lokad.CQRS) and projections on top, designed to get the most out of Windows Azure and store gigabytes/terrabytes of messages.

That’s a small and obvious step for a lot of people reading my blog, however for a lot of enterprise companies that’s a huge leap. If it were to be made, then suddenly there is a huge number of opportunities for moving forward: enterprise-wide Domain-Driven Design, efficient development organization around CQRS models, continuous delivery of certain elements, inherent scalability for these reports, capabilities for real-time and occasional connectivity etc.

Lokad Data Platform

Plus Lokad can stop wasting time on data integration issues and actually focus on business analytics.

PS: If all works out, DataPlatform will be pluggable into Lokad.CQRS as another event storage engine.