Source Code Reveals: Microsoft Plans Cloud Response to Amazon's Elastic MapReduce

There was a question about cloud computing: Could Windows Azure catch up with Amazon?

Well, it turns that Microsoft could. First, it came up with Windows Azure Platform, then it announced rather competitive pricing for it. As the next step Microsoft has just released DryadLINQ project to public with an academic license.

This project has shed some more light upon future plans of Microsoft in this area.

DryadLINQ is a research project by Microsoft that allows to execute LINQ queries against some cluster. In other words, that the most simple and strongly-typed way to execute MapReduce queries that has been implemented so far.

MapReduce concept has been made popular by Google and gained a lot of momentum since then.

Well-known open source project Apache Hadoop is currently used by Amazon to offer Amazon Elastic MapReduce. It allows businesses and researchers to process vast amounts of data in the computing cloud.

DryadLINQ currently needs Windows HPC Cluster as a host environment. You wouldn't call HPC cluster a commodity. Yet, one line in the released DryadLINQ client code promises the situation to improve later:

case SchedulerKind.Azure:
  this.executionKind = ExecutionKind.JobScheduler;
  throw new Exception("Azure is not supported in this version.");

When that feature really gets implemented, we'll have a .NET equivalent of Amazon Elastic MapReduce. More than that, future versions of DryadLINQ are promised to get released at the CodePlex. This might get us an open-source equivalent of Apache Hadoop but for .NET developers.

This will be quite a jump forward for all the CPU-intensive computations in .NET world. Now we just need for Google to join the crowd and balance the prices. Then cloud bursting scenarios will be much easier for small and medium companies to benefit from.