Microsoft VP of Data Platform T.K. Rengarajan Speaking At PASS Summit 2014

Microsoft Corp., the poster child of proprietary software, has developed a sudden appetite for open-source technologies. Barely three days after revealing plans to make future versions of Windows Server compatible with the Docker container engine, which currently only runs on Linux, the Redmond giant is rolling out support for Apache Storm for its Azure infrastructure-as-a-service platform.

Originally developed at a little-known social media analytics provider with an affinity for open-source software called BackType Inc., Storm fell into the hands of Twitter Inc. when it acquired the startup in 2011 and was promptly donated to the Apache Software Foundation following the deal. Written in Clojure, the event processor offers a fast and reliable way to ingest vast amounts of fast-moving data such as tweets, live media and sensory transmissions in Hadoop. It graduated from incubation into a top-level project at the beginning of the month.

Owing to its commercial roots, Storm boasts a higher level of maturity than many of the other components in the upstream ecosystem of the data crunching platform. Most notably, the engine includes fault tolerance functionality that automatically reacts to infrastructure failures and mechanisms guaranteeing that every last tidbit of incoming information is processed at least once.

Coupled with the tremendous momentum of the project, those features made Storm the most obvious option for Microsoft to match the real-time analytics capabilities offered by arch-rival Amazon Inc., whose Kinesis service already supports the engine. The choice was made even easier for the company in face of the alternatives: simply ignore the growing trend of stream processing in Hadoop or spend months developing a proprietary solution for analyzing real-time data that would not only compete with Kinesis but Storm as well.

Microsoft Corporate Vice President of Data Platform T.K. Rengarajan wrote in a blog that his firm is folding the project into HDInsight, a version of Hortonworks Inc.’s freshly updated Hadoop distribution adapted to run on Azure. The Redmond giant is also rolling out new pre-packaged machine learning capabilities aimed at making it possible for users of the infrastructure-as-a-service platform – most of whom aren’t trained statisticians – to take advantage of the integration, which is currently in preview.

Photo credit: Striking Photography by Bo Insogna via photopin cc