Microsoft Corp. today open-sourced Trill, a data processing engine capable of analyzing billions of events per second.

Used in Microsoft’s Azure cloud platform, the project adds to the growing list of internal technologies that the company has shared with the world. Its open-source portfolio includes several artificial intelligence tools, including a system called Infer.NET that is likewise used in Azure, as well as the code for the popular .NET development toolkit.

Trill is a particularly significant addition. The engine is versatile enough to process both real-time and historical data, a combination that only a few existing open-source tools offer. Trill pairs this multimodal analytics support with a set of other features that Microsoft said is not fully matched by any other system.

The engine’s standout capability is its performance. Trill can ingest events, for instance, temperature change logs from soil sensors, by the billions per second. It provides more moderate but still impressive speeds of 10 million to 100 million events per second when performing advanced operations, such as extracting the highest temperature reading from a batch of sensor logs.

“Trill works equally well over real-time and offline datasets, achieving best of breed performance across the spectrum,” James Terwilliger, a principal research engineer at Microsoft’s Azure group, wrote in a blog post. “This makes it the engine of choice for users who just want one tool for all their analyses.”

Another feature that sets Trill apart from the pack is its customizability. The engine provides the ability to change the intervals at which incoming data is processed, which enables users to granularity manage latency levels. Companies can also extend the native query language with analytic functions tailored for their requirements.

Microsoft uses Trill for a wide range of applications. The engine powers the Azure Stream Analytics service, supports the company’s Bing Ads business and helps its video game development unit analyze data generated by players.

The fact that Trill is so widely used within Microsoft could be an important selling point for potential enterprise adopters. It means that the engine is battle-hardened enough to be used in production environments.

In the blog post announcing the project, Azure Data engineering manager Zhong Chen hinted that companies won’t necessarily have to deploy Trill on their own to take advantage of its features.

“Azure Stream Analytics went from the first line of code to public preview within 10 months by using Trill as the on-node processing engine,” Chen was quoted as saying. “In the near future, we are considering exposing Trill’s programming model through our user-defined operator model so that all of our customers can take advantage of the expressive power.”

Photo: Microsoft