Cloudera has acquired an emerging startup called Xplain.io Inc. to help organizations run their analytic clusters more efficiently. The deal could provide it with needed differentiation amid increased competition from rival Hadoop distributors that are making substantial gains on other fronts.

Former Oracle engineer Anupam Singh founded Xplain.io in his back yard 15 months ago together with Cisco veteran Rituparna Agrawal to bring one of the most important concepts from the world of relational databases into the new world of Hadoop. The startup has developed a kind of cost-based optimizer that analyzes logs generated from queries to identify execution paths and map out the most efficient routes for common operations.

The optimizations appear in the form of recommendations on Xplain.io’s sleek management console that administrators can implement automatically without having to make any manual adjustments to their existing data models, a process that is not only complicated but often risky as well. Added up across the millions of queries that some organizations run against their Hadoop clusters, those individual improvements can amount to a potentially massive speed boost.

That has the immediate benefit of enabling business users to receive results faster and becoming that much more productive, which can empower the entire business to react quicker, especially as analytics and Hadoop in particular take on an increasingly central role in decision-making. The technology is already in use with some 100 organizations, most of which are running Cloudera’s distribution of the data-crunching platform.

Both Hortonworks Inc. and MapR Technologies Inc., its two biggest rivals, implement Apache Calcite as the cost-based optimizer in their respective SQL-on-Hadoop technologies. The former even went as far as hiring the project’s creator. Cloudera has cost based optimization in Impala since 2013.

The friendly graphic interface of the startup’s solution pushes Cloudera from the back of the pack to the lead, at least when it comes to usability. No terms were disclosed for the deal other than that Singh and the rest of the Xplain.io team are joining the company, which is something of a victory in itself in view of the massive shortage of data science skills.

Photo credit: Magnifying Glass via photopin (license)