Yahoo! and Benchmark Capital launched Hortonworks today with the mission to revolutionize and commoditize the storage and processing of big data via open source.

It’s not unexpected news. There has been talk over the past few months of a new company coming out of the venture capital group and Yahoo! that played off Apache Hadoop, the distributed storage and analytics technology. This venture is the outcome.

What’s significant is how it departs from what Yahoo! has traditionally done. Hortonworks is largely a services play and puts it in competition with the largest enterprise technology companies in the world.  My bet is that Hortonworks may just beat them, too, and give Yahoo! a new edge that would serve it well.

Market leader Cloudera might think differently as they have amassed a large brain trust from Yahoo over the years and by far have the most experience.

Forming Hortonworks makes sense for a few reasons:

• Yahoo! has a deep understanding of Hadoop. Doug Cutting joined Yahoo in 2006 and set up a 300-node research cluster at the search giant. From that research, as O’Reilly Radar points out, Yahoo! has developed a number of use cases, including e-commerce customer segmentation; A/B testing; fraud detection; machine learning and medical research. Cutting has since gone on to join Cloudera which is the most widely used Hadoop distro. Here’s Doug in an interview on The Cube at last year’s Hadoop Summit:

• Big data is a significant trend that will continue to gain momentum over the next several years.
• The services angle is a right fit for Yahoo! This positions the company as a visionary technology services provider. In turn, that experience can be plowed back into Yahoo! for its own future development.

Architects and developers from the Yahoo! Hadoop engineering group make up the team at Hortonworks. These are the same developers who developed Apache Hadoop from its earliest release. The team is also made up of  the largest concentration of Apache Hadoop committers. These are people who have contributed a sizable amount of source code to core Apache Hadoop projects including HDFS, MapReduce, Pig and others.

The Services Angle

Open-source technology is key to Hortonworks but it is the services that will make the business a success. The goal is to make it easier, more manageable and cost-effective to store, manage, process and analyze the data that flows into and throughout organizations.

Hortonworks will include “support subscriptions, training and certification for developers, architects and administrators at enterprises, systems integrators and technology vendors such as ISVs, OEMs and service providers.”

From the Partner page:

Hortonworks Technology Partner Program – for technology vendors (ISVs, OEMs and Service Providers) that need Hortonworks technical enablement and support to better utilize Apache Hadoop within their product offerings.

Hortonworks Certified Training Partner Program – for geographically focused training organizations that need education enablement and joint marketing to deliver public and private training courses.

Hortonworks Certified Systems Integrator Partner Program – for systems integrators building Apache Hadoop solutions for clients that require Hortonworks technical enablement and joint marketing assistance.

Can Hortonworks Beat the Enterprise at its Own Game?

Hortonworks clearly shows how a data driven company like Yahoo! can do two things:

• It can help companies both large and small become data driven companies just like Yahoo!
• And it can beat the enterprise giants at their own game.

It’s undoubtedly true that Yahoo! faces competition from the likes of Cloudera, IBM and new startups such as MapR.

But there is a sense that Yahoo! has developed something significant that comes from the heart of the company. And it shows why innovative technology and services are such a potent combination in this fast changing world.

But Hortonworks is facing a big battle. The enterprise giants have battalions committed to big data efforts. To succeed against this crowd, Hortonworks will have to show it can build a business from scratch and far outperform its well armed competitors.