This week, ODPi (a nonprofit organization accelerating the open ecosystem of Big Data solutions) announced that DataTorrent, IBM, Pivotal, SAS, Syncsort, WANdisco and Xavient have committed to the ODPi Interoperable Compliance Program. This program will make it simpler for enterprises to choose and adopt Big Data technologies and ensures these applications are interoperable across a wider range of commercial Hadoop platforms.

Berni Schiefer, IBM fellow, joined Dave Vellante (@dvellante) and Peter Burris (@plburris), cohosts of theCUBE, from the SiliconANGLE Media team, during BigDataNYC 2016 in New York, NY, to discuss the implications of ODPi’s compliance program for the industry, as well as its impact on future Big Data ventures.

Defining Hadoop differently

Vellante asked if IBM is building into a new Hadoop distribution with ODPi as the framework.

“We’ve had [IBM’s] Big SQL for many years … but the market in the distribution market is fragmented. … You can make software portable to different platforms. … Linux standardization [has] made it easier to port and run on different levels of Linux. ODPi is trying to do the same thing for the Hadoop ecosystem,” said Schiefer.

He explained that while the traditional definition of Hadoop is one with HDFS, MapReduce and YARN, people have replaced components with open-source tools, especially Spark. “Today, I think that Hadoop should be redefined as an ecosystem of collaborative tools … very much like Unix became. … We should engineer tools to sit on top of that platform,” said Schiefer.

How ODPi came to be

Vellante asked about how ODPi came about, and wasn’t it unusual for competitors to be working together on a project like this? Schiefer explained that it is not a disconnect between competing at the product-offering level and collaborating [with competitors] to standard SQL for the benefit of clients and the industry as a whole. ODPi works to bring together different companies to make sure the “plumbing” is sufficiently standardized for everyone’s benefit.

Burris asked about IBM’s role in SQL and how that relates to Hadoop. “When I was first introduced to Hadoop, it was kind of like the no-SQL space. … It’s kind of turned into the new SQL space,” Schiefer said. “Not that SQL is perfect, but it is powerful, there are abundant skills and there’s lots of tools. Those are things that really help people get going.”

Schiefer concluded: “The ‘holy grail’ of Big Data is not about the data itself, it’s just sitting there doing nothing, [but rather] to get insight from the data.”

Watch the complete video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of BigDataNYC 2016.

Photo by SiliconANGLE