Apache Spark claims to be the largest and most active open-source big data project in existence, but is it delivering on the big data promise to businesses in the real world?

At the Spark Summit East 2017 conference in Boston, George Gilbert (pictured, left) of theCUBE, SiliconANGLE Media’s mobile live streaming studio, argued that Spark has not replaced the more venerable open-source big data software Hadoop, but it has carved out a huge chunk of what Hadoop has traditionally done. (* Disclosure below.)

As to whether it can solve all of the big data problems companies face today, Gilbert said, “It’s still teething. And every release, which is approximately every 12 […] months, it gets broader in its applicability. So there’s no question that everyone is piling on, which means that will help it mature faster,” he said.

Gilbert went on to say that just because Spark has not fully matured does not mean that it’s not making applications substantially better right now. Instead it’s in the way that companies use it, he explained.

“Ultimately, it’s not these infrastructure analytic frameworks and that sort of thing; it’s the machine learning model in its flywheel feedback state where it’s getting trained and retrained from the data that comes in from the app,” he said. “And that beating heart, that living model that’s always getting better, that’s the core value-add, and that’s going to belong to end customers and to application companies.”

Dead tree, meet living model

Dave Vellante (pictured, right), Gilbert’s co-host, noted how data use cases are popping up in unexpected domains.

“A lot of those outcomes are related to adtech and selling and retail-oriented use cases,” he said. “But we’re hearing more and more around education and deep learning and affecting consumers and human life in different ways.”

One use case the analysts discussed was that of McGraw-Hill. Known as a publishing company dealing in printing presses and paper, it has used data to develop, just in the last two years, a high-tech education software arm.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of the Spark Summit East 2017 Boston. (*Disclosure: TheCUBE is a media partner at the conference. Neither Databricks nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo by SiliconANGLE