With event-by-event data streaming and serverless application development, has the Apache Spark open-source community readied real-time, continuous applications for enterprise use?

“They definitely turned up the volume on what they can do with continuous apps,” said George Gilbert (@ggilbert41) (pictured, right) in a discussion with David Goad (@davidgoad) (pictured, left) and James Kobielus (@jameskobielus), co-hosts of theCUBE, SiliconANGLE Media’s mobile live streaming studio, during this year’s Spark Summit. (* Disclosure below.)

The shaving of mere milliseconds off network latency now makes a whole new class of applications possible, according to Gilbert.

“The Spark community had their back against the wall with structured streaming that they couldn’t fully provide a true sub-millisecond, end-to-end latency environment,” said Kobielus, noting Apache Flink’s lead around that particular curve.

Now that Spark has caught up, it can drive event-by-event streaming into continuous computation for new applications, he added.

Go-fetch with serverless DevOps

The new serverless application platform from Databricks Inc. (a cloud-based data management service founded by Spark’s creators) might give Spark development itself a streaming flavor, according to Kobielus.

This serverless platform will abstract away much complexity and potential hiccups from building apps on Spark. “You don’t think of it as a cluster of resources — you just know that it’s sort of out there, and you ask requests of it and it figures out how to fulfill it,” Gilbert explained.

So with data in Spark running continuously, this serverless platform could enable “continuous build, deploy and testing DevOps within a Spark machine learning and, now, deep learning context,” Kobielus said.

“The Spark community, as it evolves and matures, needs robust DevOps tools to ‘production-ize’ these machine learning and deep learning models,” Kobielus added, pointing to a growing number of enterprises using Spark for 24-7 application artifacts.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of Spark Summit 2017. (* Disclosure: DataBricks Inc. sponsored this Spark Summit 2017 segment on SiliconANGLE Media’s theCUBE. Neither DataBricks nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE