BIG DATA
BIG DATA
BIG DATA
Can this tiny SQL engine surpass Spark’s data streaming efforts?
Last summer Apache Spark claimed to be the best (or at least easiest) engine for data streaming with its 2.0 release. Now the Chinese company Transwarp Technologies Co. is challenging that with an engine that can stream a single event at a time — something Spark still cannot do.
The company’s Co-founder and Chief Technical Officer, Yuanhao Sun, said it has accomplished this by changing the Spark execution model to be event-driven, and tweaking SQL to be smaller and faster.
“That tiny SQL engine is about five to 10 times faster than Spark 2.0,” he said at BigData SV 2017 in San Jose, CA.
Sun told John Furrier (@furrier) and George Gilbert (@ggilbert41), co-hosts of theCUBE, SiliconANGLE Media’s mobile live streaming studio, that the company’s just announced 5.0 release offers this along with other new features. (*Disclosure below.)
Version 5.0 has also integrated Kubernetes (an open-source system for automating deployment, scaling and management of containerized applications) to enable Docker containers for isolating workloads. It additionally has a set of tools to help companies migrate to Hadoop.
Customers can build what Sun calls a “data mart,” which allows them to interactively query their data.
The feed at speed
Transwarp Technologies does draw on Spark for some of its streaming capabilities, but adds its own ingredients to enable low-latency streaming one event at a time. This is valuable to IoT applications with continuous streams of data coming in, Sun explained.
“The engines are very low latency, but they also need very complicated models on top of streams, so that’s why we are providing full SQL support and machine learning support on top of streaming events,” he said.
Credit card fraud detection is one area where this engine could be useful.
“We are also embedding the CEP [complex event processing] capability into our streaming engine so that you can use a pattern to match the events and to send alerts,” Sun concluded.
Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of BigData SV 2017. (*Disclosure: Some segments on SiliconANGLE Media’s theCUBE are sponsored. Sponsors have no editorial control over content on theCUBE or SiliconANGLE.
Photo: SiliconANGLE
Since you’re here …
… We’d like to tell you about our mission and how you can help us fulfill it. SiliconANGLE Media Inc.’s business model is based on the intrinsic value of the content, not advertising. Unlike many online publications, we don’t have a paywall or run banner advertising, because we want to keep our journalism open, without influence or the need to chase traffic.The journalism, reporting and commentary on SiliconANGLE — along with live, unscripted video from our Silicon Valley studio and globe-trotting video teams at theCUBE — take a lot of hard work, time and money. Keeping the quality high requires the support of sponsors who are aligned with our vision of ad-free journalism content.
If you like the reporting, video interviews and other ad-free content here, please take a moment to check out a sample of the video content supported by our sponsors, tweet your support, and keep coming back to SiliconANGLE.
