IBM has legitimized Spark and wants to shift the axis of Big Data processing, the way it did the operating system world when it committed to Linux across all its platforms 16 years ago, writes Wikibon Big Data analyst George Gilbert.  Back then the competition for Linux was several proprietary flavors of Unix and the Microsoft server platform. Today the competition for Spark is Hadoop.

Spark is a unified engine that can handle real-time, interactive and batch processing, applying machine learning, graph processing, SQL and streaming capabilities (see figure above). This makes it simpler than Hadoop, which is an ecosystem of tools unified around the YARN cluster resource manager on top of HDFS. Each distro has a slightly different set of component products.

Spark shines in applications that involve multiple data environments, such as omni-channel marketing. Spark can use batch machine learning to profile each customer from the historical database, then apply streaming data analysis to update individual profiles in real time as customers browse products. Spark can then use that information to influence the customer’s purchase choices.

Hadoop’s advantage is that each of its individual tools is optimized for a single function, making them more efficient at that particular task than Spark can be. For example, the Presto SQL data warehouse will probably have greater capacity and speed than anything built on Spark, writes Gilbert.

As a result, Hadoop is not going away. However, with IBM’s commitment Spark is no longer an academic project, Gilbert writes, and it may unify many of the processing layers that are separate in Hadoop.

The full Alert and other Big Data analysis is available on the Wikibon Premium Web site.

Graphic ©2016 Wikibon