NEWS
NEWS
NEWS
Demystifying the complexities of Spark and Hadoop | #SparkSummit
There is a lot of confusion around about Spark and Hadoop, but Matthew Hunt, head of Big Data at Bloomberg LP, thinks this is only due to a lack of understanding.
In an interview at Spark Summit East 2016 at the New York Hilton Midtown in NYC, Hunt talked with Jeff Frick and George Gilbert, cohosts of theCUBE from the SiliconANGLE Media team, to demystify the complexities of working with Spark and Hadoop, as well as provide an analysis of how Spark and Hadoop fit within the market.
Hadoop developed to answer a practical problem
In a conversation that delved deep into the mechanics of Big Data frameworks, Hunt explained the differences between Spark and Hadoop, answering where they come from, where they are going, what they do today and how they fit together.
For example, Hadoop was created to resolve a practical problem — how to download and index the web economically — by engineers rolling up their sleeves to solve real issues. As the platform grew, layers were added on top and the complexity and number of tools grew, yet the instruction set remained basic. Spark was developed with a more complicated instruction set that increased its speed yet simplified the many tools of Hadoop into one.
Spark takes language constructs and makes them performant
Hunt gave a practical example to help viewers create a mental model of Spark: When you compile a program, you write code, hit a button and the computer turns it into machine-level instructions. The same thing happens in Spark — it has an instruction set under the hood where whatever you are writing in is transformed.
Gilbert summarized this as Spark “taking language constructs and making it performant.”
Mental model shift
People assume what will be in a fast computation engine, but they are often not accurate and this causes confusion, said Hunt. “There is a mental model shift, and there are pieces that haven’t come together yet to make that happen.”
Watch the full video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit East 2016. You can also join in on the conversation by CrowdChatting with theCUBE hosts.
Photo by SiliconANGLE
Since you’re here …
… We’d like to tell you about our mission and how you can help us fulfill it. SiliconANGLE Media Inc.’s business model is based on the intrinsic value of the content, not advertising. Unlike many online publications, we don’t have a paywall or run banner advertising, because we want to keep our journalism open, without influence or the need to chase traffic.The journalism, reporting and commentary on SiliconANGLE — along with live, unscripted video from our Silicon Valley studio and globe-trotting video teams at theCUBE — take a lot of hard work, time and money. Keeping the quality high requires the support of sponsors who are aligned with our vision of ad-free journalism content.
If you like the reporting, video interviews and other ad-free content here, please take a moment to check out a sample of the video content supported by our sponsors, tweet your support, and keep coming back to SiliconANGLE.
