Netflix is not only the world’s largest provider of streaming media, but also one of the most prominent contributors to the Hadoop ecosystem. Jeff Magnusson, Netflix’s head of Data Platform Architecture, discussed the open-source technologies his company uses to keep track of consumer trends in an interview with SiliconANGLE NewsDesk Head Writer Winston Edmondson.

Winston starts the session by asking about Netflix’s famed corporate culture. Magnusson says that the firm is constantly on the prowl for intelligent people, including analysts and other professionals who don’t necessarily possess advanced technical skills. Netflix’s engineers are tasked with developing tools that can help their less tech-savvy colleagues gain insights from data without concerning themselves with the underlying technologies.

Magnusson highlights that a lot has changed since the days of DVD deliveries: back then, most consumers didn’t interact with Netflix after ordering titles. Today, viewers generate so much data that members of his team have to use custom visualization software just to keep up.

Netflix utilizes Apache Pig for recommendations, video optimization and other important workflows. Pig syntax is highly abstracted, Magnusson explains, but raw code is often difficult to understand due to the complexity of certain scripts. He tells Winston that Netflix solved this problem with a visualization tool called Lipstick. The homegrown program transforms code into directed acyclic graphs, or DAGs, that make it easier to spot bugs in large projects. These charts also enable developers to view MapReduce jobs as they execute.

Mangusson says that Netflix’s appetite for data will continue to grow as it expands to new markets. He predicts that real-time technologies such as Storm will shape the future of the Big Data ecosystem and revolutionize user experience in the process.

Check out the video below for the full interview.