Big Data is a big field with big promise and big challenges. It seems that for every creative or profitable use-case we see, a snafu with storing or working with data rears its head. Some questions facing companies collecting Big Data are: Who’s going to parse it, prepare it and turn it into useful information? And how will the company pay them?

In terms of corporate budgeting for the data team, “We are hitting a wall,” said Dan Graham, GM of enterprise systems at Teradata Corp.

Graham told John Furrier (@furrier) and Peter Burris (@plburris), cohosts of theCUBE, from the SiliconANGLE Media team, that hiring more people for more data is not an operable model. He added that the data teams are going to need to become more productive, and the software is going to have to get smarter. “You have to automate it,” he said, referring to many data tasks not handled by workers.

The conversation took place during during the BigDataSV 2016 event in San Jose, California, where theCUBE is celebrating #BigDataWeek, including news and events from the #StrataHadoop conference.

theCUBE hosts also spoke with Stephanie McReynolds, VP of marketing at Alation, Inc., who spoke about the possibility of making data tasks social through crowdsourcing and other methods.

Graham’s pick for the future was sensor data (data-collected recording devices, such as cameras and satellites). “Sensor data is about to flood the world and make Big Data look like really small stuff,” he said. “It’s going to dwarf Big Data as we know it.”

All about the algorithm

McReynolds also said that defining the difference between the data and the algorithm is still a challenge when working with businesses. She cited studies showing that 50 to 70 percent of business managers still make decisions based on gut feeling even with data in front of them — and it’s because they don’t trust the data.

“You can send two of your best analysts out on the same inquiry, and they come back with two different answers, because they transformed the data in a different way,” she said. “We’ve got to figure out how to be more descriptive about the algorithm and how to apply it.”

Stay tuned for the full video interview, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of BigDataSV 2016. And make sure to weigh in during theCUBE’s live coverage of the event by joining in on CrowdChat.

Photo by SiliconANGLE