Big data software provider Cloudera Inc. is forging a partnership with its biggest investor to advance its footprint in the security business — without formally announcing a product.

The company last week announced support for Apache Spot, an incubating project contributed by Intel Corp. that gives organizations a single consolidation platform for security data that can support an ecosystem of machine-learning applications. Intel donated Spot to the open-source community in September with support from Cloudera. The chip maker invested $740 million in Cloudera nearly three years ago as part of a massive fundraising round.

Spot is an attempt to address the notorious fragmentation that characterizes enterprise security operations centers, each of which typically maintains its own data models and standards for capturing information from sources such as server and network logs, directories and endpoint devices. Spot creates a single, shared open data model that any vendor or security application can plug into. Spot also works with Apache Spark to enable parallel in-memory processing to be applied for machine learning and artificial intelligence analysis.

“Our core proposition is that security organizations are competing against an adversary that’s highly coordinated and collaborative,” said Sam Heywood, director of cybersecurity strategy at Cloudera. “It’s no longer a SQL-sized problem. This is a big data problem.”

Not a product

Cloudera is being careful not to announce that it’s becoming a security vendor because it already has an ecosystem of more than a dozen certified third-party developers that build security products on top of its big data platform. “We sell a big data platform, and that’s what we’re going to continue to sell,” Heywood said. “With Spot we’re coming to an agreement on how data is stored. If partners see value in delivering analytics on that, that’s fantastic.”

The lack of a consistent data model or schema for capturing security information has frustrated development of security software based upon machine learning algorithms, Heywood said. Enterprise adoption of Spot can accelerate that process by giving independent software vendors both a level playing field and larger potential pool of customers.

“We want people to establish a security hub rather than a security lake,” Heywood said, referring to the large but somewhat amorphous pools of data that organizations extract from Hadoop. “One key part of Spot is open data models, which provide a taxonomy for ingesting the universe of data into a single data set organized around core logic that’s relevant to the end user.” In other words, it creates a common set of definitions and formats.

Cybersecurity technology based on machine learning is considered one of the most promising areas of threat detection and response, but models can be difficult to set up and tend to be unique to each situation. Bringing Hadoop to bear offers not only a broader range of data but also deeper history.

“No matter how many panes of glass security professionals are looking at, they only have partial visibility,” Heywood said. “At best they only have a couple of months of historical data.” That’s where Hadoop can be a game-changer, he noted.

Apache Spot logo via Cyphon Design