Security emerged as a central talking point at the Hadoop Summit today after Hortonworks Inc. introduced a set of new features for its distribution of the platform aimed at helping customers protect their deployments more effectively. Standing out in particular is the addition of integration with Apache Metron, an emerging open-source system for identifying malicious activity.

The project is derived from a tool called OpenSOC that was originally created by Cisco Systems Inc. to improve the security of its internal Hadoop deployment and became available under a free license in November of 2014. Its main selling point is a real-time threat detection engine based on another open-source component called Apache Storm that can scan the activity records from an analytics cluster for signs of known malware. Upon finding a positive hit, Metron automatically takes action to try and block the attack before alerting the security department, which is then able to dig into the log data from the incident from a deeper understanding of the situation.

A particularly savvy organization could also harness the functionality to defend its Hadoop cluster against more serious threats like zero-day vulnerabilities by adding custom detection algorithms to the tool. According to Hortonworks, Metron thereby makes it possible to dramatically reduce the time it takes to identify breaches while also cutting the amount of manual effort involved in the process. And less operator involvement in turns translates into a lower risk of human error, which should be a major selling point for customers that are using their analytics clusters to process sensitive internal data.

Hortonworks hopes to increase the appeal of its distribution even further by adding integration between two existing security components: The Apache Ranger access control system and Apache Atlas, a record classification tool. As a result, organizations can now individually restrict the usage of every dataset in their Hadoop clusters based on its sensitivity. An insurance company, for instance, could employ the functionality to disallow the transfer of patient files to third party analytics tools. It’s also possible to account for factors such as the location of an analyst trying to access the records and regulatory time limits on information retention.

Hortonworks rolled out the new security functionality alongside several other smaller updates focused mainly on improving the usability of Hadoop, especially for data scientists. The company also announced that long-time ally Pivotal Software Inc. is finally becoming a reseller after a more than three-year collaboration, which should provide an important boost amid the increasing competition from rivaling distributors like Cloudera Inc.

Photo via Pixelcreatures