The records inside a Hadoop cluster constitute are only part of its overall information footprint. The other portion is made up of the metadata that is produced as those records change and evolve over time, which Syncsort Inc. wants to make more accessible by launching a new log management capability for its DMX-h tool.

Information about the operations that the ETL system performs in Hadoop can now be exported as a standardized .csv file described as compatible with practically any external data management platform. Since DMX-h serves as the lynchpin of many Syncsort-powered environments, the output can provide insight into key aspects of the analytics workflow such as where the processed records come from, how they’re ingested and what modifications they undergo as part of the process. Syncsort says that customers are able to put this information to use in a wide range of different areas.

For starters, DMX-h’s operational logs can provide information about issues in the analytics pipeline that may be inaccessible or difficult to uncover otherwise. And since the metadata is available a .csv format, administrators are able to export it to an external analytics tool like Splunk Inc. instead of having to manually dig through the raw output for problem indicators. This also makes the new log management functionality useful for performing audits aimed at uncovering security and compliance problems in a Hadoop deployment.

The latter benefit is especially important for companies in regulated industries like finance and healthcare, which constitute a sizable percentage of Syncsort’s customer base. Such organizations are subject to strict privacy laws that require keep a close eye on how customer records are processed. By opening up DMX-h’s metadata, the vendor is making it much easier to assemble a complete picture of the Hadoop analytics lifecycle while easing everyday diagnostics tasks in the process.

Image via Pixabay