In the enterprise, the exponential growth of information is driving the need for a consistent and highly scalable data access layer that can keep up with an ever-expanding workload, while also meeting the performance requirements of end-users. LucidWorks is taking a fresh approach to this challenge with a unique development platform that utilizes open source software to let customers create custom applications for querying and processing Big Data.

The Redwood City, Calif.-based startup, formerly known as Lucid Imagination, today introduced a new solution designed to help companies correlate documents, customer records and other unstructured information with machine data. Solr Monitor App for Splunk Enterprise enables admins to “glean insights from search and usage patterns and spot security concerns to improve end user experiences and derive more business value from data-driven applications,” according to the company.

As the name suggests, the platform is geared towards applications that utilize the Apache Solr and Lucene search technologies that power LucidWorks’ flagship offering and the user applications built in it. The tool features analytical capabilities, predefined dashboards, alerts and educational resources, as well the ability to sent NoSQL queries from Solr to Splunk Enterprise.

“LucidWorks is enhancing the Solr experience by combining best-in-class system event management from Splunk with Solr,” said Will Hayes, the chief product officer at LucidWorks. “Splunk now has the power of the most widely deployed open source enterprise content search platform in the world as part of its solution stack, creating the next generation of machine and unstructured data applications. This application bridges Splunk’s machine data with enterprise data in Solr to uncover context and relationships in ways never before possible.”

Relevance in data

What’s interesting about LucidWorks’ approach to enterprise search is its ability to help an end user filter through amassing data, even when they’re unsure of the right questions to ask of that data.  Recognizing patterns is key to highlighting important data points to anticipate the needs of enterprise users.

“One of the key components of enterprise search is relevancy. In a simple context like eCommerce, relevancy is helping shoppers identify items of interest in vast product catalog by comparing shopping patters and item interest with other users to crowd source recommendations,” explains Will Hayes, Chief Product Officer at LucidWorks.

“In the broader enterprise application context, this same approach is used to proactively drive relevant results to users based on interaction and usage patterns but can also leverage additional signals such as user profile, departmental information, geo-locaiton and job function. This allows the application to leverage the search engine to drive results when users aren’t sure what questions (queries) to ask.” Hayes says.

LucidWorks previously partnered with MapR to layer advanced querying and data mining capabilities over Hadoop, accelerating time-to-insight for users.  Relevancy remains central to the products LucidWorks provides in conjunction with its partners, addressing the mounting integration obstacles facing data-driven solutions in today’s enterprise IT environments.  For today’s Splunk integration, the result is a new level of introspection benefitting any business looking to better leverage its machine data.

“Introspection is DoA without automation,” Hayes continues. “Data driven businesses will need to rely heavily on analytics and insight gained by observing how users are utilizing the services and underlying data made available. Reporting, statistical modeling and log analysis help applications determine the ‘next best action’ for users, which is the underlying premise of being data driven. Scheduling these jobs to run models and reports in an automated fashion or even in near-real time enables applications to optimize data decisions automatically, which provides a truly data driven experience for the user.”

photo credit: B Tal via photopin cc