The driver in creating the Systems of Intelligence analytic data pipeline is cutting the time between capturing data and making a decision, writes Wikibon Big Data Analyst George Gilbert in part two of his series on adding real-time data analysis capabilities to the batch business intelligence (BI) suite.

Streaming analytics may sound like the answer to everything, but in fact there are tradeoffs, particularly in the accuracy of predictions. Basically, RDBMS analysis, although getting faster, is still too slow when the requirement is near real-time analysis to support demanding applications. Streaming data analysis is faster but also less accurate. Whereas traditional BI-type analysis can incorporate information from multiple sources, including patterns of behavior and other recent activity, streaming analysis has to depend on the information immediately available.

To illustrate the tradeoff, Gilbert takes the example of fraud analysis. A traditional DBMS analysis system can identify fraud with a high degree of accuracy, but it cannot do that before the transaction is completed. The best it can do is block further use of a credit card once it recognizes a transaction with a high probability of fraud.

Streaming analysis, on the other hand, can make a decision – do nothing or bounce the transaction – before the transaction is completed. But it has a higher degree of inaccuracy. That means some fraudulent transactions will slip through, while some legitimate transactions may be flagged as fraudulent, leading to unhappy customers.

The answer, therefore, is to make decisions on a case-by-case basis, weighing the relative advantages and weaknesses of each. In future installments in this series Gilbert will discuss how one choice may eventually subsume the other.

Photo via sethink