Flash storage and new database platforms challenge CIOs and CTOs to redesign business services around real-time Big Data analytic and transactional systems to support new business models that are a key to business growth, writes Wikibon Co-Founder and CTO David Floyer in his Wikibon Alert “Real-time Big Data: eXelate Case Study.”

eXelate is a pioneer in this, having built a Big Data analysis system to support real-time bidding for ad placements on the Internet on top of the Aerospike noSQL database engine running on clusters of 12 industry standard x86 processor nodes each, with 5 (growing to 7) SSDs and 128 GB DRAM per node.

Advertising on the Internet is highly customized to the immediate concerns of individual consumers. Ideally, the ad should speak to the consumer’s activities online at that moment. An ad for a new car is much more effective if it is presented when a consumer is focused on cars — perhaps searching for a mechanic — than it would be when sent randomly to large numbers of individuals who are doing other things. That latter model, which is of course the traditional model for advertising, is called “spam”. In the world of Internet advertising, “seize the moment” is the rule, and a few minutes makes the difference between a click through and sale and a lost opportunity.

However, this model requires a radical new approach to IT. eXelate processes 60 billion transactions per month for more than 200 publishers and markets across multiple geographic regions. It must process all that click-stream data in real time while at the same time controlling costs.

To do this, it has built an architecture using 200 front-end servers at locations worldwide, high-availability Arospike clusters with Cross Data Center Replication for redundancy, proprietary prediction models built on Revolution R Enterprise software and IBM PureData System for Analytics running on IBM Netezza “shared nothing” parallel database appliances. The Aerospike noSQL database engine is vital to this architecture, optimized to ingest and manage these very large volumes of data with maximum efficiency while controlling costs.

Traditional SQL database systems would cost an order-of-magnitude more in equipment and software to support these volumes of data, a cost that could negate eXelate’s business model. “The most important technology to ensure that these systems can scale effectively is NAND flash memory, both to enable very rapid IO and extend main memory,” Floyer writes.

He predicts that these technologies will support a new class of real-time business decision systems based on analysis of Big Data that will find applications throughout business. Retailers wanting real-time price adjustments and variable marketing/pricing for “a customer segment of one” will, he predicts, be the next major wave of adopters of similar technology archictures.

As with all Wikibon research, this report is available online without charge on the Wikibon Web site. IT professionals are invited to register for free membership in the Wikibon community, which allows them to comment on the research and post their own relevant questions, tips, Professional Alerts, and white papers on the site.