Twitter has just launched a new backend for search during the last few weeks, switching from old architecture to a new one while maintaining the consistency of their search results.

“One of our main goals, but also biggest challenges, was a smooth switch from the old architecture to the new one, without any downtime or inconsistencies in search results,” said Twitter.

The previous real-time search engine was based on Summize technology. They shifted to Lucene, all for Twitter’s love of Open Source, in their latest development of a “modern search architecture that is based on a highly efficient inverted index instead of a relational database.” Some of Lucene’s modification highlights include:

* significantly improved garbage collection performance
* lock-free data structures and algorithms
* posting lists, that are traversable in reverse order
* efficient early query termination

Twitter gets 1000 Tweets/sec, 12,000 queries/sec amounting to a billion queries per day. This demands for better scalability and low indexing latencies. Twitter aims at making it possible for a tweet to be search in less than 10 seconds prior to tweeting. Twitter is only using about 5% of their backend at the moment because of this latest tweak. Its database has the potential to offer a wealth of information to end users, not just third-party developers, organizing and presenting that data in a cohesive format will really take twitter to the next level of usability and monetization.

Of course, it will also greatly benefit the users. The new indexer could index 50 times more tweets per second, no inconsistencies or instabilities in the search result. Also, the indexing will be twice as long but still maintains search speed. Also, this will make room for other cool new features.

Facebook is also doing some big tweaks to its search, layering in news search results. It’s another social network that’s still sitting on a ton of potential, which can be extracted through a comprehensive search mechanism. Google recognizes the power of socially-oriented search tools.  It started with search, and now has to become more social.