While the work of the U.K’s Government Communications Headquarters (GCHQ) is a closely guarded secret known only to select personnel, the internally-developed software powering its intelligence gathering operations is now publicly available for anyone to download. The source code of the aptly-named Gaffer database was released to GitHub yesterday under a free license to make it easier for developers to store complicated hierarchical records that can’t easily fit into traditional schemas.

A relevant example would be phone records, which need to be organized in a format that enables fast access not only to caller identities but also related details such as the date and duration of a particular conversation if they are to be properly analyzed. Maintaining a sufficiently high standard of availability, however, becomes increasingly difficult over time as new information is added and existing entries are changed. Gaffer minimizes clutter by making it possible to neatly group all of the data associated with an individual in a logical construct called a “node” with multiple “edges” each containing a different item.

That way, a GCHQ analyst can quickly view the activities of a suspect or search for details that might link them to co-conspirators without having to waste time manually filtering the information first. Gaffer further improves response times with a unique updating mechanism that makes it possible to add and change edges without nearly as much overhead as other graph databases, a capability that should lend itself particularly well to large deployments with a constant stream of new information to process. It’s one of the two main features that set the project apart, with the other being the fact that all records are kept by default in Apache Accumulo.

The likewise open key-value store is arguably the most secure in its category thanks to a granular authorization mechanism that allows administrators to restrict access all the way down to individual data points. The functionality reflects the strict requirements that the original creators of Accumulo were tasked to address for their employer, the National Security Agency (NSA), which has been investing even more than its U.K counterpart in forging ties with the open-source community. The spy agency most recently saw a team of former staffers who spun off to commercialize the Nifi network orchestration framework, another internal project, join Hortonworks Inc. through an acquisition.

Image via Geralt