The branch of artificial intelligence called deep learning has led to breakthroughs in speech and image recognition, partly thanks to an almost accidental discovery that certain chips such as graphics processing units work really well for it.

But just how well has been hard to measure. So today, the Chinese Internet company Baidu Inc. is releasing into open source a new deep learning benchmarking tool that can help both AI researchers and processor manufacturers measure how various kinds of chips work on the software, which is roughly modeled on the brain’s neural networks.

DeepBench was born of the reality that the big bottleneck in running deep learning neural networks remains computing power, said Sharan Narang (pictured), a software engineer in Baidu’s Silicon Valley AI Lab in Sunnyvale, California. Although the parallel processing power of GPUs has helped make deep learning models more useful, he said in an interview, “we still need computers with more processing power.”

The problem is figuring out which computers deliver, because the theoretical performance metrics provided by Nvidia Corp., Intel Corp. and others don’t always match real-world results. Greg Diamos, a senior researcher at Baidu, said in an interview that the company hopes DeepBench will help encourage the creation of more processors tuned especially for particular deep learning algorithms for speech and image recognition, language translation and other tasks.

“We’re hoping this will lower the barrier to entry for processors with higher performance,” Diamos said. “GPUs definitely aren’t the end of the story. We think this will encourage competition.”

DeepBench currently will provide benchmark results on three Nvidia GPUs and one Intel Xeon Phi processor, but others are planned. Other companies such as Google Inc. and Wave Computing Inc. are creating processors specifically for deep learning, and Intel this year has bought two companies, Movidius and Nervana Systems, that were making or planning to make deep learning chips. So there’s a rising need for benchmarks that can compare them.

For now, DeepBench measures only the operations for training deep learning models, but it may be extended to measuring the “inference” operations, the actual running of the models for image and speech recognition and the like.

It’s available now on GitHub. The Baidu researchers say they welcome input from deep learning researchers and hardware companies.

Photo from Baidu