Home PC News Google claims TensorFlow’s OpenCL can double inference performance

Google claims TensorFlow’s OpenCL can double inference performance

Google at the moment launched an OpenCL-based cell GPU inference engine for its TensorFlow framework on Android. It’s accessible now within the newest model of the TensorFlow Lite library, and the corporate claims it affords a two occasions speedup over the prevailing OpenGL backend with “reasonably-sized” AI fashions.

OpenGL, which is sort of three many years previous, is a platform-agnostic API for rendering 2D and 3D vector graphics. Compute shaders have been added with OpenGL ES 3.1, however the TensorFlow crew says backward-compatible design selections restricted them from reaching machine GPUs’ full potential. On the opposite hand, OpenCL was designed for computation with varied accelerators from the start and was thus extra related to the area of cell GPU inference. This motivated the TensorFlow crew’s investigation into — and eventual adoption of — an OpenCL-based cell inference engine.

The new TensorFlow inference engine options an optimizer that chooses the appropriate workgroup dimension to spice up efficiency, leading to as much as a 50% speedup over the common on {hardware} like Qualcomm Adreno GPUs. It helps FP16 natively and requires accelerators to specify information varieties’ availability, lowering reminiscence and bandwidth utilization and coaching time by dashing up algorithmic computations. (Google notes that some older GPUs just like the circa-2012 Adreno 305 can now function at their full capabilities because of FP16 help.) And OpenCL is ready to drastically outperform OpenGL’s efficiency by sustaining synergy with bodily fixed reminiscence, a {hardware} function in chips like Adreno GPUs that reserves RAM for storing fixed arrays and variables.

Inference latency of MNASNet 1.3 on select Android devices with OpenCL

Above: Inference latency of MNASNet 1.Three on choose Android units with OpenCL.

Image Credit: Google

In one benchmark take a look at, the TensorFlow crew lowered the latency of MNASNet 1.3, a so-called neural structure search system, from over 100 milliseconds on the Vivo Z3 with the OpenGL-based backend to 25 milliseconds with the OpenCL different. In one other take a look at with the item detection algorithm SSD MobileNet v3, the crew lowered latency from practically 100 milliseconds on the Huawei Mate 20 to lower than 25 milliseconds.

Inference latency of SSD MobileNet v3 (large) on select Android devices with OpenCL

Above: Inference latency of SSD MobileNet v3 (giant) on choose Android units with OpenCL.

Image Credit: Google

Google notes that OpenCL isn’t part of the usual Android distribution, making it unavailable to some customers. As a stopgap measure, TensorFlow Lite now checks for the provision of OpenCL at runtime in order that if it isn’t accessible or can’t be loaded, the library falls again to the previous OpenGL backend.

“While the TensorFlow Lite GPU team continuously improves the existing OpenGL-based mobile GPU inference engine, we also keep investigating other technologies,” TensorFlow software program engineers Juhyun Lee and Raman Sarokin wrote in a weblog put up. “OpenCL brings quite a lot of features that let us optimize our mobile GPU inference engine.”

Most Popular

Will RISC-V be a contender now that Nvidia is buying Arm?

The microprocessor industry’s unfolding saga got a big plot twist a couple of weeks ago when Nvidia paid $40 billion to buy Arm, the...

The tech sector can — and must — disrupt social inequity

As scores of headlines expose systemic racial injustice and COVID-19 thrusts organizations even deeper into digital transformation, it’s clear that we’ve arrived at the...

AMD Radeon RX 6000 GPUs revealed in macOS Big Sur code: up to 5120 cores, 2.5 GHz

Highly anticipated: One brave Redditor who trawled through the deep mines of...

Ekto’s robotic boots may solve VR locomotion problems

Ekto VR thinks it might have just solved VR locomotion. The Pittsburgh-based company has revealed its first product: the Ekto One. It’s...

Recent Comments