Page 52 - EE Times Europe November 2021 final

52 EE|Times EUROPE

NEUROMORPHIC COMPUTING
Benchmarking Neuromorphic Computing:

Devil Is in the Details

By Sunny Bains

nyone building a new technology understands that success that task in mind. You also have to consider whether all steps in the
partly depends on showing value added — demonstrating process — from loading up the task to running it and getting an output
that your technology is better than your competitors’. Only — have been fully optimized. If not — and you wouldn’t necessarily
A in this way can innovators attract investors and satisfy man- expect them to be in an emerging technology — you will have to break
agers. When you are making a smaller, faster, lighter, more efficient down your metrics so they measure only the relevant systems, not the
replacement for something that already exists, this is easy — at least in infrastructure (temporarily) needed to support them.
principle — but it’s much more difficult when you are trying to create Of course, to do this work at all, you have to acquire the systems you
something genuinely new and different. want to compare — not always easy when only a small number have
Neuromorphic computing is among the fields in which engineers are been produced. You also have to hope that the hardware has been built
attempting something genuinely new, and the lack of easy comparisons in such a way that you can extract the information you need to com-
between different systems — neuromorphic and otherwise — can be a plete your study, such as the speed or power consumption at different
problem. stages. Unfortunately, this will not always be the case.
Part of the issue has to do with the complexity of the field. Neuro- Much as it would make things easier, to date, there is no equivalent
morphic technology is brain-inspired, but as discussed in an earlier of the floating-point operations per second (flops) metric. Engineers
article, there are many ways to implement that inspiration at the have attempted to use multiply-and-accumulate operations, but
1
hardware level: analog or digital, spikes or not, continuous or discrete MACs, while somewhat applicable to deep learning, do not reflect the
time, virtual or direct connections between neurons. complexity of neuromorphic engineering. Nor do synaptic operations.
There are also competing goals and emphases within groups. Some Why? Because there are too many ways to get the job done, too many
wish to simulate biology, some focus on energy efficiency, others want learning rules that can be used, too many encoding methods, too
to simulate human-like intelligence, and still others simply seek practi- many synapse and neuron functions.
cal solutions to everyday machine-learning problems. Even if that weren’t the case, it’s well known that, all things being
How to benchmark systems that use different interfaces, encodings, equal, a compiler must be tailored to implement the system software to
technology, and approaches has emerged as a hot topic in neuromor- be run. If not, then even the best system will perform poorly.
phic engineering over the past few years. Going back to 2016, there
have been many attempts to compare systems for different applica- STEPPING BACK
tions, or to run different algorithms, networks, or sets of algorithms/ To get some clarity, it helps to look beyond neuromorphic technology.
networks. Three studies on the subject were published just this year. Robin Blume-Kohout of Sandia National Laboratories is interested
It’s worth considering the larger picture to understand why bench- in benchmarking quantum computers. In a 2020 talk titled “Not All
2
IMAGE: SHUTTERSTOCK IDENTIFYING THE OBSTACLES benchmarks for any technology still at a very early development stage.
Benchmarks Are Created Equal,” he discusses the difficulty of relying on
marking remains a hard problem.
In a rehearsal of that argument that appears in a 2018 technical
report, Blume-Kohout states: “Today’s most advanced quantum
Consider trying to run a learning or recognition task, then comparing
3
processors are like infants. Metrics and benchmarks that are useful for
how it performs on competing experimental systems. First, you have
to choose a task that is at least achievable on all the systems you’re
adult humans (e.g., IQ or SAT scores) are blatantly inapplicable to an
comparing — despite the fact they may not have been designed with
NOVEMBER 2021 | www.eetimes.eu infant [whose] whole purpose is to grow into an adult. Monitoring its