Page 23 - EE Times Europe Magazine – November 2023

EE|Times EUROPE 23

The Mind-Boggling Cost of Generative AI Ownership

The latest algorithms pose
Attributes Google Search GPT-3 GPT-4
a challenge to the current
Cost Per Query ¢0.2 ~¢3 ~¢10
state-of-the-art processing
Table 2: Comparing the cost per query of GPT-3 and GPT-4 against Google search shows hardware, and GenAI
the leap in cost associated with GPT-4. (Source: Vsora)
accelerators are not
keeping up.
• Theoretical throughput of one (25 W / 0.007 queries per second)
leading-edge GenAI system processing • Total energy cost: US$0.11 per kWh
ChatGPT-4: ~0.055 queries/second • Energy cost per query: US$1.2 –4 processing hardware, and GenAI accelerators
• Number of systems needed to meet a • Total power consumption for are not keeping up. In fact, no hardware on
processing capability of 100,000 queries/second: ~363.7 MW the market today is capable of running the
100,000 queries/second: The energy cost amounts to about full GPT-4.
~1,800,000 (100,000 / 0.055) US$1 million per day (power consumption for Current LLM development efforts that focus
• Total acquisition cost: the chips × 24 hours × 0.11). on creating smaller but more specialized LLMs
~US$900,000,000,000 Clearly, the cost is dominated by hardware that can run on existing hardware are a diver-
(1,800,000 × 500,000), approaching acquisition. sion. The GenAI industry needs semiconductor
US$1 trillion innovations in computing methods and archi-
The daily depreciation amounts to about The best-guess total daily cost is in the ball- tectures capable of delivering performance of
~US$820 million (900,000,000,000 / 1,095). park of US$820 million. multiple petaFLOPS with efficiency greater than
The above leads to a GPT-4 cost per query for 50%, reducing latency to less than 2 seconds per
Estimated energy costs to execute the a system running 100,000 queries/second of query, constraining energy consumption and
hardware $US0.095 (820,000,000 / (100,000 × 24 × 60 × 60) shrinking cost to $US0.002 per query.
Assumptions: [(cost per day) / (# of queries × # of hours × # of Once this is in place—and it is only a
• Average power consumption per chip: seconds)]. Table 2 compares costs per query. matter of time—the promise of transformers’
25 W, based on nominal power, efficiency deployment on edge devices will be fully
and memory bandwidth FULFILLING THE PROMISE OF GenAI AT exploited. ■
• Throughput per chip: THE EDGE
~0.007 queries/second (0.055 / 8) The latest algorithms, such as GPT-4, pose Lauro Rizzatti is a business adviser for
• Energy consumption per query: 3,637 J a challenge to the current state-of-the-art Vsora.

GaN & SiC: Design,

Devices and the

Evolving Market

Explore the future of
power electronics with our

comprehensive report on
wide-bandgap semiconductors:
SiC and GaN.

DOWNLOAD THE EBOOK:

https://www.powerelectronicsnews.com/
special-report-gan-sic-worldwide/