Page 41 - EETEurope FlipBook February

EE|Times EUROPE 41

OPINION | MARKET & TECHNOLOGY TRENDS

Are Today’s Huge AI

Language Models Out

of Control?

By Sally Ward-Foxton

Natural-language processing (NLP) is a popular and
rapidly evolving field of AI. But the evolution of more sophisticated
language models comes with an overarching mega-trend: These
models are growing at a staggering rate.
Google’s latest language model has 1.6 trillion parameters — a
figure even its inventors called “outrageous” in their paper. The
model’s parameters are the figures that the training process
adjusts, and in general, more parameters mean a more sophisti-
cated AI. Models like this one that purport to understand language
are usually given a text prompt and then create more text in the same vein. The AI does not
really understand language; it merely mimics it to a convincing degree.
Google’s model topped the previous world-record holder, OpenAI’s GPT-3, by a factor of 9.
GPT-3’s 175 billion parameters were widely heralded as enormous, and examples of the texts
it created were marvelled over by many news outlets. In many cases, GPT-3’s text output was
indistinguishable from articles written by humans; the model can mimic language concepts
such as analogies and even write basic software code. GPT-3 itself was multiples bigger than
its predecessor, GPT-2, which had 1.5 billion parameters.
Language models like these have grown to the
The biggest language size at which the financial and environmental costs
associated with computing them invite scrutiny. A
model yet, from 2019 study by researchers from the University of Mas-
Google, has 1.6 trillion sachusetts Amherst found that training GPT-2 took
state-of-the-art specialized hardware (32× TPUv3
parameters — a figure chips) 168 hours with an estimated cloud compute
even its inventors called cost of between US$12,000 and US$43,000. That’s
training one model, one time. Models are typically
“outrageous.” trained and tweaked and retrained many times over
during the development process.
Now consider that Google’s latest model is 1,000×
bigger than GPT-2. If we assume the compute required to train scales roughly with the model’s
size, we start to get an idea of the scale of resources required to develop and use these models.
(Hint: It’s many millions of dollars). And the carbon footprint associated with compute on this
scale is significant, too.
Are today’s language models too large, and aside from financial and environmental cost, able to readily produce convincing misin-
what are the implications? formation at scale and the havoc that rogue
In practice, these models are now so large that Google is one of only a few companies that chatbots could wreak given the ability to
can do this kind of research. The company has access to huge amounts of computing power pass for a human.
in-house which, while accessible via the cloud, are not economically viable at such scale for Only a few weeks before Google published
smaller companies or academic researchers. This allows tech giants like Google to effectively its latest work on the trillion-parameter
dictate the direction of research in this field, and naturally, research will follow whatever path model, a prominent ethics researcher with
is in their commercial interest. Whether their commercial interest aligns with ethical best the company said she had been forced out
practices or will lead to language models that can actually understand language is unclear. after authoring a paper on the ethics of large
Where does one get the vast amount of training data required for a model this large? The language models. In light of the criticism
short answer is the internet. Text is scraped from sources such as Wikipedia and Reddit. There this move drew, Google will find it difficult
is a significant risk that trained models’ output will reflect the nature of the training data, to keep answering questions about the ethics
i.e., that it will mimic things people write in the internet’s darkest corners. If the model sees of research into huge — and growing — AI
humanity at its worst, we should expect its output to reflect that. language models. ■ IMAGE: SHUTTERSTOCK
There are also questions about the potential for misuse of such powerful models. If a
machine can write on any topic as well as a human can, it suddenly becomes very easy to fool Sally Ward-Foxton is editor-in-chief of
people — especially over the internet, where people communicate via text. Imagine being EE Times Weekend.

www.eetimes.eu | FEBRUARY 2021