{"id":2020,"date":"2022-03-25T15:39:04","date_gmt":"2022-03-25T15:39:04","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/25\/what-is-a-transformer-model\/"},"modified":"2022-03-25T15:39:04","modified_gmt":"2022-03-25T15:39:04","slug":"what-is-a-transformer-model","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/25\/what-is-a-transformer-model\/","title":{"rendered":"What Is a Transformer Model?"},"content":{"rendered":"
\n

If you want to ride the next big wave in AI, grab a transformer.<\/p>\n

They\u2019re not the shape-shifting toy robots on TV or the trash-can-sized tubs on telephone poles.<\/p>\n

So, What\u2019s a Transformer Model?<\/b><\/h2>\n

A transformer model is a neural network that learns context and thus meaning by tracking relationships in sequential data like the words in this sentence.<\/p>\n

Transformer models apply an evolving set of mathematical techniques, called attention or self-attention, to detect subtle ways even distant data elements in a series influence and depend on each other.<\/p>\n

First described in a 2017 paper<\/a> from Google, transformers are among the newest and one of the most powerful classes of models invented to date. They\u2019re driving a wave of advances in machine learning some have dubbed transformer AI.<\/p>\n

Stanford researchers called transformers \u201cfoundation models\u201d in an August 2021 paper<\/a> because they see them driving a paradigm shift in AI. The \u201csheer scale and scope of foundation models over the last few years have stretched our imagination of what is possible,\u201d they wrote.<\/p>\n

What Can Transformer Models Do?<\/b><\/h2>\n

Transformers are translating text and speech in near real-time, opening meetings and classrooms to diverse and hearing-impaired attendees.<\/p>\n

They\u2019re helping researchers understand the chains of genes in DNA and amino acids in proteins in ways that can speed drug design.<\/p>\n

<\/p>\n

\"Transformers<\/p>\n

<\/a>

Transformers, sometimes called foundation models, are already being used with many data sources for a host of applications.<\/figcaption><\/figure>\n

Transformers can detect trends and anomalies to prevent fraud, streamline manufacturing, make online recommendations or improve healthcare.<\/p>\n

People use transformers every time they search on Google or Microsoft Bing.<\/p>\n

The Virtuous Cycle of Transformer AI<\/b><\/h2>\n

Any application using sequential text, image or video data is a candidate for transformer models.<\/p>\n

That enables these models to ride a virtuous cycle in transformer AI. Created with large datasets, transformers make accurate predictions that drive their wider use, generating more data that can be used to create even better models.<\/p>\n

<\/p>\n

\"Transformer<\/p>\n

<\/a>

Stanford researchers say transformers mark the next stage of AI\u2019s development, what some call the era of transformer AI.<\/figcaption><\/figure>\n

\u201cTransformers made self-supervised learning possible, and AI jumped to warp speed,\u201d said NVIDIA founder and CEO Jensen Huang in his keynote address this week<\/a> at GTC.<\/p>\n

Transformers Replace CNNs, RNNs<\/b><\/h2>\n

Transformers are in many cases replacing convolutional and recurrent neural networks (CNNs and RNNs), the most popular types of deep learning models just five years ago.<\/p>\n

Indeed, 70 percent of arXiv<\/a> papers on AI posted in the last two years mention transformers. That\u2019s a radical shift from a 2017 IEEE study<\/a> that reported RNNs and CNNs were the most popular models for pattern recognition.<\/p>\n

No Labels, More Performance<\/b><\/h2>\n

Before transformers arrived, users had to train neural networks with large, labeled datasets that were costly and time-consuming to produce. By finding patterns between elements mathematically, transformers eliminate that need, making available the trillions of images and petabytes of text data on the web and in corporate databases.<\/p>\n

In addition, the math that transformers use lends itself to parallel processing, so these models can run fast.<\/p>\n

Transformers now dominate popular performance leaderboards like SuperGLUE<\/a>, a benchmark developed in 2019<\/a> for language-processing systems.<\/p>\n

How Transformers Pay Attention<\/b><\/h2>\n

Like most neural networks, transformer models are basically large encoder\/decoder blocks that process data.<\/p>\n

Small but strategic additions to these blocks (shown in the diagram below) make transformers uniquely powerful.<\/p>\n

<\/p>\n

\"Example<\/p>\n

<\/a>

A look under the hood from a presentation by Aidan Gomez, one of eight co-authors of the 2017 paper that defined transformers.<\/figcaption><\/figure>\n

Transformers use positional encoders to tag data elements coming in and out of the network. Attention units follow these tags, calculating a kind of algebraic map of how each element relates to the others.<\/p>\n

Attention queries are typically executed in parallel by calculating a matrix of equations in what\u2019s called multi-headed attention.<\/p>\n

With these tools, computers can see the same patterns humans see.<\/p>\n

Self-Attention Finds Meaning<\/b><\/h2>\n

For example, in the sentence:<\/p>\n

She poured water from the pitcher to the cup until it was full.\u00a0<\/i><\/p>\n

We know \u201cit\u201d refers to the cup, while in the sentence:<\/p>\n

She poured water from the pitcher to the cup until it was empty.<\/i><\/p>\n

We know \u201cit\u201d refers to the pitcher.<\/p>\n

\u201cMeaning is a result of relationships between things, and self-attention is a general way of learning relationships,\u201d said Ashish Vaswani, a former senior staff research scientist at Google Brain who led work on the seminal 2017 paper.<\/p>\n

\u201cMachine translation was a good vehicle to validate self-attention because you needed short- and long-distance relationships among words,\u201d said Vaswani.<\/p>\n

\u201cNow we see self-attention is a powerful, flexible tool for learning,\u201d he added.<\/p>\n

How Transformers Got Their Name<\/b><\/h2>\n

Attention is so key to transformers the Google researchers almost used the term as the name for their 2017 model. Almost.<\/p>\n

\u201cAttention Net didn\u2019t sound very exciting,\u201d said Vaswani, who started working with neural nets in 2011.<\/p>\n

.Jakob Uszkoreit, a senior software engineer on the team, came up with the name Transformer.<\/p>\n

\u201cI argued we were transforming representations, but that was just playing semantics,\u201d Vaswani said.<\/p>\n

The Birth of Transformers<\/b><\/h2>\n

In the paper for the 2017 NeurIPS conference, the Google team described their transformer and the accuracy records it set for machine translation.<\/p>\n

Thanks to a basket of techniques, they trained their model in just 3.5 days on eight NVIDIA GPUs, a small fraction of the time and cost of training prior models. They trained it on datasets with up to a billion pairs of words.<\/p>\n

\u201cIt was an intense three-month sprint to the paper submission date,\u201d recalled Aidan Gomez, a Google intern in 2017 who contributed to the work.<\/p>\n

\u201cThe night we were submitting, Ashish and I pulled an all-nighter at Google,\u201d he said. \u201cI caught a couple hours sleep in one of the small conference rooms, and I woke up just in time for the submission when someone coming in early to work opened the door and hit my head.\u201d<\/p>\n

It was a wakeup call in more ways than one.<\/p>\n

\u201cAshish told me that night he was convinced this was going to be a huge deal, something game changing. I wasn\u2019t convinced, I thought it would be a modest gain on a benchmark, but it turned out he was very right,\u201d said Gomez, now CEO of startup Cohere<\/a> that\u2019s providing a language processing service based on transformers.<\/p>\n

A Moment for Machine Learning<\/b><\/h2>\n

Vaswani recalls the excitement of seeing the results surpass similar published by a Facebook team using CNNs.<\/p>\n

\u201cI could see this would likely be an important moment in machine learning,\u201d he said.<\/p>\n

A year later, another Google team tried processing text sequences both forward and backward with a transformer. That helped capture more relationships among words, improving the model\u2019s ability to understand the meaning of a sentence.<\/p>\n

Their Bidirectional Encoder Representations from Transformers (BERT<\/a>) model set 11 new records and became part of the algorithm behind Google search.<\/p>\n

Within weeks, researchers around the world were adapting BERT<\/a> for use cases across many languages and industries \u201cbecause text is one of the most common data types companies have,\u201d said Anders Arpteg, a 20-year veteran of machine learning research.<\/p>\n

Putting Transformers to Work<\/b><\/h2>\n

Soon transformer models were being adapted for science and healthcare.<\/p>\n

DeepMind, in London, advanced the understanding of proteins, the building blocks of life, using a transformer called AlphaFold2, described in a recent Nature article<\/a>. It processed amino acid chains like text strings to set a new watermark for describing how proteins fold, work that could speed drug discovery.<\/p>\n

AstraZeneca and NVIDIA developed MegaMolBART<\/a>, a transformer tailored for drug discovery. It\u2019s a version of pharmaceutical company\u2019s MolBART transformer, trained on a large, unlabeled database of chemical compounds using the NVIDIA Megatron<\/a> framework for building large-scale transformer models.<\/p>\n

Reading Molecules, Medical Records<\/b><\/h2>\n

\u201cJust as AI language models can learn the relationships between words in a sentence, our aim is that neural networks trained on molecular structure data will be able to learn the relationships between atoms in real-world molecules,\u201d said Ola Engkvist, head of molecular AI, discovery sciences and R&D at AstraZeneca, when the work was announced last year<\/a>.<\/p>\n<\/p>\n

Separately, the University of Florida<\/a>\u2019s academic health center collaborated with NVIDIA<\/a> researchers to create GatorTron<\/a>. The transformer model aims to extract insights from massive volumes of clinical data to accelerate medical research.<\/p>\n

Transformers Grow Up<\/b><\/h2>\n

Along the way, researchers found larger transformers performed better.<\/p>\n

For example, researchers from the Rostlab<\/a> at the Technical University of Munich, which helped pioneer work at the intersection of AI and biology, used natural-language processing to understand proteins<\/a>. In 18 months, they graduated from using RNNs with 90 million parameters to transformer models with 567 million parameters.<\/p>\n

<\/p>\n

\"Transformer<\/p>\n

<\/a>

Rostlab researchers show language models trained without labeled samples picking up the signal of a protein sequence.<\/figcaption><\/figure>\n

The OpenAI lab showed bigger is better with its Generative Pretrained Transformer (GPT). The latest version, GPT-3<\/a>,<\/b> has 175 billion parameters, up from 1.5 billion for GPT-2.<\/p>\n

With the extra heft, GPT-3 can respond to a user\u2019s query even on tasks it was not specifically trained to handle. It\u2019s already being used by companies including Cisco, IBM and Salesforce.<\/p>\n

Tale of a Mega Transformer<\/b><\/h2>\n

NVIDIA and Microsoft hit a high watermark in November, announcing the Megatron-Turing Natural Language Generation model<\/a> (MT-NLG<\/a>) with 530 billion parameters. It debuted along with a new framework, NVIDIA NeMo Megatron<\/a>, that aims to let any business create its own billion- or trillion-parameter transformers to power custom chatbots, personal assistants and other AI applications that understand language.<\/p>\n

MT-NLG had its public debut as the brain for TJ, the Toy Jensen avatar that gave part of the keynote at NVIDIA\u2019s November 2021 GTC.<\/p>\n

\u201cWhen we saw TJ answer questions \u2014 the power of our work demonstrated by our CEO \u2014 that was exciting,\u201d said Mostofa Patwary, who led the NVIDIA team that trained the model.<\/p>\n

<\/p>\n

\"The<\/p>\n

<\/a>

\u201cMegatron helps me answer all those tough questions Jensen throws at me,\u201d TJ said at GTC 2022.<\/figcaption><\/figure>\n

Creating such models is not for the faint of heart. MT-NLG was trained using hundreds of billions of data elements, a process that required thousands of GPUs running for weeks.<\/p>\n

\u201cTraining large transformer models is expensive and time-consuming, so if you\u2019re not successful the first or second time, projects might be canceled,\u201d said Patwary.<\/p>\n

Trillion-Parameter Transformers<\/b><\/h2>\n

Today, many AI engineers are working on trillion-parameter transformers and applications for them.<\/p>\n

\u201cWe\u2019re constantly exploring how these big models can deliver better applications. We also investigate in what aspects they fail, so we can build even better and bigger ones,\u201d Patwary said.<\/p>\n

To provide the computing muscle those models need, our latest accelerator \u2014 the NVIDIA H100 Tensor Core GPU<\/a> \u2014 packs a Transformer Engine<\/a> and supports a new FP8 format. That speeds training while preserving accuracy.<\/p>\n

With those and other advances, \u201ctransformer model training can be reduced from weeks to days\u201d said Huang at GTC.<\/p>\n

MoE Means More for Transformers<\/b><\/p>\n

Last year, Google researchers described the Switch Transformer<\/a>, one of the first trillion-parameter models. It uses AI sparsity, a complex mixture-of experts (MoE) architecture and other advances to drive performance gains in language processing and up to 7x increases in pre-training speed.<\/p>\n

<\/p>\n

\"Google's<\/p>\n

<\/a>

The encoder for the Switch Transformer, the first model to have up to a trillion parameters.<\/figcaption><\/figure>\n

For its part, Microsoft Azure worked with NVIDIA<\/a> to implement an MoE transformer for its Translator<\/a> service.<\/p>\n

Tackling Transformers\u2019 Challenges<\/b><\/h2>\n

Now some researchers aim to develop simpler transformers with fewer parameters that deliver performance similar to the largest models.<\/p>\n

\u201cI see promise in retrieval-based models that I\u2019m super excited about because they could bend the curve,\u201d said Gomez, of Cohere, noting the Retro model<\/a> from DeepMind as an example.<\/p>\n

Retrieval-based models learn by submitting queries to a database. \u201cIt\u2019s cool because you can be choosy about what you put in that knowledge base,\u201d he said.<\/p>\n

<\/p>\n

\"Transformer<\/p>\n

<\/a>

In the race for higher performance, transformer models have grown larger.<\/figcaption><\/figure>\n

The ultimate goal is to \u201cmake these models learn like humans do from context in the real world with very little data,\u201d said Vaswani, now co-founder of a stealth AI startup.<\/p>\n

He imagines future models that do more computation upfront so they need less data and sport better ways users can give them feedback.<\/p>\n

\u201cOur goal is to build models that will help people in their everyday lives,\u201d he said of his new venture.<\/p>\n

Safe, Responsible Models<\/b><\/h2>\n

Other researchers are studying ways to eliminate bias or toxicity if models amplify wrong or harmful language. For example, Stanford created the Center for Research on Foundation Models<\/a> to explore these issues.<\/p>\n

\u201cThese are important problems that need to be solved for safe deployment of models,\u201d said Shrimai Prabhumoye, a research scientist at NVIDIA who\u2019s among many across the industry working in the area.<\/p>\n

\u201cToday, most models look for certain words or phrases, but in real life these issues may come out subtly, so we have to consider the whole context,\u201d added Prabhumoye.<\/p>\n

\u201cThat\u2019s a primary concern for Cohere, too,\u201d said Gomez. \u201cNo one is going to use these models if they hurt people, so it\u2019s table stakes to make the safest and most responsible models.\u201d<\/p>\n

Beyond the Horizon<\/b><\/h2>\n

Vaswani imagines a future where self-learning, attention-powered transformers approach the holy grail of AI.<\/p>\n

\u201cWe have a chance of achieving some of the goals people talked about when they coined the term \u2018general artificial intelligence\u2019 and I find that north star very inspiring,\u201d he said.<\/p>\n

\u201cWe are in a time where simple methods like neural networks are giving us an explosion of new capabilities.\u201d<\/p>\n

<\/p>\n

\"NVIDIA<\/p>\n

<\/a>

Transformer training and inference will get significantly accelerated with the NVIDIA H100 GPU.<\/figcaption><\/figure>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

https:\/\/blogs.nvidia.com\/blog\/2022\/03\/25\/what-is-a-transformer-model\/<\/p>\n","protected":false},"author":0,"featured_media":2021,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/2020"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=2020"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/2020\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/2021"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=2020"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=2020"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=2020"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}