{"id":853,"date":"2021-09-17T06:56:38","date_gmt":"2021-09-17T06:56:38","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/09\/17\/pushing-forward-the-frontiers-of-natural-language-processing\/"},"modified":"2021-09-17T06:56:38","modified_gmt":"2021-09-17T06:56:38","slug":"pushing-forward-the-frontiers-of-natural-language-processing","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/09\/17\/pushing-forward-the-frontiers-of-natural-language-processing\/","title":{"rendered":"Pushing Forward the Frontiers of Natural Language Processing\u00a0"},"content":{"rendered":"
\n

Idea generation, not hardware or software, needs to be the bottleneck to the advancement of AI, Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, said this week at the AI Hardware Summit.<\/p>\n

\u201cWe want the inventors, the researchers and the engineers that are coming up with future AI to be limited only by their own thoughts,\u201d Catanzaro told the audience.<\/p>\n

Catanzaro leads a team of researchers working to apply the power of deep learning to everything from video games to chip design. At the annual event held in Silicon Valley, he described the work that NVIDIA is doing to enable advancements in AI, with a focus on large language modeling.<\/p>\n

CUDA Is for the Dreamers<\/b><\/h2>\n

Training and deploying large neural networks is a tough computational problem, so hardware that\u2019s both incredibly fast and highly efficient is a necessity, according to Catanzaro.<\/p>\n

But, he explained, the software that accompanies that hardware might be even more important to unlocking further advancements in AI.<\/p>\n

\u201cThe core of the work that we do involves optimizing hardware and software together, all the way from chips, to systems, to software, frameworks, libraries, compilers, algorithms and applications,\u201d he said. \u201cWe optimize all of these things to give transformational capabilities to scientists, researchers and engineers around the world.\u201d<\/p>\n

This end-to-end approach yields chart-topping performance in industry-standard benchmarks, such as MLPerf<\/a>. It also ensures that developers aren\u2019t constrained by the platform as they aim to advance AI.<\/p>\n

\u201cCUDA is for the dreamers, CUDA is for the people who are thinking new thoughts,\u201d said Catanzaro. \u201cHow do they think those thoughts and test them efficiently? They need something general and flexible, and that\u2019s why we build what we build.\u201d<\/p>\n

Large Language Models Are Changing the World<\/b><\/h2>\n

One of the most exciting areas of AI is language modeling, which is enabling groundbreaking applications in natural language understanding and conversational AI<\/a>.<\/p>\n

The complexity of large language models is growing at an incredible rate, with parameter counts doubling <\/i>every two months.<\/p>\n

A well-known example of a large and powerful language model is GPT-3, developed by OpenAI. Packing 175 billion parameters, it required 314 zettaflops (1021 floating point operations) to train.<\/p>\n

\u201cIt\u2019s a staggering amount of compute,\u201d Catanzaro said. \u201cAnd that means language modeling is now becoming constrained by economics.\u201d<\/p>\n

Estimates suggest that GPT-3 would cost about $12 million to train and, Catanzaro observed, the rapid growth in model complexity means that, despite NVIDIA\u2019s tireless work to advance the performance and efficiency of its hardware and software, the cost to train these models is set to grow.<\/p>\n

And, according to Catanzaro, this trend suggests that it might not be too long before a single model might require more than a billion dollars\u2019 worth of computer time to train.<\/p>\n

\u201cWhat would it look like to build a model that took a billion dollars to train a single model? Well, it would need to reinvent an entire company, and you\u2019d need to be able to use it in a lot of different contexts,\u201d Catanzaro explained.<\/p>\n

Catanzaro expects that these models will unlock an incredible amount of value, inspiring continued innovation. During his talk, Catanzaro showed an example of the surprising capabilities of large language models to solve new tasks without being explicitly trained to do so.<\/p>\n

After inputting just a few examples into a large language model \u2014 four sentences, with two written in English and their corresponding translations into Spanish \u2014 he then entered an English sentence, which the model then translated into Spanish properly.<\/p>\n

The model was able to do this despite never being trained to do translation. Instead, it was trained \u2014 using, as Catanzaro described, \u201can enormous amount of data from the internet\u201d \u2014 to predict the next word that should follow a given sequence of text.<\/p>\n

To perform that very generic task, the model needed to come up with higher-level representations of concepts, such as the existence of languages in general, English and Spanish vocabularies and grammar, and the concept of a translation task, in order to understand the query and properly respond.<\/p>\n

\u201cThese language models are first steps towards generalized artificial intelligence with few shot learning, and that is enormously valuable and very exciting,\u201d explained Catanzaro.<\/p>\n

A Full-Stack Approach to Language Modeling\u00a0<\/b><\/h2>\n

Catanzaro then went on to describe NVIDIA Megatron<\/a>, a framework created by NVIDIA using PyTorch \u201cfor efficiently training the world\u2019s largest, transformer-based language models.\u201d<\/p>\n

A key feature of NVIDIA Megatron, which Catanzaro notes has already been used by various companies and organizations to train large transformer-based models, is model parallelism.<\/p>\n

Megatron supports both inter-layer (pipeline) parallelism, which allows different layers of a model to be processed on different devices, as well as intra-layer (tensor) parallelism, which allows a single layer to be processed by multiple different devices.<\/p>\n

Catanzaro further described some of the optimizations that NVIDIA applies to maximize the efficiency of pipeline parallelism and minimize so-called \u201cpipeline bubbles,\u201d during which a GPU is not performing useful work.<\/p>\n

A batch is split into microbatches, the execution of which is pipelined. This boosts the utilization of the GPU resources in a system during training. With further optimizations, pipeline bubbles can be reduced even more.<\/p>\n

Catanzaro described an optimization, recently published<\/a>, that entails \u201cround-robining each (pipeline) stage among multiple GPUs so that we can further reduce the amount of pipeline bubble overhead in this schedule.\u201d<\/p>\n

Although this optimization puts additional stress on the communication fabric within the system, Catanzaro showed that, by leveraging the full suite of NVIDIA\u2019s high-bandwidth, low-latency interconnect technologies, this optimization is able to deliver sizable speedups when training GPT-3 style models.<\/p>\n

<\/p>\n

\"\"<\/p>\n

<\/a><\/p>\n

Catanzaro then highlighted the impressive performance scaling of Megatron on NVIDIA DGX SuperPOD<\/a>, achieving 502 petaflops sustained across 3,072 GPUs, representing an astonishing 52 percent of Tensor Core peak at scale.<\/p>\n

\u201cThis represents an achievement by all of NVIDIA and our partners in the industry: to be able to deliver that level of end-to-end performance requires optimizing the entire computing stack, from algorithms to interconnects, from frameworks to processors,\u201d said Catanzaro.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

http:\/\/feedproxy.google.com\/~r\/nvidiablog\/~3\/1aTPPXV3J4s\/<\/p>\n","protected":false},"author":0,"featured_media":854,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/853"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=853"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/853\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/854"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=853"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=853"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=853"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}