{"id":1872,"date":"2022-03-01T18:40:21","date_gmt":"2022-03-01T18:40:21","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/01\/train-175-billion-parameter-nlp-models-with-model-parallel-additions-and-hugging-face-on-amazon-sagemaker\/"},"modified":"2022-03-01T18:40:21","modified_gmt":"2022-03-01T18:40:21","slug":"train-175-billion-parameter-nlp-models-with-model-parallel-additions-and-hugging-face-on-amazon-sagemaker","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/01\/train-175-billion-parameter-nlp-models-with-model-parallel-additions-and-hugging-face-on-amazon-sagemaker\/","title":{"rendered":"Train 175+ billion parameter NLP models with model parallel additions and Hugging Face on Amazon SageMaker"},"content":{"rendered":"
\n

The last few years have seen rapid development in the field of natural language processing (NLP). While hardware has improved, such as with the latest generation of accelerators from NVIDIA and Amazon, advanced machine learning (ML) practitioners still regularly run into issues scaling their large language models across multiple GPU\u2019s.<\/p>\n

In this blog post, we briefly summarize the rise of large- and small- scale NLP models, primarily through the abstraction provided by Hugging Face and with the modular backend of Amazon SageMaker. In particular we highlight the launch of four additional features within the SageMaker model parallel library that unlock\u00a0175 billion parameter NLP model pretraining and fine-tuning for customers.<\/p>\n

We used this library on the SageMaker training platform and achieved a throughput of 32\u00a0samples per second on 120 ml.p4d.24xlarge instances and 175 billion parameters. We anticipate that if we increased this up to 240 instances, the full model would take 25 days to train.<\/p>\n

For more information about model parallelism, see the paper Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training<\/a>.<\/p>\n

You can also see the GPT2 notebook we used to generate these performance numbers on our GitHub repository<\/a>.<\/p>\n

To learn more about how to use the new features within SageMaker model parallel, refer to Extended Features of the SageMaker Model Parallel Library for PyTorch<\/a>, and Use with the SageMaker Python SDK<\/a>.<\/p>\n

NLP on Amazon SageMaker \u2013 Hugging Face and model parallelism<\/strong><\/h2>\n

If you\u2019re new to Hugging Face and NLP, the biggest highlight you need to know is that applications using natural language processing (NLP) are starting to achieve human level performance. This is largely driven by a learning mechanism, called attention<\/a>,\u00a0 which gave rise to a deep learning model, called the transformer<\/em>, that is much more scalable than previous deep learning sequential methods. The now-famous BERT model<\/a> was developed to capitalize on the transformer, and developed several useful NLP tactics along the way. Transformers and the suite of models, both within and outside of NLP, which have all been inspired by BERT, are the primary engine behind your Google search results<\/a>, in your Google translate results<\/a>, and a host of new startups<\/a>.<\/p>\n

SageMaker and Hugging Face partnered to make this easier for customers than ever before. We\u2019ve launched Hugging Face deep learning containers (DLC\u2019s) for you to train and host pre-trained models directly from Hugging Face\u2019s repository of over 26,000 models.<\/a>\u00a0We\u2019ve launched\u00a0the SageMaker Training Compiler\u00a0<\/a>for you to speed up the runtime of your Hugging Face training loops by up to 50%. We\u2019ve also integrated the Hugging Face flagship Transformers SDK <\/a>with our distributed training libraries<\/a>\u00a0to make scaling out your NLP models easier than ever before.<\/p>\n

For more information about Hugging Face Transformer models on Amazon SageMaker, see Support for Hugging Face Transformer Models.<\/a><\/p>\n

New features for large-scale NLP model training with the SageMaker model parallel library\u00a0<\/strong><\/h2>\n

At AWS re:Invent 2020, SageMaker launched distributed libraries that provide the best performance on the cloud for training computer vision models like Mask-RCNN<\/a>\u00a0and NLP models like T5-3B.<\/a>\u00a0This is possible through enhanced communication primitives that are 20-40% faster than NCCL on AWS, and model distribution techniques that enable extremely large language models to scale across tens to hundreds to thousands of GPUs.<\/p>\n

The SageMaker model parallel library (SMP) has always given you the ability to take your predefined NLP model in PyTorch, be that through Hugging Face or elsewhere, and partition that model onto multiple GPUs in your cluster. Said another way, SMP breaks up your model into smaller chunks so you don\u2019t experience out of memory (OOM) errors. We\u2019re pleased to add additional memory-saving techniques that are critical for large scale models, namely:<\/p>\n