{"id":304,"date":"2020-09-29T13:04:25","date_gmt":"2020-09-29T13:04:25","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/29\/bert-inference-on-g4-instances-using-apache-mxnet-and-gluonnlp-1-million-requests-for-20-cents\/"},"modified":"2020-09-29T13:04:25","modified_gmt":"2020-09-29T13:04:25","slug":"bert-inference-on-g4-instances-using-apache-mxnet-and-gluonnlp-1-million-requests-for-20-cents","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/29\/bert-inference-on-g4-instances-using-apache-mxnet-and-gluonnlp-1-million-requests-for-20-cents\/","title":{"rendered":"BERT inference on G4 instances using Apache MXNet and GluonNLP: 1 million requests for 20 cents"},"content":{"rendered":"
\n

Bidirectional Encoder Representations from Transformers (BERT) [1] has become one of the most popular models for natural language processing (NLP) applications. BERT can outperform other models in several NLP tasks, including question answering and sentence classification.<\/p>\n

Training the BERT model on large datasets is expensive and time consuming, and achieving low latency when performing inference on this model is challenging. Latency and throughput are key factors to deploy a model in production. In this post, we focus on optimizing these factors for BERT inference tasks. We also compare the cost of deploying BERT on different Amazon Elastic Compute Cloud<\/a> (Amazon EC2) instances.<\/p>\n

When running inference on the BERT-base model, the g4dn.xlarge GPU instance achieves between 2.6\u20135 times lower latency (3.8 on average) than a c5.24xlarge CPU instance. The g4dn.xlarge instance also achieves the best cost-effective ratio (cost per requests) compared to c5.xlarge, c5.24xlarge, and m5.xlarge CPU instances. Specifically, the cost of processing 1 million BERT-inference requests with sequence length 128 is $0.20 on g4dn.xlarge, whereas on c5.xlarge (the best of these CPU instances), the cost is $3.31\u2014the GPU instance is 16.5 times more efficient.<\/p>\n

We achieved these results after a set of GPU optimizations on MXNet, described in the section Optimizing BERT model performance on MXNET<\/strong> 1.6 and 1.7<\/strong> of this post.<\/p>\n

Amazon EC2 G4 instances<\/h2>\n

G4 instances are optimized for machine learning application deployments. They\u2019re equipped with NVIDIA T4 GPUs<\/a>, powered by Tensor Cores<\/a>, and deliver groundbreaking AI performance: up to 65 TFLOPS in FP16 precision and up to 130 TOPS in INT8 precision.<\/p>\n

Amazon EC2 offers a variety of G4 instances with one or multiple GPUs, and with different amounts of vCPU and memory. You can perform BERT inference below 5 milliseconds on a single T4 GPU with 16 GB, such as on a g4dn.xlarge instance (the cost of this instance at the time of writing is $0.526 per hour on demand in the US East (N. Virginia) Region.<\/p>\n

For more information about G4 instances, see Amazon EC2 G4 Instances<\/a>.<\/p>\n

GluonNLP and MXNet<\/h2>\n

GluonNLP is a deep learning framework built on top of MXNet<\/a>, which was specifically designed for NLP applications. It extends MXNet, providing NLP models, datasets, and examples.<\/p>\n

GluonNLP includes an efficient implementation of the BERT model, scripts for training and performing inference, and several datasets (such as GLUE benchmark and SQuAD). For more information, see GluonNLP: NLP made easy<\/a>.<\/p>\n

For this post, we use the GluonNLP BERT implementation to perform inference on NLP tasks. Specifically, we use MXNet version 1.7 and GluonNLP version 0.10.0.<\/p>\n

BERT-base inference results<\/h2>\n

We present results performing two different BERT tasks: question answering and classification (sentimental analysis using the Stanford Sentiment Treebank (SST2) dataset). We achieved the results after a set of GPU optimizations<\/a> on MXNet.<\/p>\n

In the following graphs, we compare latency achieved by a single GPU on a g4dn.xlarge instance with FP16 precision vs. the most efficient CPU instance in terms of latency, c5.24xlarge with INT8 precision, MKL BLAS and 24 OpenMP threads.<\/p>\n

The following graph shows BERT-base latency on c5.25xlarge (INT8) and g4dn.xlarge (FP16) instances performing a classification inference task (SST2 dataset). Different sequence length values (80, 128, 384), and different batch sizes (1, 4, 16, 8 ,32, 64, 128, 300) are shown. In the case of sequence length, 128 values are included as labels.<\/p>\n

\"\"<\/p>\n

The following graph shows BERT-base latency on c5.24xlarge (INT8) and g4dn.xlarge (FP16) instances performing a question answering inference task (SQuAD dataset). Different sequence length values (80, 128, 384), and different batch sizes (1, 4, 16, 8 ,32, 64, 128, 300) are shown. In the case of sequence length, 128 values are included as labels.<\/p>\n

\"\"\u00a0<\/strong><\/p>\n

In the following two graphs, we present a cost comparison between several instances based on the throughput (sentences\/s) and the cost of each instance on demand (cost per hour) in the US East (N. Virginia) Region.<\/p>\n

The following graph shows dollars per 1 million sequence classification requests, for different instances, batch size 128, and several sequence lengths (80, 128 and 384). The price on demand of each instance per hour was based on the US East (N. Virginia) Region: $0.192 for m5.xlarge, $0.17 for c5.xlarge, $4.08 for c5.24xlarge, and $0.526 g4dn.xlarge.<\/p>\n

\"\"<\/p>\n

The following graph shows dollars per 1 million question answering requests, for different instances, batch size 128, and several sequence lengths (80, 128 and 384). The price on demand of each instance per hour was based on the US East (N. Virginia) Region: $0.192 for m5.xlarge, $0.17 for c5.xlarge, $4.08 for c5.24xlarge, and $0.526 g4dn.xlarge.<\/p>\n

\"\"<\/u><\/strong><\/p>\n

Deploying BERT on G4 instances<\/h2>\n

You can easily reproduce the results in the preceding section on a g4dn.xlarge instance. You can start from a pretrained model and fine-tune it for a specific task before running inference, or you can download one of the following fine-tuned models:<\/p>\n

Then complete the following steps:<\/p>\n

    \n
  1. To initialize a G4 instance, on the Amazon EC2 console, choose Deep Learning AMI<\/strong> (Ubuntu 18.04) Version 28.1 (or posterior) and a G4 instance.<\/li>\n
  2. Connect to the instance and set MXNet 1.7 and GluonNLP 0.10.x:<\/li>\n<\/ol>\n
    \n
    pip install mxnet-cu102==1.7.0\r\ngit clone --branch v0.10.x https:\/\/github.com\/dmlc\/gluon-nlp.git\r\ncd gluon-nlp; pip install -e .; cd scripts\/bert\r\npython setup.py install\r\n<\/code><\/pre>\n<\/div>\n
    \u00a0<\/strong>The command python setup.py install<\/code> generates a custom graph pass (bertpass_lib.so<\/code>) that optimizes the graph, and therefore performance. It can be passed to the inference script as an argument.<\/p>\n
      \n
    1. If you didn\u2019t download any fine-tuned parameters, you can now fine-tune your model to specify a sequence length and use a GPU.\n