{"id":1445,"date":"2022-01-05T20:23:32","date_gmt":"2022-01-05T20:23:32","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/05\/deploying-ml-models-using-sagemaker-serverless-inference-preview\/"},"modified":"2022-01-05T20:23:32","modified_gmt":"2022-01-05T20:23:32","slug":"deploying-ml-models-using-sagemaker-serverless-inference-preview","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/05\/deploying-ml-models-using-sagemaker-serverless-inference-preview\/","title":{"rendered":"Deploying ML models using SageMaker Serverless Inference (Preview)"},"content":{"rendered":"
\n

Amazon SageMaker Serverless Inference (Preview) was recently announced at re:Invent 2021 as a new model hosting feature that lets customers serve model predictions without having to explicitly provision compute instances or configure scaling policies to handle traffic variations. Serverless Inference is a new deployment capability that complements SageMaker\u2019s existing options for deployment that include: SageMaker Real-Time Inference for workloads with low latency requirements in the order of milliseconds, SageMaker Batch Transform to run predictions on batches of data, and SageMaker Asynchronous Inference for inferences with large payload sizes or requiring long processing times.<\/p>\n

Serverless Inference means that you don\u2019t need to configure and manage the underlying infrastructure hosting your models. When you host your model on a Serverless Inference endpoint, simply select the memory and max concurrent invocations. Then, SageMaker will automatically provision, scale, and terminate compute capacity based on the inference request volume. SageMaker Serverless Inference also means that you only pay for the duration of running the inference code and the amount of data processed, not for idle time. Moreover, you can scale to zero to optimize your inference costs.<\/p>\n

Serverless Inference is a great choice for customers that have intermittent or unpredictable prediction traffic. For example, a document processing service used to extract and analyze data on a periodic basis. Customers that choose Serverless Inference should make sure that their workloads can tolerate cold starts. A cold start can occur when your endpoint doesn\u2019t receive traffic for a period of time. It can also occur when your concurrent requests exceed the current request usage. The cold start time will depend on your model size, how long it takes to download, and your container startup time.<\/p>\n

Let\u2019s look at how it works from a high level view.<\/p>\n

How it works<\/h2>\n

A Serverless Inference endpoint can be setup using the AWS Management Console, any standard AWS SDKs, or the AWS CLI. Because Serverless Inference uses the same APIs as SageMaker Hosting persistent endpoints to configure and deploy endpoints, the steps to create a Serverless Inference endpoint are identical. The only modification required is changes to configuration parameters that are setup on your endpoint configuration.<\/p>\n

\"\"<\/a><\/p>\n

To create a Serverless Inference endpoint, you perform three basic steps:<\/p>\n

Step 1:<\/strong> Create a SageMaker Model that packages your model artifacts for deployment on SageMaker using the CreateModel<\/em><\/a> API. This step can also be done via AWS CloudFormation<\/a> using the AWS::SageMaker::Model <\/a>resource.<\/p>\n

Step 2:<\/strong> Create an endpoint configuration using the CreateEndpointConfig <\/em><\/a>API and the new configuration ServerlessConfig<\/em> options, or selecting the serverless option in the AWS Management Console as shown in the following image. Note that this step can also be done via AWS CloudFormation using the AWS::SageMaker::EndpointConfig<\/a> resource. You must specify the Memory Size<\/em> which, at a minimum, should be as big as your runtime model object, and the Max Concurrency<\/em>, which represents the max concurrent invocations for a single endpoint.<\/p>\n

\"\"<\/a><\/p>\n

Step 3:<\/strong> Finally, using the endpoint configuration that you created in Step 2, create your endpoint using either the AWS Management Console, or programmatically using the CreateEndpoint<\/em><\/a> API. This step can also be done via AWS CloudFormation<\/a> using the AWS::SageMaker::Endpoint<\/a> resource.<\/p>\n

That\u2019s it! Then, SageMaker creates an HTTPS URL that you can use to invoke your endpoint through your client applications using the existing runtime client and the invoke_endpoint<\/em> request.<\/p>\n

\"\"<\/a><\/p>\n

Deep Dive<\/h2>\n

Next, we\u2019ll dive deeper into the high-level steps above by showcasing a detailed how-to for creating a new SageMaker Serverless Inference endpoint.<\/p>\n

Setup and training<\/h3>\n

For preview<\/strong> the following regions<\/strong> are supported<\/strong> so make sure to create a SageMaker Notebook Instance or SageMaker Studio Notebook in one of these regions: us-east-1<\/strong>, us-east-2<\/strong>, us-west-2<\/strong>, eu-west-1<\/strong>, ap-northeast-1<\/strong>, and ap-southeast-2<\/strong>. For this example, we\u2019ll be using the Amazon provided XGBoost Algorithm<\/a> to solve a regression problem with the Abalone dataset<\/a>. The notebook<\/a> code can be found in the sagemaker-examples repository.<\/p>\n

First, we must setup the appropriate SDK clients and retrieve the public dataset for model training. Note that an upgrade to the SDK may be required if you are running on an older version.<\/p>\n

\n
# Setup clients\nimport boto3\nimport sagemaker\nfrom sagemaker.estimator import Estimator\n\n#client setup\nclient = boto3.client(service_name=\"sagemaker\")\nruntime = boto3.client(service_name=\"sagemaker-runtime\")\n\nboto_session = boto3.session.Session()\nregion = boto_session.region_name\nprint(region)\nsagemaker_session = sagemaker.Session()\nbase_job_prefix = \"xgboost-example\"\nrole = sagemaker.get_execution_role()\nprint(role)\ndefault_bucket = sagemaker_session.default_bucket()\ns3_prefix = base_job_prefix\ntraining_instance_type = \"ml.m5.xlarge\"\n\n# retrieve data\n!aws s3 cp s3:\/\/sagemaker-sample-files\/datasets\/tabular\/uci_abalone\/train_csv\/abalone_dataset1_train.csv .\n# upload data to S3\n!aws s3 cp abalone_dataset1_train.csv s3:\/\/{default_bucket}\/xgboost-regression\/train.csv<\/code><\/pre>\n<\/p><\/div>\n
After setting up the clients and downloading the data that will be used to train the model, we can now prepare for model training using SageMaker Training Jobs. In the following, we are performing the steps to configure and fit our model that will be deployed to a serverless endpoint.<\/p>\n
\n
from sagemaker.inputs import TrainingInput\ntraining_path = f\"s3:\/\/{default_bucket}\/xgboost-regression\/train.csv\"\ntrain_input = TrainingInput(training_path, content_type=\"text\/csv\")\n\nmodel_path = f\"s3:\/\/{default_bucket}\/{s3_prefix}\/xgb_model\"\n\n# retrieve xgboost image\nimage_uri = sagemaker.image_uris.retrieve(\n    framework=\"xgboost\",\n    region=region,\n    version=\"1.0-1\",\n    py_version=\"py3\",\n    instance_type=training_instance_type,\n)\n\n# Configure Training Estimator\nxgb_train = Estimator(\n    image_uri=image_uri,\n    instance_type=training_instance_type,\n    instance_count=1,\n    output_path=model_path,\n    sagemaker_session=sagemaker_session,\n    role=role,\n)\n\n# Set Hyperparameters\nxgb_train.set_hyperparameters(\n    objective=\"reg:linear\",\n    num_round=50,\n    max_depth=5,\n    eta=0.2,\n    gamma=4,\n    min_child_weight=6,\n    subsample=0.7,\n    silent=0,\n)\n\n# Fit model\nxgb_train.fit({\"train\": train_input})<\/code><\/pre>\n<\/p><\/div>\n
Model creation<\/h3>\n
Next, we must package our model for deployment on SageMaker. For the Model Creation step, we need two parameters: Image<\/em> and ModelDataUrl<\/em>.<\/p>\n
Image <\/em>points to the container image for inference. Because we are using a SageMaker managed container, we retrieved this for training under the variable image_uri<\/strong>, and we can use the same image for inference. If you are bringing your own custom container, then you must supply your own container image that is compatible for hosting on SageMaker as you would today for hosting a SageMaker Hosting persistent endpoint.<\/p>\n
ModelDataUrl<\/em> points to the Amazon Simple Storage Service (S3)<\/a> URL for the trained model artifact that we will pull from the training estimator.<\/p>\n
\n
# Retrieve model data from training job\nmodel_artifacts = xgb_train.model_data\nmodel_artifacts\nfrom time import gmtime, strftime\nmodel_name = \"xgboost-serverless\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\nprint(\"Model name: \" + model_name)\n\n# dummy environment variables\nbyo_container_env_vars = {\"SAGEMAKER_CONTAINER_LOG_LEVEL\": \"20\", \"SOME_ENV_VAR\": \"myEnvVar\"}\ncreate_model_response = client.create_model(\n ModelName=model_name,\n Containers=[\n {\n \"Image\": image_uri,\n \"Mode\": \"SingleModel\",\n \"ModelDataUrl\": model_artifacts,\n \"Environment\": byo_container_env_vars,\n }\n ],\n ExecutionRoleArn=role,\n)\n\nprint(\"Model Arn: \" + create_model_response[\"ModelArn\"])<\/code><\/pre>\n<\/p><\/div>\n
We can now use our created model to work with creating an Endpoint Configuration, which is where you will add a serverless configuration.<\/p>\n
Endpoint configuration creation<\/h3>\n
Up until now, the steps look identical to if you were deploying a SageMaker Hosting endpoint. This next step is the same. However, you\u2019ll take advantage of a new serverless configuration option in your endpoint configuration. There are two inputs required, and they can be configured to meet your use case:<\/p>\n