{"id":1439,"date":"2022-01-04T20:47:43","date_gmt":"2022-01-04T20:47:43","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/04\/take-advantage-of-advanced-deployment-strategies-using-amazon-sagemaker-deployment-guardrails\/"},"modified":"2022-01-04T20:47:43","modified_gmt":"2022-01-04T20:47:43","slug":"take-advantage-of-advanced-deployment-strategies-using-amazon-sagemaker-deployment-guardrails","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/04\/take-advantage-of-advanced-deployment-strategies-using-amazon-sagemaker-deployment-guardrails\/","title":{"rendered":"Take advantage of advanced deployment strategies using Amazon SageMaker deployment guardrails"},"content":{"rendered":"
\n

Deployment guardrails<\/a> in Amazon SageMaker<\/a> provide a new set of deployment capabilities allowing you to implement advanced deployment strategies that minimize risk when deploying new model versions on SageMaker hosting. Depending on your use case, you can use a variety of deployment strategies to release new model versions. Each of these strategies relies on a mechanism to shift inference traffic to one or more versions of a deployed model. The chosen strategy depends on your business requirements for your machine learning (ML) use case. However, any strategy should include the ability to monitor the performance of new model versions and automatically roll back to a previous version as needed to minimize potential risk of introducing a new model version with errors. Deployment guardrails offer new advanced deployment capabilities and as of this writing supports two new traffic shifting policies, canary<\/a> and linear<\/a>, as well as the ability to automatically roll back when issues are detected.<\/p>\n

As part of your MLOps strategy to create repeatable and reliable mechanisms to deploy your models, you should also ensure that the chosen deployment strategy is implemented as part of your automated deployment pipeline. Deployment guardrails use the existing SageMaker CreateEndpoint<\/a> and UpdateEndpoint<\/a> APIs, so you can modify your existing deployment pipeline configurations to take advantage of the new deployment capabilities.<\/p>\n

In this post, we show you how to use the new deployment guardrail capabilities to deploy your model versions using both a canary and linear deployment strategy.<\/p>\n

Solution overview<\/h2>\n

Amazon SageMaker inference<\/a> provides managed deployment strategies for testing new versions of your models in production. We cover two new traffic shifting policies in this post: canary<\/a> and linear<\/a>. For each of these traffic shifting modes, two HTTPS endpoints are provisioned. Two endpoints are provisioned to reduce deployment risk as traffic is shifted from the original endpoint variant to the new endpoint variant. You configure the endpoints to contain one or more compute instances to deploy your trained model and perform inference requests. SageMaker manages the routing of traffic between the two endpoints. You define Amazon CloudWatch<\/a> metrics and alarms to monitor metrics on the new endpoint, when traffic is shifted, for a set baking period. If a CloudWatch alarm is triggered, SageMaker performs an auto-rollback to route all traffic to the original endpoint variant. If no CloudWatch alarms are triggered, the original endpoint variant is stopped and the new endpoint variant continues to receive all traffic. The following diagrams illustrate shifting traffic to the new endpoint.<\/p>\n

\"\"<\/a><\/p>\n

\"\"<\/a> \"\"<\/a><\/p>\n

Let\u2019s dive deeper into examples of the canary and linear traffic shifting policies.<\/p>\n

We go over the following high-level steps as part of the deployment procedure:<\/p>\n

    \n
  1. Create the model and endpoint configurations required for the three scenarios: the baseline, the update containing the incompatible model version, and the update with the correct model version.<\/li>\n
  2. Invoke the baseline endpoint prior to the update.<\/li>\n
  3. Specify the CloudWatch alarms used to trigger the rollbacks.<\/li>\n
  4. Update the endpoint to trigger a rollback using either the canary or linear strategy.<\/li>\n<\/ol>\n

    First, let\u2019s start with canary deployment.<\/p>\n

    Canary deployment<\/h2>\n

    The canary deployment option lets you shift one small portion of your traffic (a canary<\/em>) to the green fleet and monitor it for a baking period. If the canary succeeds on the green fleet, the rest of the traffic is shifted from the blue fleet to the green fleet before stopping the blue fleet.<\/p>\n

    To demonstrate canary deployments and the auto-rollback feature, we update an endpoint with an incompatible model version and deploy it as a canary fleet, taking a small percentage of the traffic. Requests sent to this canary fleet result in errors, which trigger a rollback using preconfigured CloudWatch alarms. We also demonstrate a success scenario where no alarms are tripped and the update succeeds.<\/p>\n

    Create and deploy the models<\/h3>\n

    First, we upload our pre-trained models to Amazon Simple Storage Service<\/a> (Amazon S3). These models were trained using the XGBoost churn prediction notebook<\/a> in SageMaker. You can also use your own pre-trained models in this step. If you already have a pre-trained model in Amazon S3, you can add it by specifying the s3_key<\/code>.<\/p>\n

    The models in this example are used to predict the probability of a mobile customer leaving their current mobile operator. The dataset we use is publicly available and was mentioned in the book Discovering Knowledge in Data<\/a> by Daniel T. Larose.<\/p>\n

    Upload the models with the following code:<\/p>\n

    \n
    model_url = S3Uploader.upload(local_path=\"model\/xgb-churn-prediction-model.tar.gz\",\ndesired_s3_uri=f\"s3:\/\/{bucket}\/{prefix}\")\nmodel_url2 = S3Uploader.upload(local_path=\"model\/xgb-churn-prediction-model2.tar.gz\",\ndesired_s3_uri=f\"s3:\/\/{bucket}\/{prefix}\")<\/code><\/pre>\n<\/p><\/div>\n
    Next, we create our model definitions. We start with deploying the pre-trained churn prediction models. Here, we create the model objects with the image and model data. The three URIs correspond to the baseline version, the update containing the incompatible version, and the update containing the correct model version:<\/p>\n
    \n
    image_uri = image_uris.retrieve('xgboost', boto3.Session().region_name, '0.90-1')\n# using newer version of XGBoost which is incompatible, in order to simulate model faults\n\nimage_uri2 = image_uris.retrieve('xgboost', boto3.Session().region_name, '1.2-1')\nimage_uri3 = image_uris.retrieve('xgboost', boto3.Session().region_name, '0.90-2')\nmodel_name = f\"DEMO-xgb-churn-pred-{datetime.now():%Y-%m-%d-%H-%M-%S}\" \nmodel_name2 = f\"DEMO-xgb-churn-pred2-{datetime.now():%Y-%m-%d-%H-%M-%S}\"\nmodel_name3 = f\"DEMO-xgb-churn-pred3-{datetime.now():%Y-%m-%d-%H-%M-%S}\"\n\nresp = sm.create_model(\n    ModelName=model_name,\n    ExecutionRoleArn=role,\n    Containers=[{\n       'Image': image_uri,\n       'ModelDataUrl': model_url\n     }])\n\nresp = sm.create_model(\n    ModelName=model_name2,\n    ExecutionRoleArn=role,\n    Containers=[{\n       'Image':image_uri2,\n       'ModelDataUrl': model_url2\n     }])\n\nresp = sm.create_model(\n    ModelName=model_name3,\n    ExecutionRoleArn=role,\n    Containers=[{\n       'Image':image_uri3,\n       'ModelDataUrl': model_url2\n     }])<\/code><\/pre>\n<\/p><\/div>\n
    Now that the three models are created, we create the three endpoint configs:<\/p>\n
    \n
    ep_config_name = f\"DEMO-EpConfig-1-{datetime.now():%Y-%m-%d-%H-%M-%S}\" \nep_config_name2 = f\"DEMO-EpConfig-2-{datetime.now():%Y-%m-%d-%H-%M-%S}\" \nep_config_name3 = f\"DEMO-EpConfig-3-{datetime.now():%Y-%m-%d-%H-%M-%S}\"\n\nresp = sm.create_endpoint_config(\n     EndpointConfigName=ep_config_name,\n     ProductionVariants=[\n        {\n          'VariantName': \"AllTraffic\",\n          'ModelName': model_name,\n          'InstanceType': \"ml.m5.xlarge\",\n          \"InitialInstanceCount\": 3\n        }\n      ])\n\nresp = sm.create_endpoint_config(\n     EndpointConfigName=ep_config_name2,\n     ProductionVariants=[\n        {\n          'VariantName': \"AllTraffic\",\n          'ModelName': model_name2,\n          'InstanceType': \"ml.m5.xlarge\",\n          \"InitialInstanceCount\": 3\n        }\n      ])\n\n\nresp = sm.create_endpoint_config(\n      EndpointConfigName=ep_config_name3,\n      ProductionVariants=[\n         {\n           'VariantName': \"AllTraffic\",\n           'ModelName': model_name3,\n           'InstanceType': \"ml.m5.xlarge\",\n           \"InitialInstanceCount\": 3\n         }\n     ])<\/code><\/pre>\n<\/p><\/div>\n
    We then deploy the baseline model to a SageMaker endpoint:<\/p>\n
    \n
    resp = sm.create_endpoint(\n          EndpointName=endpoint_name,\n          EndpointConfigName=ep_config_name\n)<\/code><\/pre>\n<\/p><\/div>\n
    Invoke the endpoint<\/h3>\n
    This step invokes the endpoint with sample data with a maximum invocations count and waiting intervals. See the following code:<\/p>\n
    \n
    def invoke_endpoint(endpoint_name, max_invocations=300, wait_interval_sec=1, should_raise_exp=False):\n    print(f\"Sending test traffic to the endpoint {endpoint_name}. nPlease wait...\")\n \n    count = 0\n    with open('test_data\/test-dataset-input-cols.csv', 'r') as f:\n        for row in f:\n            payload = row.rstrip('n')\n            try:\n                response = sm_runtime.invoke_endpoint(EndpointName=endpoint_name,\n                                                      ContentType='text\/csv', \n                                                      Body=payload)\n                response['Body'].read()\n                print(\".\", end=\"\", flush=True)\n            except Exception as e:\n                print(\"E\", end=\"\", flush=True)\n                if should_raise_exp:\n                    raise e\n            count += 1\n            if count > max_invocations:\n                break\n            time.sleep(wait_interval_sec)\n \n    print(\"nDone!\")\n \ninvoke_endpoint(endpoint_name, max_invocations=100)<\/code><\/pre>\n<\/p><\/div>\n
    For a full list of metrics, see Monitor Amazon SageMaker with Amazon CloudWatch<\/a>.<\/p>\n
    Then we plot graphs to show the metrics Invocations<\/code>, Invocation4XXErrors<\/code>, Invocation5XXErrors<\/code>, ModelLatency<\/code>, and OverheadLatency<\/code> against the endpoint over time.<\/p>\n
    You can observe a flat line for Invocation4XXErrors<\/code> and Invocation5XXErrors<\/code> because we\u2019re using the correct version model version and configs. Additionally, ModelLatency<\/code> and OverheadLatency<\/code> start decreasing over time.<\/p>\n
    Create CloudWatch alarms to monitor endpoint performance<\/h3>\n
    We create CloudWatch alarms to monitor endpoint performance with the metrics Invocation5XXErrors<\/code> and ModelLatency<\/code>.<\/p>\n
    We use metric dimensions EndpointName<\/code> and VariantName<\/code> to select the metric for each endpoint config and variant. See the following code:<\/p>\n
    \n
    def create_auto_rollback_alarm(alarm_name, endpoint_name, variant_name, metric_name, statistic, threshold):\ncw.put_metric_alarm(\nAlarmName=alarm_name,\nAlarmDescription='Test SageMaker endpoint deployment auto-rollback alarm',\nActionsEnabled=False,\nNamespace='AWS\/SageMaker',\nMetricName=metric_name,\nStatistic=statistic,\nDimensions=[\n{\n'Name': 'EndpointName',\n'Value': endpoint_name\n},\n{\n'Name': 'VariantName',\n'Value': variant_name\n}\n],\nPeriod=60,\nEvaluationPeriods=1,\nThreshold=threshold,\nComparisonOperator='GreaterThanOrEqualToThreshold',\nTreatMissingData='notBreaching'\n)\n\n# alarm on 1% 5xx error rate for 1 minute\ncreate_auto_rollback_alarm(error_alarm, endpoint_name, 'AllTraffic', 'Invocation5XXErrors', 'Average', 1)\n# alarm on model latency >= 10 ms for 1 minute\ncreate_auto_rollback_alarm(latency_alarm, endpoint_name, 'AllTraffic', 'ModelLatency', 'Average', 10000)<\/code><\/pre>\n<\/p><\/div>\n
    Update the endpoint with deployment configurations<\/h3>\n
    We define the following deployment configuration to perform a blue\/green update strategy with canary traffic shifting from the old to the new stack. The canary traffic shifting option can reduce the blast ratio of a regressive update to the endpoint. In contrast, for the all-at-once traffic shifting option, the invocation requests start faulting at 100% after flipping the traffic. In canary mode, invocation requests are shifted to the new version of model gradually, preventing errors from impacting 100% of the traffic. Additionally, the auto-rollback alarms monitor the metrics during the canary stage.<\/p>\n
    The following diagram illustrates the workflow of our rollback use case.<\/p>\n
    \"\"<\/a><\/p>\n
    We update the endpoint with an incompatible model version to simulate errors and trigger a rollback:<\/p>\n
    \n
    canary_deployment_config = {\n    \"BlueGreenUpdatePolicy\": {\n        \"TrafficRoutingConfiguration\": {\n            \"Type\": \"CANARY\",\n            \"CanarySize\": {\n                \"Type\": \"INSTANCE_COUNT\", # or use \"CAPACITY_PERCENT\" as 30%, 50%\n                \"Value\": 1\n            },\n            \"WaitIntervalInSeconds\": 300, # wait for 5 minutes before enabling traffic on the rest of fleet\n        },\n        \"TerminationWaitInSeconds\": 120, # wait for 2 minutes before terminating the old stack\n        \"MaximumExecutionTimeoutInSeconds\": 1800 # maximum timeout for deployment\n    },\n    \"AutoRollbackConfiguration\": {\n        \"Alarms\": [\n            {\n                \"AlarmName\": error_alarm\n            },\n            {\n                \"AlarmName\": latency_alarm\n            }\n        ],\n    }\n}\n \n# update endpoint request with new DeploymentConfig parameter\nsm.update_endpoint(\n    EndpointName=endpoint_name,\n    EndpointConfigName=ep_config_name2,\n    DeploymentConfig=canary_deployment_config\n)<\/code><\/pre>\n<\/p><\/div>\n
    When we invoke the endpoint, we encounter errors because of the incompatible version of the model (ep_config_name2<\/code>), and this leads to the rollback to a stable version of the model (ep_config_name1<\/code>). This is reflected in the following graphs as Invocation5XXErrors<\/code> and ModelLatency<\/code> increase during this rollback phase.<\/p>\n
    The following diagram shows a success case where we use the same canary deployment configuration but a valid endpoint configuration.<\/p>\n
    \"\"<\/a><\/p>\n
    We update the endpoint configuration to a valid version (using the same canary deployment config as the rollback case):<\/p>\n
    \n
    # update endpoint with a valid version of DeploymentConfig\nsm.update_endpoint(\n    EndpointName=endpoint_name,\n    EndpointConfigName=ep_config_name3,\n    RetainDeploymentConfig=True\n)<\/code><\/pre>\n<\/p><\/div>\n
    We plot graphs to show the Invocations<\/code>, Invocation5XXErrors<\/code>, and ModelLatency<\/code> metrics against the endpoint. When the new endpoint config-3<\/code> (correct model version) starts getting deployed, it takes over endpoint config-2<\/code> (incompatible due to model version) without any errors. We can see this in the graphs as Invocation5XXErrors<\/code> and ModelLatency<\/code> decrease during this transition phase.<\/p>\n
    Next, let\u2019s see how linear deployments are configured and how it works.<\/p>\n
    Linear deployment<\/h2>\n
    The linear deployment option provides even more customization over how many traffic-shifting steps to make and what percentage of traffic to shift for each step. Whereas canary shifting lets you shift traffic in two steps, linear shifting extends this to n<\/em> linearly spaced steps.<\/p>\n
    To demonstrate linear deployments and the auto-rollback feature, we update an endpoint with an incompatible model version and deploy it as a linear fleet, taking a small percentage of the traffic. Requests sent to this linear fleet result in errors, which triggers a rollback using preconfigured CloudWatch alarms. We also demonstrate a success scenario where no alarms are tripped and the update succeeds.<\/p>\n
    The steps to create the models, invoke the endpoint, and create the CloudWatch alarms are the same as with the canary method.<\/p>\n
    We define the following deployment configuration to perform a blue\/green update strategy with linear traffic shifting from old to new stack. The linear traffic shifting option can reduce the blast ratio of a regressive update to the endpoint. In contrast, for the all-at-once traffic shifting option, the invocation requests start faulting at 100% after flipping the traffic. In linear mode, invocation requests are shifted to the new version of the model gradually, with a controlled percentage of traffic shifting for each step. You can use the auto-rollback alarms to monitor the metrics during the linear traffic shifting stage.<\/p>\n
    The following diagram shows the workflow for our linear rollback case.<\/p>\n
    \"\"<\/a><\/p>\n
    We update the endpoint with an incompatible model version to simulate errors and trigger a rollback:<\/p>\n
    \n
    linear_deployment_config = {\n    \"BlueGreenUpdatePolicy\": {\n        \"TrafficRoutingConfiguration\": {\n            \"Type\": \"LINEAR\",\n            \"LinearStepSize\": {\n                \"Type\": \"CAPACITY_PERCENT\",\n                \"Value\": 33, # 33% of whole fleet capacity (33% * 3 = 1 instance)\n            },\n            \"WaitIntervalInSeconds\": 180, # wait for 3 minutes before enabling traffic on the rest of fleet\n        },\n        \"TerminationWaitInSeconds\": 120, # wait for 2 minutes before terminating the old stack\n        \"MaximumExecutionTimeoutInSeconds\": 1800 # maximum timeout for deployment\n    },\n    \"AutoRollbackConfiguration\": {\n        \"Alarms\": [\n            {\n                \"AlarmName\": error_alarm\n            },\n            {\n                \"AlarmName\": latency_alarm\n            }\n        ],\n    }\n}\n \n# update endpoint request with new DeploymentConfig parameter\nsm.update_endpoint(\n    EndpointName=endpoint_name,\n    EndpointConfigName=ep_config_name2,\n    DeploymentConfig=linear_deployment_config\n)<\/code><\/pre>\n<\/p><\/div>\n
    When we invoke the endpoint, we encounter errors because of the incompatible version of the model (ep_config_name2<\/code>), which leads to the rollback to a stable version of the model (ep_config_name1<\/code>). We can see this in the following graphs as the Invocation5XXErrors<\/code> and ModelLatency metrics increase during this rollback phase.<\/p>\n
    Let\u2019s look at a success case where we use the same linear deployment configuration but a valid endpoint configuration. The following diagram illustrates our workflow.<\/p>\n
    \"\"<\/a><\/p>\n
    We update the endpoint to a valid endpoint configuration version with the same linear deployment configuration:<\/p>\n
    \n
    # update endpoint with a valid version of DeploymentConfig\nsm.update_endpoint(\n    EndpointName=endpoint_name,\n    EndpointConfigName=ep_config_name3,\n    RetainDeploymentConfig=True\n)<\/code><\/pre>\n<\/p><\/div>\n
    Then we plot graphs to show the Invocations<\/code>, Invocation5XXErrors<\/code>, and ModelLatency<\/code> metrics against the endpoint.<\/p>\n
    As the new endpoint config-3<\/code> (correct model version) starts getting deployed, it takes over endpoint config-2<\/code> (incompatible due to model version) without any errors. We can see this in the following graphs as Invocation5XXErrors<\/code> and ModelLatency<\/code> decrease during this transition phase.<\/p>\n
    Considerations and best practices<\/h2>\n
    Now that we\u2019ve walked through a comprehensive example, let\u2019s recap some best practices and considerations:<\/p>\n