{"id":648,"date":"2020-12-02T17:45:57","date_gmt":"2020-12-02T17:45:57","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/12\/02\/configuring-autoscaling-inference-endpoints-in-amazon-sagemaker\/"},"modified":"2020-12-02T17:45:57","modified_gmt":"2020-12-02T17:45:57","slug":"configuring-autoscaling-inference-endpoints-in-amazon-sagemaker","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/12\/02\/configuring-autoscaling-inference-endpoints-in-amazon-sagemaker\/","title":{"rendered":"Configuring autoscaling inference endpoints in Amazon SageMaker"},"content":{"rendered":"
\n

Amazon SageMaker<\/a> is a fully managed service that provides every developer and data scientist with the ability to quickly build, train, and deploy machine learning (ML) models at scale. Amazon SageMaker removes the heavy lifting from each step of the ML process to make it easier to develop high-quality models. You can one-click deploy your ML models for making low latency inferences in real-time on fully managed inference endpoints. Autoscaling is an out-of-the-box feature that monitors your workloads and dynamically adjusts the capacity to maintain steady and predictable performance at the possible lowest cost. When the workload increases, autoscaling brings more instances online. When the workload decreases, autoscaling removes unnecessary instances, helping you reduce your compute cost.<\/p>\n

The following diagram is a sample architecture that showcases how a model is invoked for inference using an Amazon SageMaker endpoint.<\/p>\n

\"\"<\/p>\n

Amazon SageMaker automatically attempts to distribute your instances across Availability Zones. So, we strongly recommend that you deploy multiple instances for each production endpoint for high availability. If you\u2019re using a VPC, configure at least two subnets in different Availability Zones so Amazon SageMaker can distribute your instances across those Availability Zones.<\/p>\n

Amazon SageMaker supports four different ways to implement horizontal scaling of Amazon SageMaker endpoints. You can configure some of these policies using the Amazon SageMaker console, the AWS Command Line Interface<\/a> (AWS CLI), or the AWS SDK\u2019s Application Auto Scaling API for the advanced options. In this post, we showcase how to configure using the boto3 SDK for Python and outline different scaling policies and patterns.\u00a0<\/strong><\/p>\n

Prerequisites<\/h2>\n

This post assumes that you have a functional Amazon SageMaker endpoint deployed. Models are hosted within an Amazon SageMaker endpoint; you can have multiple model versions being served via the same endpoint. Each model is referred to as a production variant<\/a>.<\/p>\n

If you\u2019re new to Amazon SageMaker and have not created an endpoint yet, complete the steps in Identifying bird species on the edge using the Amazon SageMaker built-in Object Detection algorithm and AWS DeepLens<\/a> until the section Testing the model<\/strong> to develop and host an object detection model.<\/p>\n

If you want to get started directly with this post, you can also fetch a model from the MXNet model zoo<\/a>. For example, if you plan to use ResidualNet152, you need the model definition<\/a> and the model weights<\/a> inside a tarball. You can also create custom models that can be hosted as an Amazon SageMaker endpoint. For instructions on building a tarball with Gluon and Apache MXNet, see Deploying custom models built with Gluon and Apache MXNet on Amazon SageMaker<\/a>.<\/p>\n

Configuring autoscaling<\/h2>\n

The following are the high-level steps for creating a model and applying a scaling policy:<\/p>\n

    \n
  1. Use Amazon SageMaker to create a model or bring a custom model.<\/li>\n
  2. Deploy the model.<\/li>\n<\/ol>\n

    If you use the MXNet estimator to train the model, you can call deploy to create an Amazon SageMaker endpoint:<\/p>\n

    \n
    # Train my estimator\r\nmxnet_estimator = MXNet('train.py',\r\n                framework_version='1.6.0',\r\n                py_version='py3',\r\n                instance_type='ml.p2.xlarge',\r\n                instance_count=1)\r\n\r\nmxnet_estimator.fit('s3:\/\/my_bucket\/my_training_data\/')\r\n\r\n# Deploy my estimator to an Amazon SageMaker endpoint and get a Predictor\r\npredictor = mxnet_estimator.deploy(instance_type='ml.m5.xlarge',\r\n                initial_instance_count=1)#Instance_count=1 is not recommended for production use. Use this only for experimentation.<\/code><\/pre>\n<\/div>\n
    If you use a pretrained model like ResidualNet152, you can create an MXNetModel<\/code> object and call deploy<\/code> to create the Amazon SageMaker endpoint:<\/code><\/p>\n
    \n
    mxnet_model = MXNetModel(model_data='s3:\/\/my_bucket\/pretrained_model\/model.tar.gz',\r\n                         role=role,\r\n                         entry_point='inference.py',\r\n                         framework_version='1.6.0',\r\n                         py_version='py3')\r\npredictor = mxnet_model.deploy(instance_type='ml.m5.xlarge',#\r\n                               initial_instance_count=1)<\/code><\/pre>\n<\/div>\n
      \n
    1. Create a scaling policy and apply the scaling policy to the endpoint. The following section discusses your scaling policy options.<\/li>\n<\/ol>\n
      Scaling options<\/h2>\n
      You can define minimum, desired, and maximum number of instances per endpoint and, based on the autoscaling configurations, instances are managed dynamically. The following diagram illustrates this architecture.<\/p>\n
      \"\"<\/p>\n
      To scale the deployed Amazon SageMaker endpoint, first fetch its details:<\/p>\n
      \n
      import pprint\r\nimport boto3\r\nfrom sagemaker import get_execution_role\r\nimport sagemaker\r\nimport json\r\n\r\npp = pprint.PrettyPrinter(indent=4, depth=4)\r\nrole = get_execution_role()\r\nsagemaker_client = boto3.Session().client(service_name='sagemaker')\r\nendpoint_name = 'name-of-the-endpoint'\r\nresponse = sagemaker_client.describe_endpoint(EndpointName=endpoint_name)\r\npp.pprint(response)\r\n\r\n#Let us define a client to play with autoscaling options\r\nclient = boto3.client('application-autoscaling') # Common class representing Application Auto Scaling for SageMaker amongst other services<\/code><\/pre>\n<\/div>\n
      Simple scaling or TargetTrackingScaling<\/h3>\n
      Use this option when you want to scale based on a specific Amazon CloudWatch<\/a> metric. You can do this by choosing a specific metric and setting threshold values. The recommended metrics for this option are average CPUUtilization<\/code> or SageMakerVariantInvocationsPerInstance<\/code>.<\/p>\n
      SageMakerVariantInvocationsPerInstance<\/code> is the average number of times per minute that each instance for a variant is invoked. CPUUtilization<\/code> is the sum of work handled by a CPU.\u00a0<\/em><\/p>\n
      The following code snippets show how to scale using these metrics. You can also push custom metrics to CloudWatch or use other metrics. For more information, see Monitor Amazon SageMaker with Amazon CloudWatch<\/a>.<\/p>\n
      \n
      resource_id='endpoint\/' + endpoint_name + '\/variant\/' + 'AllTraffic' # This is the format in which application autoscaling references the endpoint\r\n\r\nresponse = client.register_scalable_target(\r\n    ServiceNamespace='sagemaker', #\r\n    ResourceId=resource_id,\r\n    ScalableDimension='sagemaker:variant:DesiredInstanceCount',\r\n    MinCapacity=1,\r\n    MaxCapacity=2\r\n)\r\n\r\n#Example 1 - SageMakerVariantInvocationsPerInstance Metric\r\nresponse = client.put_scaling_policy(\r\n    PolicyName='Invocations-ScalingPolicy',\r\n    ServiceNamespace='sagemaker', # The namespace of the AWS service that provides the resource. \r\n    ResourceId=resource_id, # Endpoint name \r\n    ScalableDimension='sagemaker:variant:DesiredInstanceCount', # SageMaker supports only Instance Count\r\n    PolicyType='TargetTrackingScaling', # 'StepScaling'|'TargetTrackingScaling'\r\n    TargetTrackingScalingPolicyConfiguration={\r\n        'TargetValue': 10.0, # The target value for the metric. - here the metric is - SageMakerVariantInvocationsPerInstance\r\n        'PredefinedMetricSpecification': {\r\n            'PredefinedMetricType': 'SageMakerVariantInvocationsPerInstance', # is the average number of times per minute that each instance for a variant is invoked. \r\n        },\r\n        'ScaleInCooldown': 600, # The cooldown period helps you prevent your Auto Scaling group from launching or terminating \r\n                                # additional instances before the effects of previous activities are visible. \r\n                                # You can configure the length of time based on your instance startup time or other application needs.\r\n                                # ScaleInCooldown - The amount of time, in seconds, after a scale in activity completes before another scale in activity can start. \r\n        'ScaleOutCooldown': 300 # ScaleOutCooldown - The amount of time, in seconds, after a scale out activity completes before another scale out activity can start.\r\n        \r\n        # 'DisableScaleIn': True|False - ndicates whether scale in by the target tracking policy is disabled. \r\n                            # If the value is true , scale in is disabled and the target tracking policy won't remove capacity from the scalable resource.\r\n    }\r\n)\r\n\r\n#Example 2 - CPUUtilization metric\r\nresponse = client.put_scaling_policy(\r\n    PolicyName='CPUUtil-ScalingPolicy',\r\n    ServiceNamespace='sagemaker',\r\n    ResourceId=resource_id,\r\n    ScalableDimension='sagemaker:variant:DesiredInstanceCount',\r\n    PolicyType='TargetTrackingScaling',\r\n    TargetTrackingScalingPolicyConfiguration={\r\n        'TargetValue': 90.0,\r\n        'CustomizedMetricSpecification':\r\n        {\r\n            'MetricName': 'CPUUtilization',\r\n            'Namespace': '\/aws\/sagemaker\/Endpoints',\r\n            'Dimensions': [\r\n                {'Name': 'EndpointName', 'Value': endpoint_name },\r\n                {'Name': 'VariantName','Value': 'AllTraffic'}\r\n            ],\r\n            'Statistic': 'Average', # Possible - 'Statistic': 'Average'|'Minimum'|'Maximum'|'SampleCount'|'Sum'\r\n            'Unit': 'Percent'\r\n        },\r\n        'ScaleInCooldown': 600,\r\n        'ScaleOutCooldown': 300\r\n    }\r\n)<\/code><\/pre>\n<\/div>\n
      With the scale-in cooldown period<\/em>, the intention is to scale-in conservatively to protect your application\u2019s availability, so scale-in activities are blocked until the cooldown period has expired. With the scale-out cooldown period<\/em>, the intention is to continuously (but not excessively) scale out. After Application Auto Scaling<\/a> successfully scales out using a target tracking scaling policy, it starts to calculate the cooldown time.<\/p>\n
      Step scaling<\/h3>\n
      This is an advanced type of scaling where you define additional policies to dynamically adjust the number of instances to scale based on size of the alarm breach. This helps you configure a more aggressive response when demand reaches a certain level. The following code is an example of a step scaling policy based on the OverheadLatency<\/code> metric:<\/p>\n
      \n
      #Example 3 - OverheadLatency metric and StepScaling Policy\r\nresponse = client.put_scaling_policy(\r\n    PolicyName='OverheadLatency-ScalingPolicy',\r\n    ServiceNamespace='sagemaker',\r\n    ResourceId=resource_id,\r\n    ScalableDimension='sagemaker:variant:DesiredInstanceCount',\r\n    PolicyType='StepScaling', \r\n    StepScalingPolicyConfiguration={\r\n        'AdjustmentType': 'ChangeInCapacity', # 'PercentChangeInCapacity'|'ExactCapacity' Specifies whether the ScalingAdjustment value in a StepAdjustment \r\n                                              # is an absolute number or a percentage of the current capacity.\r\n        'StepAdjustments': [ # A set of adjustments that enable you to scale based on the size of the alarm breach.\r\n            {\r\n                'MetricIntervalLowerBound': 0.0, # The lower bound for the difference between the alarm threshold and the CloudWatch metric.\r\n                 # 'MetricIntervalUpperBound': 100.0, # The upper bound for the difference between the alarm threshold and the CloudWatch metric.\r\n                'ScalingAdjustment': 1 # The amount by which to scale, based on the specified adjustment type. \r\n                                       # A positive value adds to the current capacity while a negative number removes from the current capacity.\r\n            },\r\n        ],\r\n        # 'MinAdjustmentMagnitude': 1, # The minimum number of instances to scale. - only for 'PercentChangeInCapacity'\r\n        'Cooldown': 120,\r\n        'MetricAggregationType': 'Average', # 'Minimum'|'Maximum'\r\n    }\r\n)<\/code><\/pre>\n<\/div>\n
      Scheduled scaling<\/h3>\n
      You can use this option when you know that the demand follows a particular schedule in the day, week, month, or year. This helps you specify a one-time schedule or a recurring schedule or cron expressions along with start and end times, which form the boundaries of when the autoscaling action starts and stops. See the following code:<\/p>\n
      \n
      #Example 4 - Scaling based on a certain schedule.\r\nresponse = client.put_scheduled_action(\r\n    ServiceNamespace='sagemaker',\r\n    Schedule='at(2020-10-07T06:20:00)', # yyyy-mm-ddThh:mm:ss You can use one-time schedule, cron, or rate\r\n    ScheduledActionName='ScheduledScalingTest',\r\n    ResourceId=resource_id,\r\n    ScalableDimension='sagemaker:variant:DesiredInstanceCount',\r\n    #StartTime=datetime(2020, 10, 7), #Start date and time for when the schedule should begin\r\n    #EndTime=datetime(2020, 10, 8), #End date and time for when the recurring schedule should end\r\n    ScalableTargetAction={\r\n        'MinCapacity': 2,\r\n        'MaxCapacity': 3\r\n    }\r\n)<\/code><\/pre>\n<\/div>\n
      On-demand scaling<\/h3>\n
      Use this option only when you want to increase or decrease the number of instances manually. This updates the endpoint weights and capacities without defining a trigger. See the following code:<\/p>\n
      \n
      response = client.update_endpoint_weights_and_capacities(EndpointName=endpoint_name,\r\n                            DesiredWeightsAndCapacities=[\r\n                                {\r\n                                    'VariantName': 'string',\r\n                                    'DesiredWeight': ...,\r\n                                    'DesiredInstanceCount': 123\r\n                                }\r\n                            ])\r\n<\/code><\/pre>\n<\/div>\n
      Comparing scaling methods<\/h3>\n
      Each of these methods, when successfully applied, results in the addition of instances to an already deployed Amazon SageMaker endpoint. When you make a request to update your endpoint with autoscaling configurations, the status of the endpoint moves to Updating<\/a>. While the endpoint is in this state, other update operations on this endpoint fail. You can monitor the state by using the DescribeEndpoint API<\/a>. There is no traffic interruption while instances are being added to or removed from an endpoint.<\/p>\n
      When creating an endpoint, we specify initial_instance_count<\/code>; this<\/em> value is only used at endpoint creation time. That value is ignored afterward, and autoscaling or on-demand scaling uses the change in desiredInstanceCount<\/code> to set the instance count behind an endpoint.<\/p>\n
      Finally, if you do use UpdateEndpoint<\/a> to deploy a new EndpointConfig<\/code> to an endpoint, to retain the current number of instances, you should set RetainAllVariantProperties<\/a> to true.<\/p>\n
      Considerations for designing an autoscaling policy to scale your ML workload<\/h2>\n
      You should consider the following when designing an efficient autoscaling policy to minimize traffic interruptions and be cost-efficient:<\/p>\n