{"id":216,"date":"2020-09-12T16:01:50","date_gmt":"2020-09-12T16:01:50","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/12\/automated-monitoring-of-your-machine-learning-models-with-amazon-sagemaker-model-monitor-and-sending-predictions-to-human-review-workflows-using-amazon-a2i\/"},"modified":"2020-09-12T16:01:50","modified_gmt":"2020-09-12T16:01:50","slug":"automated-monitoring-of-your-machine-learning-models-with-amazon-sagemaker-model-monitor-and-sending-predictions-to-human-review-workflows-using-amazon-a2i","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/12\/automated-monitoring-of-your-machine-learning-models-with-amazon-sagemaker-model-monitor-and-sending-predictions-to-human-review-workflows-using-amazon-a2i\/","title":{"rendered":"Automated monitoring of your machine learning models with Amazon SageMaker Model Monitor and sending predictions to human review workflows using Amazon A2I"},"content":{"rendered":"<div id=\"\">\n<p>When machine learning (ML) is deployed in production, monitoring the model is important for maintaining the quality of predictions. Although the statistical properties of the training data are known in advance, real-life data can gradually deviate over time and impact the prediction results of your model, a phenomenon known as <em>data drift<\/em>. Detecting these conditions in production can be challenging and time-consuming, and requires a system that captures incoming real-time data, performs statistical analyses, defines rules to detect drift, and sends alerts for rule violations. Furthermore, the process must be repeated for every new iteration of the model.<\/p>\n<p><a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/model-monitor.html\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon SageMaker Model Monitor<\/a> enables you to continuously monitor ML models in production. You can set alerts to detect deviations in the model quality and take corrective actions, such as retraining models, auditing upstream systems, or fixing data quality issues. You can use insights from Model Monitor to proactively determine model prediction variance due to data drift and then use <a href=\"https:\/\/aws.amazon.com\/augmented-ai\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Augmented AI<\/a> (Amazon A2I), a fully managed feature in Amazon SageMaker, to send ML inferences to human workflows for review. You can use Amazon A2I for multiple purposes, such as:<\/p>\n<ul>\n<li>Reviewing results below a threshold<\/li>\n<li>Human oversight and audit use cases<\/li>\n<li>Augmenting AI and ML results as required<\/li>\n<\/ul>\n<p>In this post, we show how to set up an ML workflow on <a href=\"https:\/\/aws.amazon.com\/sagemaker\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon SageMaker<\/a> to train an XGBoost algorithm for breast cancer predictions. We deploy the model on a real-time inference endpoint, launch a model monitoring schedule, evaluate monitoring results, and trigger a human review loop for below-threshold predictions. We then show how the human loop workers review and update the predictions.<\/p>\n<p>We walk you through the following steps using this accompanying Jupyter <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>:<\/p>\n<ol>\n<li>Preprocess your input dataset.<\/li>\n<li>Train an XGBoost model and deploy to a real-time endpoint.<\/li>\n<li>Generate baselines and start Model Monitor.<\/li>\n<li>Review the model monitor reports and derive insights.<\/li>\n<li>Set up a human review loop for low-confidence detection using Amazon A2I.<\/li>\n<\/ol>\n<h2>Prerequisites<\/h2>\n<p>Before getting started, you need to create your human workforce and set up your Amazon SageMaker Studio notebook.<\/p>\n<h3>Creating your human workforce<\/h3>\n<p>For this post, you create a private work team and add only one user (you) to it. For instructions, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/sms-workforce-create-private-console.html#create-workforce-sm-console\" target=\"_blank\" rel=\"noopener noreferrer\">Create an Amazon Cognito Workforce Using the Labeling Workforces Page<\/a>.<img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15751\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/1-Create-Private-Team.jpg\" alt=\"\" width=\"900\" height=\"675\"><\/p>\n<p>Enter your email in the email addresses box for workers. To invite your colleagues to participate in reviewing tasks, include their email addresses in this box.<\/p>\n<p>After you create your private team, you receive an email from no-reply@verificationemail.com that contains your workforce username, password, and a link that you can use to log in to the worker portal. Enter the username and password you received in the email to log in. You must then create a new, non-default password. This is your private worker\u2019s interface.<\/p>\n<p>When you create an Amazon A2I human review task using your private team (explained in the <strong>Starting a human loop<\/strong> section), your task should appear in the <strong>Jobs<\/strong> section. See the following screenshot.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15752\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/2-Instructions.jpg\" alt=\"\" width=\"900\" height=\"423\"><\/p>\n<p>After you create your private workforce, you can view it on the <strong>Labeling workforces <\/strong>page, on the <strong>Private<\/strong> tab.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15753\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/3-Private-tab.jpg\" alt=\"\" width=\"900\" height=\"413\"><\/p>\n<h3>Setting up your Amazon SageMaker Studio notebook<\/h3>\n<p>To set up your notebook, complete the following steps:<\/p>\n<ol>\n<li>\n<a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/onboard-quick-start.html\" target=\"_blank\" rel=\"noopener noreferrer\">Onboard to Amazon SageMaker Studio with the quick start procedure<\/a>.<\/li>\n<li>When you create an <a href=\"http:\/\/aws.amazon.com\/iam\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Identity and Access Management<\/a> (IAM) role to the notebook instance, be sure to specify access to <a href=\"http:\/\/aws.amazon.com\/s3\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Simple Storage Service<\/a> (Amazon S3). You can choose <strong>Any S3 Bucket<\/strong> or specify the S3 bucket you want to enable access to. You can use the AWS-managed policies <a href=\"https:\/\/us-west-2.console.aws.amazon.com\/iam\/home#\/policies\/arn:aws:iam::aws:policy\/AmazonSageMakerFullAccess%24jsonEditor\" target=\"_blank\" rel=\"noopener noreferrer\">AmazonSageMakerFullAccess<\/a> and <a href=\"https:\/\/console.aws.amazon.com\/iam\/home#\/policies\/arn%3Aaws%3Aiam%3A%3Aaws%3Apolicy%2FAmazonAugmentedAIFullAccess\" target=\"_blank\" rel=\"noopener noreferrer\">AmazonAugmentedAIFullAccess<\/a> to grant general access to these two services.<\/li>\n<\/ol>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15754\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/4-Permissions-policies.jpg\" alt=\"\" width=\"900\" height=\"405\"><\/p>\n<ol start=\"3\">\n<li>When user is created and is active, choose <strong>Open Studio<\/strong>.<\/li>\n<\/ol>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15755\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/5-Open-studio.jpg\" alt=\"\" width=\"900\" height=\"297\"><\/p>\n<ol start=\"4\">\n<li>On the Studio landing page, from the <strong>File <\/strong>drop-down menu, choose <strong>New<\/strong>.<\/li>\n<li>Choose <strong>Terminal<\/strong>.<\/li>\n<\/ol>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15756\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/6-SageMaker-studio.jpg\" alt=\"\" width=\"900\" height=\"742\"><\/p>\n<ol start=\"6\">\n<li>In the terminal, enter the following code:<\/li>\n<\/ol>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-bash\">git clone https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks<\/code><\/pre>\n<\/div>\n<\/div>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15757\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/7-SageMaker-studio2.jpg\" alt=\"\" width=\"900\" height=\"438\"><\/p>\n<ol start=\"7\">\n<li>Open the notebook by choosing <strong>Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb<\/strong> in the <code>amazon-a2i-sample-jupyter-notebooks<\/code> folder.<\/li>\n<\/ol>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15758\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/8-SageMaker-studio3.jpg\" alt=\"\" width=\"900\" height=\"506\"><\/p>\n<h2>Preprocessing your input dataset<\/h2>\n<p>You can follow the steps in this post using the accompanying Jupyter <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>. Make sure you provide an S3 bucket and a prefix of your choice. We then import the Python data science libraries and the Amazon SageMaker Python SDK that we need to run through our use case.<\/p>\n<h3>Loading the dataset<\/h3>\n<p>For this post, we use a dataset for breast cancer predictions from the <a href=\"http:\/\/archive.ics.uci.edu\/ml\/index.php\" target=\"_blank\" rel=\"noopener noreferrer\">UCI Machine Learning Repository<\/a>. Please refer to the accompanying Jupyter <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a> for the code to load and split this dataset. Based on the input features, we first train a model to detect a benign (<code>label=0<\/code>) or malignant (<code>label=1<\/code>) condition.<\/p>\n<p>The following screenshot shows some of the rows in the training dataset.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15759\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/9-Screenshot.jpg\" alt=\"\" width=\"900\" height=\"202\"><\/p>\n<h2>Training and deploying an Amazon SageMaker XGBoost model<\/h2>\n<p><a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/xgboost.html\" target=\"_blank\" rel=\"noopener noreferrer\">XGBoost<\/a> (eXtreme Gradient Boosting) is a popular and efficient open-source implementation of the gradient boosted trees algorithm. For our use case, we use the binary:logistic objective. The model applies logistic regression for binary classification (in this example, whether a condition is benign or malignant). The output is a probability that represents the log likelihood of the Bernoulli distribution.<\/p>\n<p>With Amazon SageMaker, you can use XGBoost as a built-in algorithm or framework. For this use case, we use the built-in algorithm. To specify the <a href=\"http:\/\/aws.amazon.com\/ecr\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Elastic Container Registry<\/a> (Amazon ECR) container location for Amazon SageMaker implementation of XGBoost, enter the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">from sagemaker.amazon.amazon_estimator import get_image_uri\r\ncontainer = get_image_uri(boto3.Session().region_name, 'xgboost', '1.0-1')\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<h3>Creating the XGBoost estimator<\/h3>\n<p>We use the XGBoost container to construct an estimator using the Amazon SageMaker Estimator API and initiate a training job (the full walkthrough is available in the accompanying Jupyter <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>):<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">sess = sagemaker.Session()\r\n\r\nxgb = sagemaker.estimator.Estimator(container,\r\n                                    role, \r\n                                    train_instance_count=1, \r\n                                    train_instance_type='ml.m5.2xlarge',\r\n                                    output_path='s3:\/\/{}\/{}\/output'.format(bucket, prefix),\r\n                                    sagemaker_session=sess)\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<h3>Specifying hyperparameters and starting training<\/h3>\n<p>We can now specify the hyperparameters for our training. You set hyperparameters to facilitate the estimation of model parameters from data. See the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">xgb.set_hyperparameters(max_depth=5,\r\n                        eta=0.2,\r\n                        gamma=4,\r\n                        min_child_weight=6,\r\n                        subsample=0.8,\r\n                        silent=0,\r\n                        objective='binary:logistic',\r\n                        num_round=100)\r\n\r\nxgb.fit({'train': s3_input_train, 'validation': s3_input_validation})\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<p>For more information, see <a href=\"https:\/\/xgboost.readthedocs.io\/en\/release_0.90\/parameter.html\" target=\"_blank\" rel=\"noopener noreferrer\">XGBoost Parameters<\/a>.<\/p>\n<h3>Deploying the XGBoost model<\/h3>\n<p>We deploy a model that\u2019s hosted behind a real-time inference endpoint. As a prerequisite, we set up a data_capture_config for the Model Monitor after the endpoint is deployed, which enables Amazon SageMaker to collect the inference requests and responses for use in Model Monitor. For more information, see the accompanying <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>.<\/p>\n<p>The deploy function returns a Predictor object that you can use for inference:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">xgb_predictor = xgb.deploy(initial_instance_count=1,\r\n                           instance_type='ml.m5.2xlarge',\r\n                           endpoint_name=endpoint_name,\r\n                           data_capture_config=data_capture_config)\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<h3>Invoking the deployed model using the endpoint<\/h3>\n<p>You can now send data to this endpoint to get inferences in real time. The request and response payload, along with some additional metadata, is saved in the Amazon S3 location that you specified in <code>DataCaptureConfig<\/code>. You can follow the steps in the walkthrough <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>.<\/p>\n<p>The following JSON code is an example of an inference request and response captured:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15771\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/9A-JSON.jpg\" alt=\"\" width=\"900\" height=\"343\"><\/p>\n<h2>Starting Amazon SageMaker Model Monitor<\/h2>\n<p>Amazon SageMaker Model Monitor continuously monitors the quality of ML models in production. To start using Model Monitor, we create a baseline, inspect the baseline job results, and create a monitoring schedule.<\/p>\n<h3>Creating a baseline<\/h3>\n<p>The baseline calculations of statistics and constraints are needed as a standard against which data drift and other data quality issues can be detected. The training dataset is usually a good baseline dataset. The training dataset schema and the inference dataset schema should match (the number and order of the features). From the training dataset, you can ask Amazon SageMaker to suggest a set of baseline constraints and generate descriptive statistics to explore the data. To create the baseline, you can follow the detailed steps in the walkthrough <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>. See the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\"># Start the baseline job\r\nfrom sagemaker.model_monitor import DefaultModelMonitor\r\nfrom sagemaker.model_monitor.dataset_format import DatasetFormat\r\n\r\nmy_default_monitor = DefaultModelMonitor(\r\n    role=role,\r\n    instance_count=1,\r\n    instance_type='ml.m5.4xlarge',\r\n    volume_size_in_gb=100,\r\n    max_runtime_in_seconds=3600,\r\n)\r\n\r\nmy_default_monitor.suggest_baseline(\r\n    baseline_dataset=baseline_data_uri+'\/train.csv',\r\n    dataset_format=DatasetFormat.csv(header=False), # changed this to header=False since train.csv does not have header. \r\n    output_s3_uri=baseline_results_uri,\r\n    wait=True\r\n)\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<h3>Inspecting baseline job results<\/h3>\n<p>When the baseline job is complete, we can inspect the results. Two files are generated:<\/p>\n<ul>\n<li>\n<strong>statistics.json<\/strong> \u2013 This file is expected to have columnar statistics for each feature in the dataset that is analyzed. For the schema of this file, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/model-monitor-byoc-statistics.html\" target=\"_blank\" rel=\"noopener noreferrer\">Schema for Statistics<\/a>.<\/li>\n<li>\n<strong>constraints.json<\/strong> \u2013 This file is expected to have the constraints on the features observed. For the schema of this file, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/model-monitor-byoc-constraints.html\" target=\"_blank\" rel=\"noopener noreferrer\">Schema for Constraints<\/a>.<\/li>\n<\/ul>\n<p>Model Monitor computes per column\/feature statistics. In the following screenshot, <code>c0<\/code> and <code>c1<\/code> in the <code>name<\/code> column refer to columns in the training dataset without the header row.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15760\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/10-Screenshot.jpg\" alt=\"\" width=\"900\" height=\"110\"><\/p>\n<p>The constraints file is used to express the constraints that a dataset must satisfy. See the following screenshot.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15761\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/11-Screenshot.jpg\" alt=\"\" width=\"900\" height=\"179\"><\/p>\n<p>Next we review the monitoring configuration in the <code>constraints.json<\/code> file:<\/p>\n<ul>\n<li>\n<strong>datatype_check_threshold<\/strong> \u2013 During the baseline step, the generated constraints suggest the inferred data type for each column. You can tune the <code>monitoring_config.datatype_check_threshold<\/code> parameter to adjust the threshold for when it\u2019s flagged as a violation.<\/li>\n<li>\n<strong>domain_content_threshold<\/strong> \u2013 If there are more unknown values for a String field in the current dataset than in the baseline dataset, you can use this threshold to dictate if it needs to be flagged as a violation.<\/li>\n<li>\n<strong>comparison_threshold<\/strong> \u2013 This value is used to calculate model drift.<\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15762\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/12-Screenshot.jpg\" alt=\"\" width=\"900\" height=\"67\"><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15763\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/13-Screenshot-1.jpg\" alt=\"\" width=\"900\" height=\"52\"><\/p>\n<p>For more information about constraints, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/model-monitor-byoc-constraints.html\" target=\"_blank\" rel=\"noopener noreferrer\">Schema for Constraints<\/a>.<\/p>\n<h3>Create a monitoring schedule<\/h3>\n<p>With a monitoring schedule, Amazon SageMaker can start processing jobs at a specified frequency to analyze the data collected during a given period. Amazon SageMaker compares the dataset for the current analysis with the baseline statistics and constraints provided and generates a violations report. To create an hourly monitoring schedule, enter the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">from sagemaker.model_monitor import CronExpressionGenerator\r\nfrom time import gmtime, strftime\r\n\r\nmon_schedule_name = 'xgb-breast-cancer-' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\r\nmy_default_monitor.create_monitoring_schedule(\r\n    monitor_schedule_name=mon_schedule_name,\r\n    endpoint_input=xgb_predictor.endpoint,\r\n    output_s3_uri=s3_report_path,\r\n    statistics=my_default_monitor.baseline_statistics(),\r\n    constraints=my_default_monitor.suggested_constraints(),\r\n    schedule_cron_expression=CronExpressionGenerator.hourly(),\r\n    enable_cloudwatch_metrics=True,\r\n\r\n)<\/code><\/pre>\n<\/div>\n<\/div>\n<p>We then invoke the endpoint continuously to generate traffic for the model monitor to pick up. Because we set up an hourly schedule, we need to wait at least an hour for traffic to be detected.<\/p>\n<h2>Reviewing model monitoring<\/h2>\n<p>The violations file is generated as the output of a MonitoringExecution, which lists the results of evaluating the constraints (specified in the constraints.json file) against the current dataset that was analyzed. For more information about violation checks, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/model-monitor-interpreting-violations.html\" target=\"_blank\" rel=\"noopener noreferrer\">Schema for Violations<\/a>. For our use case, the model monitor detects a data type mismatch violation in one of the requests sent to the endpoint. See the following screenshot.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15764\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/14-Screenshot.jpg\" alt=\"\" width=\"900\" height=\"111\"><\/p>\n<p>For more details, see the walkthrough <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>.<\/p>\n<h3>Evaluating the results<\/h3>\n<p>To determine the next steps for our experiment, we should consider the following two perspectives:<\/p>\n<ul>\n<li>\n<strong>Model Monitor violations:<\/strong> We only saw the datatype_check violation from the Model Monitor; we didn\u2019t see a model drift violation. In our use case, Model Monitor uses the robust comparison method based on the <a href=\"https:\/\/en.m.wikipedia.org\/wiki\/Kolmogorov%E2%80%93Smirnov_test\" target=\"_blank\" rel=\"noopener noreferrer\">two-sample K-S test<\/a> to quantify the distance between the empirical distribution of our test dataset and the cumulative distribution of the baseline dataset. This distance didn\u2019t exceed the value set for the comparison_threshold. The prediction results are aligned with the results in the training dataset.<\/li>\n<li>\n<strong>Probability distribution of prediction results:<\/strong> We used a test dataset of 114 requests. Out of this, we see that the model predicts 60% of the requests to be malignant (over 90% probability output in the prediction results), 30% benign (less than 10% probability output in the prediction results), and the remaining 10% of the requests are indeterminate. The following chart summarizes these findings.<\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15765\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/15-Graph.jpg\" alt=\"\" width=\"900\" height=\"566\"><\/p>\n<p>As a next step, you need to send the prediction results that are distributed with output probabilities of over 10% and less than 90% (because the model can\u2019t predict with sufficient confidence) to a domain expert who can look at the model results and identify if the tumor is benign or malignant. You use Amazon A2I to set up a human review workflow and define conditions for activating the review loop.<\/p>\n<h2>Starting the human review workflow<\/h2>\n<p>To configure your human review workflow, you complete the following high-level steps:<\/p>\n<ol>\n<li>Create the human task UI.<\/li>\n<li>Create the workflow definition.<\/li>\n<li>Set the trigger conditions to activate the human loop.<\/li>\n<li>Start your human loop.<\/li>\n<li>Check that the human loop tasks are complete.<\/li>\n<\/ol>\n<h3>Creating the human task UI<\/h3>\n<p>The following example code shows how to create a human task UI resource, giving a UI template in liquid HTML. This template is rendered to the human workers whenever a human loop is required. You can follow through the complete steps using the accompanying Jupyter <a href=\"https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\/blob\/master\/Amazon-A2I-with-Amazon-SageMaker-Model-Monitor.ipynb\" target=\"_blank\" rel=\"noopener noreferrer\">notebook<\/a>. After the template is defined, set up the UI task function and run it.<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">def create_task_ui():\r\n    '''\r\n    Creates a Human Task UI resource.\r\n\r\n    Returns:\r\n    struct: HumanTaskUiArn\r\n    '''\r\n    response = sagemaker_client.create_human_task_ui(\r\n        HumanTaskUiName=taskUIName,\r\n        UiTemplate={'Content': template})\r\n    return response\r\n# Create task UI\r\nhumanTaskUiResponse = create_task_ui()\r\nhumanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\r\nprint(humanTaskUiArn)\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<h3>Creating the workflow definition<\/h3>\n<p>We create the flow definition to specify the following:<\/p>\n<ul>\n<li>The workforce that your tasks are sent to.<\/li>\n<li>The instructions that your workforce receives. This is specified using a worker task template.<\/li>\n<li>Where your output data is stored.<\/li>\n<\/ul>\n<p>See the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">create_workflow_definition_response = sagemaker_client.create_flow_definition(\r\n        FlowDefinitionName= flowDefinitionName,\r\n        RoleArn= role,\r\n        HumanLoopConfig= {\r\n            \"WorkteamArn\": WORKTEAM_ARN,\r\n            \"HumanTaskUiArn\": humanTaskUiArn,\r\n            \"TaskCount\": 1,\r\n            \"TaskDescription\": \"Review the model predictions and determine if you agree or disagree. Assign a label of 1 to indicate malignant result or 0 to indicate a benign result based on your review of the inference request\",\r\n            \"TaskTitle\": \"Using Model Monitor and A2I Demo\"\r\n        },\r\n        OutputConfig={\r\n            \"S3OutputPath\" : OUTPUT_PATH\r\n        }\r\n    )\r\nflowDefinitionArn = create_workflow_definition_response['FlowDefinitionArn'] # let's save this ARN for future use\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<h3>Setting trigger conditions for human loop activation<\/h3>\n<p>We need to send the prediction results that are distributed with output probabilities of over 10% and under 90% (because the model can\u2019t predict with sufficient confidence in this range). We use this as our activation condition, as shown in the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\"># assign our original test dataset \r\nmodel_data_categorical = test_data[list(test_data.columns)[1:]]  \r\n\r\nLOWER_THRESHOLD = 0.1\r\nUPPER_THRESHOLD = 0.9\r\nsmall_payload_df = model_data_categorical.head(len(predictions))\r\nsmall_payload_df['prediction_prob'] = predictions\r\nsmall_payload_df_res = small_payload_df.loc[\r\n    (small_payload_df['prediction_prob'] &gt; LOWER_THRESHOLD) &amp;\r\n    (small_payload_df['prediction_prob'] &lt; UPPER_THRESHOLD)\r\n]\r\nprint(small_payload_df_res.shape)\r\nsmall_payload_df_res.head(10)\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<h3>Starting a human loop<\/h3>\n<p>A human loop starts your human review workflow and sends data review tasks to human workers. See the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\"># Activate human loops\r\nimport json\r\nhumanLoopName = str(uuid.uuid4())\r\n\r\nstart_loop_response = a2i.start_human_loop(\r\n            HumanLoopName=humanLoopName,\r\n            FlowDefinitionArn=flowDefinitionArn,\r\n            HumanLoopInput={\r\n                \"InputContent\": json.dumps(ip_content)\r\n            }\r\n        )\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<p>The workers in this use case are domain experts that can validate the request features and determine if the result is malignant or benign. The task requires reviewing the model predictions, agreeing or disagreeing, and updating the prediction as 1 for malignant and 0 for benign. The following screenshot shows a sample of tasks received.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15766\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/16-Instructions-chart.jpg\" alt=\"\" width=\"900\" height=\"374\"><\/p>\n<p>The following screenshot shows updated predictions.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15767\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/17-Predictions-table.jpg\" alt=\"\" width=\"900\" height=\"520\"><\/p>\n<p>For more information about task UI design for tabular datasets, see <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/using-amazon-sagemaker-with-amazon-augmented-ai-for-human-review-of-tabular-data-and-ml-predictions\/\" target=\"_blank\" rel=\"noopener noreferrer\">Using Amazon SageMaker with Amazon Augmented AI for human review of Tabular data and ML predictions<\/a>.<\/p>\n<h3>Checking the status of task completion and human loop<\/h3>\n<p>To check the status of the task and the human loop, enter the following code:<\/p>\n<div class=\"hide-language\">\n<div class=\"hide-language\">\n<pre class=\"unlimited-height-code\"><code class=\"lang-python\">completed_human_loops = []\r\nresp = a2i.describe_human_loop(HumanLoopName=humanLoopName)\r\nprint(f'HumanLoop Name: {humanLoopName}')\r\nprint(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\r\nprint(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\r\nprint('n')\r\n    \r\nif resp[\"HumanLoopStatus\"] == \"Completed\":\r\n    completed_human_loops.append(resp)\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<p>When the human loop tasks are complete, we inspect the results of the review and the corrections made to prediction results.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-15774\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/18-Code.jpg\" alt=\"\" width=\"900\" height=\"129\"><\/p>\n<p>You can use the human-labeled output to augment the training dataset for retraining. This keeps the distribution variance within the threshold and prevents data drift, thereby improving model accuracy. For more information about using Amazon A2I outputs for model retraining, see <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/object-detection-and-model-retraining-with-amazon-sagemaker-and-amazon-augmented-ai\/\" target=\"_blank\" rel=\"noopener noreferrer\">Object detection and model retraining with Amazon SageMaker and Amazon Augmented AI<\/a>.<\/p>\n<h2>Cleaning up<\/h2>\n<p>To avoid incurring unnecessary charges, delete the resources used in this walkthrough when not in use, including the following:<\/p>\n<h2>Conclusion<\/h2>\n<p>This post demonstrated how you can use Amazon SageMaker Model Monitor and Amazon A2I to set up a monitoring schedule for your Amazon SageMaker model endpoints; specify baselines that include constraint thresholds; observe inference traffic; derive insights such as model drift, completeness, and data type violations; and send the low-confidence predictions to a human workflow with labelers to review and update the results. For video presentations, sample Jupyter notebooks, and more information about use cases like document processing, content moderation, sentiment analysis, object detection, text translation, and more, see <a href=\"https:\/\/aws.amazon.com\/augmented-ai\/resources\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon A2I Resources<\/a>.<\/p>\n<p>\u00a0<\/p>\n<h3>References<\/h3>\n<p>[1] Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http:\/\/archive.ics.uci.edu\/ml]. Irvine, CA: University of California, School of Information and Computer Science.<\/p>\n<hr>\n<h3>About the Authors<\/h3>\n<p>\u00a0<\/p>\n<p><strong><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-15769 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/PremRanga.jpg\" alt=\"\" width=\"100\" height=\"100\">Prem Ranga<\/strong> is an Enterprise Solutions Architect at AWS based out of Houston, Texas. He is part of the Machine Learning Technical Field Community and loves working with customers on their ML and AI journey. Prem is passionate about robotics, is an Autonomous Vehicles researcher, and also built the Alexa-controlled Beer Pours in Houston and other locations.<\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><strong><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-15768 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/JasperHuang.jpg\" alt=\"\" width=\"100\" height=\"134\">Jasper Huang<\/strong> is a Technical Writer Intern at AWS and a student at the University of Pennsylvania pursuing a BS and MS in computer science. His interests include cloud computing, machine learning, and how these technologies can be leveraged to solve interesting and complex problems. Outside of work, you can find Jasper playing tennis, hiking, or reading about emerging trends.<\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><strong><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-15770 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/10\/TaliaChopra.jpg\" alt=\"\" width=\"100\" height=\"100\">Talia Chopra<\/strong> is a Technical Writer in AWS specializing in machine learning and artificial intelligence. She works with multiple teams in AWS to create technical documentation and tutorials for customers using Amazon SageMaker, MxNet, and AutoGluon. In her free time, she enjoys meditating, studying machine learning, and taking walks in nature.<\/p>\n<p>\u00a0<\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/aws.amazon.com\/blogs\/machine-learning\/automated-monitoring-of-your-machine-learning-models-with-amazon-sagemaker-model-monitor-and-sending-predictions-to-human-review-workflows-using-amazon-a2i\/<\/p>\n","protected":false},"author":0,"featured_media":217,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/216"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=216"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/216\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/217"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=216"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=216"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=216"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}