{"id":1141,"date":"2021-11-03T08:40:21","date_gmt":"2021-11-03T08:40:21","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/03\/automate-model-retraining-with-amazon-sagemaker-pipelines-when-drift-is-detected\/"},"modified":"2021-11-03T08:40:21","modified_gmt":"2021-11-03T08:40:21","slug":"automate-model-retraining-with-amazon-sagemaker-pipelines-when-drift-is-detected","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/03\/automate-model-retraining-with-amazon-sagemaker-pipelines-when-drift-is-detected\/","title":{"rendered":"Automate model retraining with Amazon SageMaker Pipelines when drift is detected"},"content":{"rendered":"
\n

Training your machine learning (ML) model and serving predictions is usually not the end of the ML project. The accuracy of ML models can deteriorate over time, a phenomenon known as model drift<\/em>. Many factors can cause model drift, such as changes in model features. The accuracy of ML models can also be affected by concept drift<\/em>, the difference between data used to train models and data used during inference. Therefore, teams must make sure that their models and solutions remain relevant and continue providing value back to the business. Without proper metrics, alarms, and automation in place, the technical debt from simply maintaining existing ML models in production can become overwhelming.<\/p>\n

Amazon SageMaker Pipelines<\/a> is a native workflow orchestration tool<\/a> for building ML pipelines that take advantage of direct Amazon SageMaker<\/a> integration. Three components improve the operational resilience and reproducibility of your ML workflows: pipelines, model registry, and projects. These workflow automation components enable you to easily scale your ability to build, train, test, and deploy hundreds of models in production, iterate faster, reduce errors due to manual orchestration, and build repeatable mechanisms.<\/p>\n

In this post, we discuss how to automate retraining with pipelines in SageMaker when model drift is detected.<\/p>\n

From static models to continuous training<\/h2>\n

Static models are a great place to start when you\u2019re experimenting with ML. However, because real-world data is always changing, static models degrade over time, and your training dataset won\u2019t represent real behavior for long. Having an effective deployment model monitoring phase<\/a> is an important step when building an MLOps pipeline. It\u2019s also one of the most challenging aspects of MLOps because it requires having an effective feedback loop between the data captured by your production system and the data distribution used during the training phase. For model retraining to be effective, you also must be continuously updating your training dataset with new ground truth labels. You might be able to use implicit or explicit feedback from users based on the predictions you provide, such as in the case of recommendations. Alternatively, you may need to introduce a human in the loop workflow through a service like Amazon Augmented AI <\/a>(Amazon A2I) to qualify the accuracy of predictions from your ML system. Other considerations are to monitor predictions for bias on a regular basis, which can be supported through Amazon SageMaker Clarify<\/a>.<\/p>\n

In this post, we propose a solution that focuses on data quality monitoring<\/a> to detect concept drift in the production data and retrain your model automatically.<\/p>\n

Solution overview<\/h2>\n

Our solution uses an open-source AWS CloudFormation<\/a> template to create a model build and deployment pipeline. We use Pipelines and supporting AWS services, including AWS CodePipeline<\/a>, AWS CodeBuild<\/a>, and Amazon EventBridge<\/a>.<\/p>\n

The following diagram illustrates our architecture.
\"\"<\/a><\/p>\n

The following are the high-level steps for this solution:<\/p>\n

    \n
  1. Create the new Amazon SageMaker Studio<\/a> project<\/a> based on the custom template.<\/li>\n
  2. Create a SageMaker pipeline to perform data preprocessing, generate baseline statistics, and train a model.<\/li>\n
  3. Register the model in the SageMaker Model Registry<\/a>.<\/li>\n
  4. The data scientist verifies the models metrics and performance and approves the model.<\/li>\n
  5. The model is deployed to a real-time endpoint in staging and, after approval, to a production endpoint.<\/li>\n
  6. Amazon SageMaker Model Monitor<\/a> is configured on the production endpoint to detect a concept drift of the data<\/a> with respect to the training baseline.<\/li>\n
  7. Model Monitor is scheduled to run every hour, and publishes metrics to Amazon CloudWatch<\/a>.<\/li>\n
  8. A CloudWatch alarm is raised when metrics exceed a model-specific threshold. This results in an EventBridge rule starting the model build pipeline.<\/li>\n
  9. The model build pipeline can also be retrained with an EventBridge rule that runs on a schedule.<\/li>\n<\/ol>\n

    Dataset<\/h2>\n

    This solution uses the New York City Taxi and Limousine Commission (TLC) Trip Record Data<\/a> public dataset to train a model to predict taxi fare based on the information available for that trip. The available information includes the start and end location and travel date and time from which we engineer datetime features and distance traveled.<\/p>\n

    For example, in the following image, we see an example trip for location 65 (Downtown Brooklyn) to 68 (East Chelsea), which took 21 minutes and cost $20.<\/p>\n

    \"\"<\/a><\/p>\n

    If you\u2019re interested in understanding more about the dataset, you can use the exploratory data analysis<\/a> notebook in the GitHub repository.<\/p>\n

    Get started<\/h2>\n

    You can use the following quick start button to launch a CloudFormation stack to publish the custom SageMaker MLOps project template to the AWS Service Catalog:<\/p>\n

    \"\"<\/a><\/p>\n

    Create a new project in Studio<\/h2>\n

    After your MLOps project template is published, you can create a new project using your new template via the Studio UI<\/a>.<\/p>\n

      \n
    1. In the Studio sidebar, choose SageMaker Components and registries.<\/strong><\/li>\n
    2. Choose Projects<\/strong> on the drop-down menu.<\/li>\n
    3. Choose Create Project<\/strong>.<\/li>\n<\/ol>\n

      On the Create project<\/strong> page, SageMaker templates is chosen by default. This option lists the built-in templates. However, you want to use the template you published for the drift detection pipeline.<\/p>\n

        \n
      1. Choose Organization templates<\/strong>.<\/li>\n
      2. Choose Amazon SageMaker drift detection template for real-time deployment<\/strong>.<\/li>\n
      3. Choose Select project template<\/strong>.
        \"\"<\/a><\/li>\n<\/ol>\n

        If you have recently updated your AWS Service Catalog project, you may need to refresh Studio to make sure it finds the latest version of your template.<\/p>\n

          \n
        1. In the Project details<\/strong> section, for Name<\/strong>, enter drift-detection<\/code>.<\/li>\n<\/ol>\n

          Your project name must have 32 characters or fewer.<\/p>\n

            \n
          1. Under Project template parameters<\/strong>, for RetrainSchedule<\/strong>, keep the default of cron(0 12 1 * ? *)<\/code>.<\/li>\n
          2. Choose Create project.
            \"\"<\/a>
            <\/strong><\/li>\n<\/ol>\n

            When you choose Create project<\/strong>, a CloudFormation stack is created in your account. This takes a few minutes. If you\u2019re interested in the components that are being deployed, navigate to the CloudFormation console. There you can find the stack that is being created.<\/p>\n

            When the page reloads, on the main project page you can find a summary of all resources created in the project. For now, we need to clone the sagemaker-drift-detection-build<\/code> repository.<\/p>\n

              \n
            1. On the Repository <\/strong>tab, choose clone repo\u2026<\/strong> and accept the default values in the dialog box.
              \"\"<\/a><\/li>\n<\/ol>\n

              This clones the repository to the Studio space for your user. The notebook build-pipeline.ipynb<\/a> is provided as an entry point for you to run through the solution and to help you understand how to use it.<\/p>\n

                \n
              1. Open the notebook.
                \"\"<\/a><\/li>\n
              2. Choose the Python 3 (Data Science) <\/strong>kernel for the notebook. <\/em><\/li>\n<\/ol>\n

                If a Studio instance isn\u2019t already running, an instance is provisioned. This can take a couple of minutes. By default, an ml.t3.medium instance is launched, which is enough to run our notebook.<\/p>\n

                  \n
                1. When the notebook is open, edit the second cell with the actual project name you chose:<\/li>\n<\/ol>\n
                  \n
                  project_name = \"drift-detection\"<\/code><\/pre>\n<\/p><\/div>\n
                    \n
                  1. Run the first couple of cells to initialize some variables we need later and to verify we have selected our project:<\/li>\n<\/ol>\n
                    \n
                    import sagemaker\nimport json\n\nsess = sagemaker.session.Session()\nregion_name = sess._region_name\nsm_client = sess.sagemaker_client\nproject_id = sm_client.describe_project(ProjectName=project_name)[\"ProjectId\"]\nprint(f\"Project: {project_name} ({project_id})\")<\/code><\/pre>\n<\/p><\/div>\n
                    Next, we must define the dataset that we\u2019re using. In this example, we use data from NYC Taxi and Limousine Commission (TLC)<\/a>.<\/p>\n
                      \n
                    1. Download the data from its public Amazon Simple Storage Service<\/a> (Amazon S3) location and upload to the artifact bucket provisioned by the project template:<\/li>\n<\/ol>\n
                      \n
                      from sagemaker.s3 import S3Downloader, S3Uploader\n\n# Download to the data folder, and upload to the pipeline input uri\ndownload_uri = \"s3:\/\/nyc-tlc\/trip data\/green_tripdata_2018-02.csv\"\nS3Downloader().download(download_uri, \"data\")\n\n# Uplaod the data to the input location\nartifact_bucket = f\"sagemaker-project-{project_id}-{region_name}\"\ninput_data_uri = f\"s3:\/\/{artifact_bucket}\/{project_id}\/input\"\nS3Uploader().upload(\"data\", input_data_uri)\n\nprint(\"Listing input files:\")\nfor s3_uri in S3Downloader.list(input_data_uri):\n    print(s3_uri.split(\"\/\")[-1])<\/code><\/pre>\n<\/p><\/div>\n
                      For your own custom template, you could also upload your data directly in the input_data_uri<\/code> location, because this is where the pipeline expects to find the training data.<\/p>\n
                      Run the training pipeline<\/h2>\n
                      To run the pipeline, you can continue running the cells in the notebook. You can also start a pipeline run through the Studio interface.<\/p>\n
                        \n
                      1. First, go back to the main project view page and choose the Pipelines<\/strong> tab.
                        \"\"<\/a><\/li>\n
                      2. Choose the pipeline drift-detection-pipeline<\/code>, which opens a tab containing a list of past runs.<\/li>\n<\/ol>\n
                        When you first get to this page, you can see a previous failed run of the pipeline. This was started when the project was initialized. It failed because at the time there was no data for the pipeline to use.<\/p>\n
                        You can now start a new pipeline with the data.<\/p>\n
                          \n
                        1. Choose Start an execution<\/strong>.<\/li>\n
                        2. For Name<\/strong>, enter First-Pipeline-execution<\/code>.<\/li>\n
                        3. Choose Start<\/strong>.
                          \"\"<\/a><\/li>\n<\/ol>\n
                          When the pipeline starts, it\u2019s added to the list of pipeline runs with a status of Executing<\/code>.
                          \"\"<\/a>
                          <\/em><\/p>\n
                            \n
                          1. Choose this pipeline run to open its details.<\/li>\n<\/ol>\n
                            Pipelines automatically constructs a graph showing the data dependency for each step in the pipeline. Based on this, you can see the order in which the steps are completed.<\/p>\n
                            \"\"<\/a><\/p>\n
                            If you wait for a few minutes and refresh the graph, you can notice that the BaselineJob<\/code> and TrainModel<\/code> steps run at the same time. This happened automatically. SageMaker understands that these steps are safe to run in parallel because there is no data dependency between them. You can explore this page by choosing the different steps and tabs on the page. Let\u2019s look at what the different steps are doing:<\/p>\n