{"id":294,"date":"2020-09-26T00:29:23","date_gmt":"2020-09-26T00:29:23","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/26\/active-learning-workflow-for-amazon-comprehend-custom-classification-models-part-1-2\/"},"modified":"2020-09-26T00:29:23","modified_gmt":"2020-09-26T00:29:23","slug":"active-learning-workflow-for-amazon-comprehend-custom-classification-models-part-1-2","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/26\/active-learning-workflow-for-amazon-comprehend-custom-classification-models-part-1-2\/","title":{"rendered":"Active learning workflow for Amazon Comprehend custom classification models \u2013 Part 1"},"content":{"rendered":"
\n

Amazon Comprehend<\/a>\u00a0\u00a0Custom Classification API enables you to easily build custom text classification models using your business-specific labels without learning ML. For example, your customer support organization can use Custom Classification to automatically categorize inbound requests by problem type based on how the customer has described the issue. \u00a0You can use custom classifiers to automatically label support emails with appropriate issue types, routing customer phone calls to the right agents, and categorizing social media posts into user segments.<\/p>\n

For custom classification, you start by creating a training job with a ground truth dataset comprising a collection of text and corresponding category labels. Upon completing the job, you have a classifier that can classify any new text into one or more named categories. When the custom classification model classifies a new unlabeled text document, it predicts what it has learned from the training data. Sometimes you may not have a training dataset with various language patterns, or once you deploy the model, you start seeing completely new data patterns. In these cases, the model may not be able to classify these new data patterns accurately. How can we ensure continuous model training to keep it up to date with new data and patterns?<\/p>\n

In this two part blog series, we discuss an architecture pattern that allows you to build an active learning workflow for Amazon Comprehend custom classification models. The first post will describe a workflow comprising real-time classification, feedback pipelines and human review workflows using Amazon Augmented AI<\/a> (Amazon A2I). The second post will cover the automated model building using the human reviewed data, selecting the best model, and automated deployment of an endpoint of the chosen model.<\/p>\n

Feedback loops play a pivotal role in keeping the models up to date. This feedback helps the models learn about their misclassifications and learn the right ones. This process of teaching the models continuously through feedback and deploying them is called active learning<\/em>.<\/p>\n

For every prediction Amazon Comprehend Custom Classification makes, it also gives a confidence score associated with its prediction. This architecture proposes that you set an acceptable threshold and only accept the predictions with a confidence score that exceeds the threshold. All the predictions that have a confidence score less than the desired threshold are flagged for human review. The human decides whether to accept the model\u2019s prediction or correct it.<\/p>\n

In some instances, the model may be confident about its predictions, but the classification might be wrong. In these scenarios, the end-user applications that receive the model predictions can request explicit feedback from its users on the prediction quality. A human moderator reviews this explicit feedback and reclassifies instances where the feedback was negative. This process of generating human-verified data and using it for model retraining helps keep the models up to date, reduce data drift, and achieve higher model accuracy.<\/p>\n

Feedback Workflow Architecture.<\/h2>\n

In this section, we discuss an architectural pattern for implementing an end-to-end active learning workflow for custom classification models in Amazon Comprehend using Amazon A2I. The active learning workflow comprises the following components:<\/p>\n

    \n
  1. Real-time classification<\/li>\n
  2. Feedback loops<\/li>\n
  3. Human classification<\/li>\n
  4. Model building<\/li>\n
  5. Model selection<\/li>\n
  6. Model deployment<\/li>\n<\/ol>\n

    The following diagram illustrates this architecture covering the first three components. In the following sections, we walk you through each step in the workflow.<\/p>\n

    \"Architecture<\/p>\n

    Real-time classification<\/h3>\n

    To use custom classification in Amazon Comprehend, you need to create a custom classification job that reads a ground truth dataset from an Amazon Simple Storage Service<\/a> (Amazon S3) bucket and builds a classification model. After the model builds successfully, you can create an endpoint that allows you to make real-time classifications of unlabeled text. This stage is represented by steps 1\u20133 in the preceding architecture:<\/p>\n

      \n
    1. The end-user application calls an API Gateway<\/a> endpoint with a text that needs to be classified.<\/li>\n
    2. The API Gateway endpoint then calls an AWS Lambda<\/a> function configured to call an Amazon Comprehend endpoint.<\/li>\n
    3. The Lambda function calls the Amazon Comprehend endpoint, which returns the unlabeled text classification and a confidence score.<\/li>\n<\/ol>\n

      Feedback collection<\/h3>\n

      When the endpoint returns the classification and the confidence score during the real-time classification, you can send instances with low-confidence scores to human review. This type of feedback is called implicit feedback<\/em>.<\/p>\n

        \n
      1. The Lambda function sends the implicit feedback to an Amazon Kinesis Data Firehose<\/a>.<\/li>\n<\/ol>\n

        The other type of feedback is called explicit feedback<\/em> and comes from the application\u2019s end-users that use the custom classification feature. This type of feedback comprises the instances of text where the user wasn\u2019t happy with the prediction. Explicit feedback can be sent either in real-time through an API or a batch process.<\/p>\n

          \n
        1. End-users of the application submit explicit real-time feedback through an API Gateway endpoint.<\/li>\n
        2. The Lambda function backing the API endpoint transforms the data into a standard feedback format and writes it to the Kinesis Data Firehose delivery stream.<\/li>\n
        3. End-users of the application can also submit explicit feedback as a batch file by uploading it to an S3 bucket.<\/li>\n
        4. A trigger configured on the S3 bucket triggers a Lambda function.<\/li>\n
        5. The Lambda function transforms the data into a standard feedback format and writes it to the delivery stream.<\/li>\n
        6. Both the implicit and explicit feedback data gets sent to a delivery stream in a standard format. All this data is buffered and written to an S3 bucket.<\/li>\n<\/ol>\n

          Human classification<\/h3>\n

          The human classification stage includes the following steps:<\/p>\n

            \n
          1. A trigger configured on the feedback bucket in Step 10 invokes a Lambda function.<\/li>\n
          2. The Lambda function creates Amazon A2I human review tasks for all the feedback data received.<\/li>\n
          3. Workers assigned to the classification jobs log in to the human review portal and either approve the classification by the model or classify the text with the right labels.<\/li>\n
          4. After the human review, all these instances are stored in an S3 bucket and used for retraining the models. Part 2 of this series covers the retraining workflow.<\/li>\n<\/ol>\n

            Solution overview<\/h2>\n

            The next few sections of the post go over how to set up this architecture in your AWS account. We classify news into four categories: World, Sports, Business, and Sci\/Tech, using the AG News dataset<\/a> for custom classification, and set up the implicit and explicit feedback loop. You need to complete two manual steps:<\/p>\n

              \n
            1. Create an Amazon Comprehend custom classifier and an endpoint.<\/li>\n
            2. Create an Amazon SageMaker<\/a> private workforce, worker task template, and human review workflow.<\/li>\n<\/ol>\n

              After this, you run the provided AWS CloudFormation<\/a> template to set up the rest of the architecture.<\/p>\n

              Prerequisites<\/h2>\n

              Before you get started, download the dataset and upload it to Amazon S3.<\/strong> This dataset comprises a collection of news articles and their corresponding category labels. We have created a training dataset called train.csv from the original dataset and made it available for download<\/a>.<\/p>\n

              The following screenshot shows a sample of the train.csv file.<\/p>\n

              \"CSV<\/p>\n

              After you download the train.csv file, upload it to an S3 bucket in your account for reference during training. For more information about uploading files, see How do I upload files and folders to an S3 bucket?<\/a><\/p>\n

              Creating a custom classifier and an endpoint<\/h2>\n

              To create your classifier for classifying news, complete the following steps:<\/p>\n

                \n
              1. On the Amazon Comprehend console<\/a>, choose Custom Classification<\/strong>.<\/li>\n
              2. Choose Train classifier<\/strong>.<\/li>\n
              3. For Name<\/strong>, enter news-classifier-demo<\/code>.<\/li>\n
              4. Select Using<\/strong> Multi-class mode<\/strong>.<\/li>\n
              5. For Training data S3 location<\/strong>, enter the path for train.csv in your S3 bucket, for example, s3:\/\/<your-bucketname><\/em>\/train.csv<\/code>.<\/li>\n
              6. For Output data S3 location<\/strong>, enter the S3 bucket path where you want the output, such as s3:\/\/<your-bucketname><\/em>\/<\/code>.<\/li>\n
              7. For IAM role<\/strong>, select Create an IAM role<\/strong>.<\/li>\n
              8. For Permissions to access<\/strong>, choose Input and output (if specified) S3 bucket<\/strong>.<\/li>\n
              9. For Name suffix<\/strong>, enter ComprehendCustom<\/code>.<\/li>\n<\/ol>\n

                \"Comprehend<\/p>\n

                  \n
                1. Scroll down and choose Train Classifier<\/strong> to start the training process.<\/li>\n<\/ol>\n

                  The training takes some time to complete. You can either wait to create an endpoint or come back to this step later after finishing the steps in the section Creating a private workforce, worker task template, and human review workflow<\/strong>.<\/p>\n

                  Creating a custom classifier real-time endpoint<\/h2>\n

                  To create your endpoint, complete the following steps:<\/p>\n

                    \n
                  1. On the Amazon Comprehend console, choose Custom Classification<\/strong>.<\/li>\n
                  2. From the Classifiers <\/strong>list, choose the name of the custom model for which you want to create the endpoint and select your model news-classifier-demo<\/code>.<\/li>\n
                  3. From the Actions<\/strong> drop-down menu, choose Create endpoint<\/strong>.<\/li>\n
                  4. For Endpoint name<\/strong>, enter classify-news-endpoint<\/code> and give it one inference unit.<\/li>\n
                  5. Choose Create endpoint<\/strong>\n<\/li>\n
                  6. Copy the endpoint ARN as shown in the following screenshot. You use it when running the CloudFormation template in a future step.<\/li>\n<\/ol>\n

                    \"Custom<\/p>\n

                    Creating a private workforce, worker task template, and human review workflow.<\/h2>\n

                    This section walks you through creating a private workforce in Amazon SageMaker, a worker task template, and your human review workflow.<\/p>\n

                    Creating a labeling workforce<\/strong>
                    \n<\/h3>\n
                      \n
                    1. For this post, you will create a private work team and add only one user (you) to it. For instructions, see Create a Private Workforce (Amazon SageMaker Console)<\/a>.<\/li>\n
                    2. Once the user accepts the invitation, you will have to add him to the workforce. For instructions, see the\u00a0Add a Worker to a Work Team<\/strong> section the Manage a Workforce (Amazon SageMaker Console)<\/a>\n<\/li>\n<\/ol>\n

                      Creating a worker task template<\/strong><\/h3>\n

                      To create a worker task template, complete the following steps:<\/p>\n

                        \n
                      1. On the Amazon A2I console,<\/a> choose Worker task templates<\/strong>.<\/li>\n
                      2. Choose to Create a <\/strong>template<\/strong>.<\/li>\n
                      3. For Template name<\/strong>, enter custom-classification-template<\/code>.<\/li>\n
                      4. For Template type<\/strong>, choose Custom,<\/strong>\n<\/li>\n
                      5. In the Template editor<\/strong>, enter the following GitHub UI template code<\/a>.<\/li>\n
                      6. Choose Create<\/strong>.<\/li>\n<\/ol>\n

                        \"Worker<\/p>\n

                        Creating a human review workflow<\/strong><\/h3>\n

                        To create your human review workflow, complete the following steps:<\/p>\n

                          \n
                        1. On the Amazon A2I console<\/a>, choose Human review workflows.<\/strong>\n<\/li>\n
                        2. \nChoose Create human review workflow<\/strong>.<\/li>\n
                        3. For Name, <\/strong>enter classify-workflow<\/code>.<\/strong>\n<\/li>\n
                        4. Specify an S3 bucket<\/strong> to store output: s3:\/\/<your bucketname><\/em>\/<\/code>.<\/li>\n<\/ol>\n

                          Use the same bucket where you downloaded your train.csv in the prerequisite step.<\/p>\n

                            \n
                          1. For IAM role<\/strong>, select Create a new role<\/strong>.<\/li>\n
                          2. For Task type<\/strong>, choose Custom.<\/strong>\n<\/li>\n
                          3. Under Worker task template creation<\/strong>, select the custom classification template you created.<\/li>\n
                          4. For Task description<\/strong>, enter Read the instructions and review the document<\/code>.<\/em>\n<\/li>\n
                          5. Under Workers,<\/strong> select Private<\/strong>.<\/li>\n
                          6. Use the drop-down list to choose the private team that you created.<\/li>\n
                          7. Choose Create<\/strong>.<\/li>\n
                          8. Copy the workflow ARN (see the following screenshot) to use when initializing the CloudFormation parameters.<\/li>\n<\/ol>\n

                            \"Human<\/p>\n

                            Deploying the CloudFormation template to set up active learning feedback<\/h2>\n

                            Now that you have completed the manual steps, you can run the CloudFormation template to set up this architecture\u2019s building blocks, including the real-time classification, feedback collection, and the human classification.<\/p>\n

                            Before deploying the CloudFormation template, make sure you have the following to pass as parameters:<\/p>\n