{"id":318,"date":"2020-09-30T21:37:42","date_gmt":"2020-09-30T21:37:42","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/30\/using-amazon-rekognition-custom-labels-and-amazon-a2i-for-detecting-pizza-slices-and-augmenting-predictions\/"},"modified":"2020-09-30T21:37:42","modified_gmt":"2020-09-30T21:37:42","slug":"using-amazon-rekognition-custom-labels-and-amazon-a2i-for-detecting-pizza-slices-and-augmenting-predictions","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/30\/using-amazon-rekognition-custom-labels-and-amazon-a2i-for-detecting-pizza-slices-and-augmenting-predictions\/","title":{"rendered":"Using Amazon Rekognition Custom Labels and\u00a0Amazon A2I for detecting pizza slices and augmenting predictions"},"content":{"rendered":"
\n

Customers need machine learning (ML) models to detect objects that are interesting for their business. In most cases doing so is hard as these models needs thousands of labelled images and deep learning expertise. \u00a0Generating this data can take months to gather, and can require large teams of labelers to prepare it for use. In addition, setting up a workflow for auditing or reviewing model predictions to validate adherence to your requirements can further add to the overall complexity.<\/p>\n

With Amazon Rekognition Custom Labels<\/a>, which is built on the existing capabilities of Amazon Rekognition<\/a>, you can identify the objects and scenes in images that are specific to your business needs. For example, you can find your logo in social media posts, identify your products on store shelves, classify machine parts in an assembly line, distinguish healthy and infected plants, or detect animated characters in videos.<\/p>\n

But what if the custom label model you trained can\u2019t recognize images with a high level of confidence, or you need your team of experts to validate the results from your model during the testing phase or review the results in production? You can easily send predictions from Amazon Rekognition Custom Labels to Amazon Augmented AI<\/a> (Amazon A2I). Amazon A2I makes it easy to integrate a human review into your ML workflow. This allows you to automatically have humans step into your ML pipeline to review results below a confidence threshold, set up review and auditing workflows, and augment the prediction results to improve model accuracy.<\/p>\n

In this post, we show you to how to build a custom object detection model trained to detect pepperoni slices in a pizza using Amazon Rekognition Custom Labels with a dataset labeled using Amazon SageMaker Ground Truth<\/a>. We then show how to create your own private workforce and set up an Amazon A2I workflow definition to conditionally trigger human loops for review and augmenting tasks. You can use the annotations created by Amazon A2I for model retraining.<\/p>\n

We walk you through the following steps using the accompanying Amazon SageMaker Jupyter Notebook<\/a>:<\/p>\n

    \n
  1. Complete prerequisites.<\/li>\n
  2. Create an Amazon Rekognition custom model.<\/li>\n
  3. Set up an Amazon A2I workflow and send predictions to an Amazon A2I human loop.<\/li>\n
  4. Evaluate results and retrain the model.<\/li>\n<\/ol>\n

    Prerequisites<\/h2>\n

    Before getting started, set up your Amazon SageMaker notebook. Follow the steps in the accompanying Amazon SageMaker Jupyter Notebook<\/a> to create your human workforce and download the datasets.<\/p>\n

      \n
    1. \nCreate a notebook instance<\/a> in Amazon SageMaker<\/a>.<\/li>\n<\/ol>\n

      Make sure your Amazon SageMaker notebook has the necessary AWS Identity and Access Management<\/a> (IAM) roles and permissions mentioned in the prerequisite section of the notebook<\/a>.<\/p>\n

        \n
      1. When the notebook is active, choose Open Jupyter<\/strong>.<\/li>\n
      2. On the Jupyter dashboard, choose New<\/strong>, and choose Terminal<\/strong>.<\/li>\n
      3. In the terminal, enter the following code:<\/li>\n<\/ol>\n
        \n
        cd SageMaker\r\ngit clone https:\/\/github.com\/aws-samples\/amazon-a2i-sample-jupyter-notebooks\r\n<\/code><\/pre>\n<\/div>\n
          \n
        1. Open the notebook by choosing Amazon-A2I-Rekognition-Custom-Label.ipynb<\/strong> in the root folder.<\/li>\n<\/ol>\n
          For this post, you create a private work team and add one user (you) to it.<\/p>\n
            \n
          1. Create your human workforce by following the appropriate section of the notebook.<\/li>\n<\/ol>\n
            Alternatively, you can create your private workforce using Amazon Cognito<\/a>. For instructions, see Create an Amazon Cognito Workforce Using the Labeling Workforces Page<\/a>.<\/p>\n
              \n
            1. After you create the private workforce, find the workforce ARN<\/a> and enter the ARN in step 4 of the notebook.<\/li>\n<\/ol>\n
              \u00a0<\/strong>The dataset is composed of 12 images for training that contain pepperoni pizza in the data\/images<\/code> folder and 3 images for testing in the data\/testimages<\/code> folder. We sourced our images from pexels.com<\/a>. We labeled this dataset for this post using Ground Truth custom labeling workflows<\/a>. A Ground Truth labeled manifest template is available in data\/manifest<\/a>.<\/p>\n
                \n
              1. Download the dataset.<\/li>\n
              2. Run the notebook steps Download the Amazon SageMaker GroundTruth object detection manifest to Amazon S3<\/strong> to process and upload the manifest in your Amazon Simple Storage Service<\/a> (Amazon S3) bucket.<\/li>\n<\/ol>\n
                Creating an Amazon Rekognition custom model<\/h2>\n
                To create our custom model, we follow these steps:<\/p>\n
                  \n
                1. Create a project in Amazon Rekognition Custom Labels.<\/li>\n
                2. Create a dataset with images containing one or more pizzas and label them by applying bounding boxes.<\/li>\n<\/ol>\n
                  Because we already have a dataset labeled using Ground Truth, we just point to that labeled dataset in this step. Alternatively, you can label the images using the user interface provided by Amazon Rekognition Custom Labels.<\/p>\n
                    \n
                  1. Train the model.<\/li>\n
                  2. Evaluate the model\u2019s performance.<\/li>\n
                  3. Test the deployed model using a sample image.<\/li>\n<\/ol>\n
                    Creating a project<\/h3>\n
                    In this step, we create a project to detect peperoni pizza slices. For instructions on creating a project, see Creating an Amazon Rekognition Custom Labels Project<\/a>.<\/p>\n
                      \n
                    1. On the Amazon Rekognition console, choose Custom Labels<\/strong>.<\/li>\n
                    2. Choose Get started<\/strong>.<\/li>\n
                    3. For Project name<\/strong>, enter a2i-rekog-pepperoni-pizza<\/code>.<\/li>\n
                    4. Choose Create project<\/strong>.<\/li>\n<\/ol>\n
                      \"\"<\/p>\n
                      Creating a dataset<\/h3>\n
                      To create your dataset, complete the following steps:<\/p>\n
                        \n
                      1. On the Amazon Rekognition Custom Labels console, choose Datasets<\/strong>.<\/li>\n
                      2. Choose Create dataset<\/strong>.<\/li>\n
                      3. For Dataset name<\/strong>, enter rekog-a2i-pizza-dataset<\/code>.<\/li>\n
                      4. Select Import images labeled by Amazon SageMaker Ground Truth<\/strong>.<\/li>\n
                      5. For .manifest file location<\/strong>, enter the S3 bucket location of your .manifest file.<\/li>\n<\/ol>\n
                        \"\"<\/p>\n
                          \n
                        1. Choose\u00a0Submit<\/strong>.<\/li>\n<\/ol>\n
                          You should get a prompt to provide the S3 bucket policy when you provide the S3 path. For more information about these steps, see SageMaker Ground Truth<\/a>.<\/p>\n
                          After you create the dataset, you should see the images with the bounding boxes and labels, as in the following screenshot.<\/p>\n
                          \"\"<\/p>\n
                          Make<\/strong> sure to upload the images for our dataset (that you downloaded in the Prerequisites<\/strong> section) to the console S3 bucket for Amazon Rekognition Custom Labels.<\/p>\n
                          Training an Amazon Rekognition custom model<\/h3>\n
                          You\u2019re now ready to train your model.<\/p>\n
                            \n
                          1. On the Amazon Rekognition console, choose Train model<\/strong>.<\/li>\n
                          2. For Choose project<\/strong>, choose your newly created project.<\/li>\n
                          3. For Choose training dataset<\/strong>, choose your newly created dataset.<\/li>\n
                          4. Select Split training dataset<\/strong>.<\/li>\n
                          5. Choose Train<\/strong>.<\/li>\n<\/ol>\n
                            \"\"<\/p>\n
                            The training takes approximately 45 minutes to complete.<\/p>\n
                            Checking training status<\/h4>\n
                            Run the following notebook cell to get information about the project you created using the describe-projects<\/a> API:<\/p>\n
                            \n
                            !aws rekognition describe-projects<\/code><\/pre>\n<\/div>\n
                            Use the accompanying Amazon SageMaker Jupyter notebook<\/a> to follow the steps in this post.<\/p>\n
                            To get the project ARN using the describe-project versions<\/a> API, run the following cell:<\/p>\n
                            \n
                            #Replace the project-arn below with the project-arn of your project from the describe-projects output above\r\n!aws rekognition describe-project-versions \u2014project-arn \"<project-arn><\/span>\"\r\n<\/code><\/pre>\n<\/div>\n
                            Enter the ARN in <project-version-arn-of-your-model><\/span><\/em> in the following code and run this cell to start running of the version of a model using the start-project-version<\/a> API:<\/p>\n
                            \n
                            # Copy\/paste the ProjectVersionArn for your model from the describe-project-versions cell output above to the --project-version-arn parameter here\r\n!aws rekognition start-project-version \r\n--project-version-arn <project-version-arn-of-your-model><\/span> \r\n--min-inference-units 1 \r\n--region us-east-1\r\n<\/code><\/pre>\n<\/div>\n
                            Evaluating performance<\/h3>\n
                            When training is complete, you can evaluate the performance of the model. To help you, Amazon Rekognition Custom Labels provides summary metrics and evaluation metrics for each label. For information about the available metrics, see Metrics for Evaluating Your Model<\/a>. To improve your model using metrics, see Improving an Amazon Rekognition Custom Labels Model<\/a>.<\/p>\n
                            The following screenshot shows our evaluation results and label performance.<\/p>\n
                            \"\"<\/p>\n
                            Custom Labels determines the assumed threshold for each label based on maximum precision and recall achieved. Your model defaults to returning predictions above that threshold. You can reset this when you start your model. For information about adjusting your assumed threshold (such as looking at predictions either below or above your assumed threshold) and optimizing the model for your use case, see Training a custom single class object detection model with Amazon Rekognition Custom Labels<\/a>.<\/p>\n
                            For a demonstration of the Amazon Rekognition Custom Labels solution, see Custom Labels Demonstration<\/a>. The demo shows you how you can analyze an image from your local computer.<\/p>\n
                            Testing the deployed model<\/h3>\n
                            Run following notebook cell to load the test image. Use the accompanying Amazon SageMaker Jupyter Notebook<\/a> to follow the steps in this post.<\/p>\n
                            \n
                            from PIL import Image, ImageDraw, ExifTags, ImageColor, ImageFont\r\nimage=Image.open('.\/data\/images\/pexels_brett_jordan_825661__1_.jpg')\r\ndisplay(image)\r\n<\/code><\/pre>\n<\/div>\n
                            Enter the project version model ARN from previous notebook step aws rekognition start-project-version<\/code> and run the following cell to analyze the response from the detect_custom_labels<\/a> API:<\/p>\n
                            \n
                            model_arn = '<project-version-arn-of-your-model><\/span>'\r\nmin_confidence=40\r\n#Call DetectCustomLabels\r\nresponse = client.detect_custom_labels(Image={'S3Object': {'Bucket': BUCKET, 'Name': PREFIX + test_photo}},\r\nMinConfidence=min_confidence,\r\nProjectVersionArn=model_arn)\r\n<\/code><\/pre>\n<\/div>\n
                            Run the next cell in the notebook to display results (see the following screenshot).<\/p>\n
                            \"\"<\/p>\n
                            The model detected the pepperoni pizza slices in our test image and drew bounding boxes. We can use Amazon A2I to send the prediction results from our model to a human loop consisting of our private workforce to review and audit the results.<\/p>\n
                            Setting up an Amazon A2I human loop<\/h2>\n
                            In this section, you set up a human review loop for low-confidence detection in Amazon A2I. It includes the following steps:<\/p>\n
                              \n
                            1. Create a human task UI.<\/li>\n
                            2. Create a worker task template and workflowflow definition.<\/li>\n
                            3. Send predictions to Amazon A2I human loops.<\/li>\n
                            4. Check the human loop status.<\/li>\n<\/ol>\n
                              Use the accompanying Amazon SageMaker Jupyter notebook<\/a> to follow the steps in this post.<\/p>\n
                              Creating a human task UI<\/h3>\n
                              Create a human task UI resource with a UI template in liquid HTML. This template is used whenever a human loop is required.<\/p>\n
                              For this post, we take an object detection UI<\/a> and fill in the object categories in the labels<\/code> variable in the template. Run the following template to create the human task AI for object detection:<\/p>\n
                              \n
                              def create_task_ui():\r\n    '''\r\n    Creates a Human Task UI resource.\r\n\r\n    Returns:\r\n    struct: HumanTaskUiArn\r\n    '''\r\n    response = sagemaker_client.create_human_task_ui(\r\n        HumanTaskUiName=taskUIName,\r\n        UiTemplate={'Content': template})\r\n    return response\r\n# Create task UI\r\nhumanTaskUiResponse = create_task_ui()\r\nhumanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\r\nprint(humanTaskUiArn)\r\n<\/code><\/pre>\n<\/div>\n
                              For over 70 pre-built UIs, see the Amazon Augmented AI Sample Task UIs<\/a> GitHub repo.<\/p>\n
                              Creating a worker task template and workflow definition<\/h3>\n
                              Workflow definitions allow you to specify the following:<\/p>\n