{"id":1167,"date":"2021-11-09T08:34:44","date_gmt":"2021-11-09T08:34:44","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/09\/automate-detection-of-broken-utility-poles-using-the-amazon-rekognition-custom-labels-sdk\/"},"modified":"2021-11-09T08:34:44","modified_gmt":"2021-11-09T08:34:44","slug":"automate-detection-of-broken-utility-poles-using-the-amazon-rekognition-custom-labels-sdk","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/09\/automate-detection-of-broken-utility-poles-using-the-amazon-rekognition-custom-labels-sdk\/","title":{"rendered":"Automate detection of broken utility poles using the Amazon Rekognition Custom Labels SDK"},"content":{"rendered":"
\n

Domestic infrastructure issues are a pain for everyone involved. Not only does it negatively affect customer satisfaction, it also has a cascading effect on businesses and their bottom line in terms of financials. Electric utility poles, for example, are an example of a cumbersome infrastructure issue to resolve. Normally, the standard wooden distribution pole is expected to last roughly 50 years. However, occasionally these poles need to be replaced earlier due to unexpected incidents such as accidents, severe weather disasters, or even power line relocation. The industry standard right now is to use drone or street cameras to generate images of these broken poles. The poles in the images are then manually inspected to ensure they\u2019re in good condition and don\u2019t require repair or replacement. As you can imagine, the process of determining whether or not these poles need replacement is a time-consuming and manual task that is susceptible to human error or neglect.<\/p>\n

To address this, we propose a solution using Amazon Rekognition Custom Labels<\/a>. You can feed images of utility poles, taken from street cameras or from drones, into a machine computer vision model trained on Amazon Rekognition Custom Labels to automatically detect whether a utility pole is in good condition or damaged.<\/p>\n

Amazon Rekognition<\/a> is a computer vision service within the AWS AI\/ML stack. It allows for the automation of image and videos analysis. With Amazon Rekognition, you can identify objects, people, text, scenes, inappropriate content, and activities in images and videos.<\/p>\n

We use Amazon Rekognition Custom Labels for our solution, which enables us to create custom machine learning (ML) models to analyze images. With Amazon Rekognition Custom Labels, you can train a robust, deployable model with a few images as opposed to thousands of images.<\/p>\n

For our use case, we use images of electric poles. The following is an example of a normal pole.<\/p>\n

\"\"<\/a><\/p>\n

The following image is an example of a damaged pole.<\/p>\n

\"\"<\/a><\/p>\n

If your images are already labeled, Amazon Rekognition Custom Labels can begin training in just a few clicks on the Amazon Rekognition console. If not, you can label them directly within the Amazon Rekognition Custom Labels labeling user interface, or use another service such as Amazon SageMaker Ground Truth<\/a> to label them. After you train your image set with Amazon Rekognition Custom Labels, it can produce a custom image computer vision model for you in just a few hours. After this custom model is trained, you can use it as an endpoint to make inferences on new images.<\/p>\n

In this post, we use the Amazon Rekognition Custom Labels API and the AWS SDK to show how easily you can integrate this technology into your applications.<\/p>\n

Prepare dataset bucket with images<\/h2>\n

As with all ML models, we begin with some data\u2014for this post, images of broken and not broken utility poles. This dataset is fully labeled and stored in Amazon Simple Storage Service<\/a> (Amazon S3). The location of this dataset is fed into the SDK to train the model.<\/p>\n

Train the model<\/h2>\n

Now that our labeled data is in our S3 bucket, let\u2019s create the model for our data.<\/p>\n

    \n
  1. Import the necessary libraries:\n <\/li>\n
  2. Because Amazon Rekognition Custom Labels requires you to create a project for a given use case, we use the following function to create a project:<\/li>\n<\/ol>\n
    \n
    def create_project(project_name):\n\n    client=boto3.client('rekognition')\n\n    #Create a project\n    print('Creating project:' + project_name)\n    response=client.create_project(ProjectName=project_name)\n    arn = response['ProjectArn']\n    print('project ARN: ' + response['ProjectArn'])\n    return arn<\/code><\/pre>\n<\/p><\/div>\n
    The function returns the project ARN, which you should write down or store in a variable because you use it later to train the model.<\/p>\n
    We next define a method to train a model using Amazon Rekognition Custom Labels. This method requires the project_arn<\/code> as an input (the project ARN variable that you saved earlier), a unique version_name<\/code> for this version of the model, out_config<\/code> to inform where to store training results, as well as locations of the training and test data manifest files.<\/p>\n
      \n
    1. Run the following train_model()<\/code> method, and make a note of the project version ARN to use later:<\/li>\n<\/ol>\n
      \n
      # Train model helper method\nimport json\n\ndef train_model(project_arn, version_name, output_config, training_dataset,testing_dataset):\n\n    client=boto3.client('rekognition')\n    \n    print('Starting training of: ' + version_name)\n      \n    try:\n        response=client.create_project_version(ProjectArn=project_arn, \n            VersionName=version_name,\n            OutputConfig=output_config,\n            TrainingData=training_dataset,\n            TestingData=testing_dataset)\n\n        # Wait for the project version training to complete\n        project_version_training_completed_waiter = client.get_waiter('project_version_training_completed')\n        project_version_training_completed_waiter.wait(ProjectArn=project_arn,\n        VersionNames=[version_name])\n    \n        #Get the completion status\n        describe_response=client.describe_project_versions(ProjectArn=project_arn,\n            VersionNames=[version_name])\n        for model in describe_response['ProjectVersionDescriptions']:\n            print('Project Version ARN: ' + model['ProjectVersionArn'])\n            print(\"Status: \" + model['Status'])\n            print(\"Message: \" + model['StatusMessage']) \n    except Exception as e:\n        print(e)\n\n    print('Done...')<\/code><\/pre>\n<\/p><\/div>\n
        \n
      1. Next, call the train_model()<\/code> method, as shown in the following code:<\/li>\n<\/ol>\n
        \n
        project_arn='arn:aws:rekognition:us-east-1:xxxxxxxxxxxx:project\/project_utility_pole_blog\/yyyyyyyyyyy'\n\nversion_name='v1'\n\noutput_config = json.loads('{\"S3Bucket\":\"my-bucket\", \"S3KeyPrefix\":\"blog\/REK-CL\/utilitypoles\/\"}')\n\ntraining_dataset= json.loads('{\"Assets\": [{ \"GroundTruthManifest\": { \"S3Object\": { \"Bucket\": \"my-bucket\", \"Name\": \"datasets\/cables-ds\/manifests\/output\/output.manifest\" } } } ] }')\n\ntesting_dataset= json.loads('{\"AutoCreate\":true}')\n\ntrain_model(project_arn, version_name, output_config, training_dataset, testing_dataset)<\/code><\/pre>\n<\/p><\/div>\n
        We set \"AutoCreate\":true<\/code> for testing_dataset<\/code> because we\u2019re using Amazon Rekognition Custom Labels to split the training data randomly into an 80\/20 split. Alternatively, you can specify the testing_dataset<\/code> as a manifest file, just as is done for the training_dataset<\/code> in the preceding code, if you have a separate test dataset.<\/p>\n
        Training can take a couple of hours to complete.<\/p>\n
          \n
        1. You can get the current status by calling DescribeProjectVersions<\/code> and, when it\u2019s complete, calling DescribeProjectVersions<\/code> to get the training results and evaluate the model:<\/li>\n<\/ol>\n
          \n
          def describe_model(project_arn, version_name):\n\n    client=boto3.client('rekognition')\n    \n    response=client.describe_project_versions(ProjectArn=project_arn,\n        VersionNames=[version_name])\n\n    for model in response['ProjectVersionDescriptions']:\n        print(json.dumps(model,indent=4,default=str))<\/code><\/pre>\n<\/p><\/div>\n
          Model results and enhancement<\/h2>\n
          We train two models in this post. The first model takes 80 images of good and broken utility poles, which are equally split between the good and broken. These images are fed into Amazon Rekognition Custom Labels and the model metrics are evaluated.<\/p>\n
          For the second model, instead of feeding the raw images as they are, we do some data augmentation on these images, which is common in computer vision problems. Amazon Rekognition Custom Labels doesn\u2019t do data augmentation by itself because it doesn\u2019t know your images too well. Therefore, we recommend explicitly doing image augmentation for scenarios where you want to further improve your model metrics.<\/p>\n
          We then compare how the model metrics such as accuracy and AUC score compare for the original model and the enhanced model.<\/p>\n
          Image augmentation is accomplished using the following code:<\/p>\n
          \n
          import random\nimport numpy as np\nfrom scipy import ndarray\nimport skimage as sk\nfrom skimage import transform\nfrom skimage import util\n\ndef random_rotation(image_array: ndarray):\n    # pick a random degree of rotation between 25% on the left and 25% on the right\n    random_degree = random.uniform(-25, 25)\n    return sk.transform.rotate(image_array, random_degree, preserve_range=True)\n\ndef horizontal_flip(image_array: ndarray):\n    # horizontal flip doesn't need skimage, it's easy as flipping the image array of pixels !\n    return image_array[:, ::-1]<\/code><\/pre>\n<\/p><\/div>\n
          The following is our original image.<\/p>\n
          \"\"<\/a><\/p>\n
          Random rotation of that image produces the following rotated image.<\/p>\n
          \"\"<\/a><\/p>\n
          Our original data of 80 images is randomly split into 60 images of training data and 20 images of test data. The original model is built using Amazon Rekognition Custom Labels on 60 images of the training data.<\/p>\n
          For building our enhanced model, we augmented the 60 training data images by running them through the preceding code, which involves rotation and horizontal flip. That increases the training data size from 60 to 180. We build a new model using this dataset.<\/p>\n
          After we build these two models, we run test data these models to see how the results compare. We use the following code to obtain the results:<\/p>\n
          \n
          # Helper method that makes a inference on a Custom Labels trained model for a binary classification problem\nimport boto3\nimport io\n\nfrom botocore.exceptions import ClientError\n\ndef getLabelAndConfidenceBinary(bucket, image, model):\n    \"\"\"\n    :param bucket: The name of the S3 bucket that contains the image that you want to analyze.\n    :param image: The name of the image that you want to analyze.\n    :param model: The ARN of the Amazon Rekognition Custom Labels model that you want to use.\n    \"\"\"\n    \n    rek_client=boto3.client('rekognition')\n    s3_connection = boto3.resource('s3')\n    \n    # set minimum confidence to 0 as we are interested in results for all condidence levels\n    min_confidence = 0\n    \n    try:\n        #Get image from S3 bucket.\n        s3_object = s3_connection.Object(bucket, image)\n        s3_response = s3_object.get()\n\n        #Call DetectCustomLabels \n        response = rek_client.detect_custom_labels(Image={'S3Object': {'Bucket': bucket, 'Name': image}},\n            MinConfidence=min_confidence,\n            ProjectVersionArn=model)\n    \n    except ClientError as err:\n        print(\"Exception in get_custom_labels()\")\n        raise\n        \n    if response['CustomLabels'][0][\"Confidence\"] >= response['CustomLabels'][1][\"Confidence\"]:\n        return response['CustomLabels'][0][\"Name\"], response['CustomLabels'][0][\"Confidence\"]\n    else:\n        return response['CustomLabels'][1][\"Name\"], response['CustomLabels'][1][\"Confidence\"]\n\n\n# Helper method that iterates through an S3 bucket and retrieves image labels\ndef getImageNames(bucket, prefix):\n    import boto3\n    client = boto3.client('s3')\n    paginator = client.get_paginator('list_objects_v2')\n    result = paginator.paginate(Bucket=bucket, Prefix=prefix)\n    images = []\n    for page in result:\n        if \"Contents\" in page:\n            for key in page[ \"Contents\" ]:\n                keyString = key[ \"Key\" ]\n                name = keyString.split(\"\/\")[-1]\n                iclass = name.split(\"-\")[0]\n                if len(iclass)>0:\n                    images.append([keyString, iclass])\n    return images\n\n# This code loops through all images and creates a y_true list objects based on image labels\nimage_folder = \"blog\/REK-CL\/utilitypoles\/new_test_images\/\"\ny_true = []\nnames = []\ntest_images = getImageNames('my-bucket', image_folder)\nfor image in test_images:\n    iclass = image[1]\n    name = image[0]\n    names.append(name)\n    if iclass==\"bad\":\n        y_true.append(1)\n    else:\n        y_true.append(0)\n\n# The helper method below makes predictions on a deploy Custom Labels model and returns predictions\n# Custom Labels associates confidence level to the label it predicts\n# In the code below, we are turning that to a [0 to 1] probability scale, where 1 indicates a broken pole \ndef getPredictions(model, bucket, images):\n    y_pred = []\n    y_prob = []\n    for image in images:\n        labelconf = getLabelAndConfidenceBinary(bucket, image[0], model)\n        if labelconf[0]==\"broken\":\n            y_pred.append(1)\n            prob = labelconf[1]\/100.0\n            y_prob.append(prob)\n        if labelconf[0]==\"good\":\n            y_pred.append(0)\n            prob = 1.0 - labelconf[1]\/100.0\n            y_prob.append(prob)\n        if labelconf[0]==\"\":\n            raise Exception(\"Invalid label\")\n    return y_pred, y_prob\n\nbucket = \"my-bucket\"\n# Assign Project Version ARN returned by train_model() below for both the models\noriginal_model = \"arn:aws:rekognition:us-east-1:xxxxxxxxxxxx:project\/upole-new-aug14\/version\/upole-new-aug14.2021-08-14T17.20.06\/1628976006624\"\nenhanced_model = \"arn:aws:rekognition:us-east-1: xxxxxxxxxxxx:project\/upole-new-aug14\/version\/upole-new-aug14.2021-08-14T18.36.45\/1628980605880\"\ny_pred_original, y_prob_original = getPredictions(original_model, bucket, test_images)\ny_pred_enhanced, y_prob_enhanced = getPredictions(enhanced_model, bucket, test_images)\n\n\nfrom sklearn.metrics import accuracy_score\nprint(\"Original model accuracy = \", round(accuracy_score(y_true, y_pred_original), 2))\nprint(\"Enhanced model accuracy = \", round(accuracy_score(y_true, y_pred_enhanced), 2))\n\nOriginal model accuracy =  0.79\nEnhanced model accuracy =  0.89\n\nimport numpy as np\nfrom sklearn import metrics\ndef calculateAUC(y_true, y_prob, pos_label):\n    fpr, tpr, thresholds = metrics.roc_curve(y_true, np.array(y_prob), pos_label=pos_label)\n    return metrics.auc(fpr, tpr)\n\nprint(\"Original model AUC = \", round(calculateAUC(y_true, y_prob_original, 1),2))\nprint(\"Enhanced model AUC = \", round(calculateAUC(y_true, y_prob_enhanced, 1),2))\n\nOriginal model AUC =  0.92\nEnhanced model AUC =  0.96<\/code><\/pre>\n<\/p><\/div>\n
          Performance review<\/h2>\n
          As we can observe from the model results, doing image augmentation has helped improve model accuracy significantly, from 0.79 to 0.89, and model AUC from 0.92 to 0.96.<\/p>\n
          Although the original model gave good results as is, doing image augmentation has further improved the results. You don\u2019t necessarily need to augment your images all the time, but you can employ this technique to see if it can further improve your model.<\/p>\n
          Clean up<\/h2>\n
          After you finish using this solution, it\u2019s important that you stop your model to stop accruing charges. To delete your project, simply run the following function:<\/p>\n
          \n
          def delete_project(project_arn):\n\n\tclient = boto3.client(\u2018rekognition\u2019)\n\t\n\tprint(\u2018Deleting project: \u2019 + project_arn)\n\tresponse = client.delete_project(ProjectArn=project_arn)\n\tprint(\u2018Status: \u2019 + response[\u2018Status\u2019])\n\tprint(\u2018Done\u2026\u2019)<\/code><\/pre>\n<\/p><\/div>\n
          Conclusion<\/h2>\n
          In this post, we successfully trained, evaluated, and inferred on a custom ML model for detecting broken and damaged utility poles. This once time-consuming and manual task was normally very susceptible to human error or neglect, but by using Amazon Rekognition Custom Labels, we were able to make this process quicker while maintaining accuracy.<\/p>\n
          For more information about using custom labels, see What Is Amazon Rekognition Custom Labels?<\/a><\/p>\n
          \n
          About the Authors:<\/strong><\/h3>\n
          \"\"<\/a>Winston Nwanne<\/strong> is an AWS Solutions Architect working with public sector partners. He specializes in AI\/ML and has helped customers and partners expand their capabilities within the AWS Cloud. Apart from supporting customers, he likes to read, make YouTube videos about financial literacy, play basketball, and spend time with his family.<\/p>\n
          \"\"<\/a>Raju Penmatcha<\/strong> is a Senior AI\/ML Specialist Solutions Architect at AWS. He works with education, government, and nonprofit customers on machine learning and artificial intelligence related projects, helping them build solutions using AWS. When not helping customers, he likes traveling to new places.<\/p>\n
                 \n      <\/div>\n","protected":false},"excerpt":{"rendered":"
          https:\/\/aws.amazon.com\/blogs\/machine-learning\/automate-detection-of-broken-utility-poles-using-the-amazon-rekognition-custom-labels-sdk\/<\/p>\n","protected":false},"author":0,"featured_media":1168,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1167"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1167"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1167\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1168"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1167"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1167"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1167"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}