{"id":1355,"date":"2021-12-14T00:02:09","date_gmt":"2021-12-14T00:02:09","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/12\/14\/amazon-lookout-for-vision-now-supports-visual-inspection-of-product-defects-at-the-edge\/"},"modified":"2021-12-14T00:02:09","modified_gmt":"2021-12-14T00:02:09","slug":"amazon-lookout-for-vision-now-supports-visual-inspection-of-product-defects-at-the-edge","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/12\/14\/amazon-lookout-for-vision-now-supports-visual-inspection-of-product-defects-at-the-edge\/","title":{"rendered":"Amazon Lookout for Vision now supports visual inspection of product defects at the edge"},"content":{"rendered":"
\n

Discrete and continuous manufacturing lines generate a high volume of products at low latency, ranging from milliseconds to a few seconds. To identify defects at the same throughput of production, camera streams of images must be processed at low latency. Additionally, factories may have low network bandwidth or intermittent cloud connectivity. In such scenarios, you may need to run the defect detection system on your on-premises compute infrastructure, and upload the processed results for further development and monitoring purposes to the AWS Cloud. This hybrid approach with both local edge hardware and the cloud can address the low latency requirements and help reduce storage and network transfer costs to the cloud. This may also fulfill your data privacy and other regulatory requirements.<\/p>\n

In this post, we show you how to detect defective parts using Amazon Lookout for Vision<\/a> machine learning (ML) models running on your on-premises edge appliance.<\/p>\n

Lookout for Vision is an ML service that helps spot product defects using computer vision to automate the quality inspection process in your manufacturing lines, with no ML expertise required. The fully managed service enables you to build, train, optimize, and deploy the models in the AWS Cloud or edge. You can use the cloud APIs<\/a> or deploy Amazon Lookout for Vision models on any NVIDIA Jetson edge appliance or x86 compute platform running Linux with an NVIDIA GPU accelerator. You can use AWS IoT Greengrass<\/a> to deploy, and manage your edge compatible customized models on your fleet of devices.<\/p>\n

Solution overview<\/h2>\n

In this post, we use a printed circuit board dataset<\/a> composed of normal and defective images such as scratches, solder blobs, and damaged components on the board. We train a Lookout for Vision model in the cloud to identify defective and normal printed circuit boards. We compile the model to a target ARM architecture, package the trained Lookout for Vision model as an AWS IoT Greengrass<\/a> component, and deploy the model to an NVIDIA Jetson edge device using the AWS IoT Greengrass console. Finally, we demonstrate a Python-based sample application running on the NVIDIA Jetson edge device that sources the printed circuit board image from the edge device file system, runs the inference on the Lookout for Vision model using the gRPC<\/a> interface, and sends the inference data to an MQTT topic in the AWS Cloud.<\/p>\n

The following diagram illustrates the solution architecture.<\/p>\n

\"\"<\/p>\n

The solution has the following workflow:<\/p>\n

    \n
  1. Upload a training dataset to Amazon Simple Storage Service<\/a> (Amazon S3).<\/li>\n
  2. Train a Lookout for Vision model in the cloud.<\/li>\n
  3. Compile the model to the target architecture (ARM) and deploy the model to the NVIDIA Jetson edge device using the AWS IoT Greengrass console.<\/li>\n
  4. Source images from local disk.<\/li>\n
  5. Run inferences on the deployed model via the gRPC interface.<\/li>\n
  6. Post the inference results to an MQTT client running on the edge device.<\/li>\n
  7. Receive the MQTT message on a topic in AWS IoT Core<\/a> in the AWS Cloud for further monitoring and visualization.<\/li>\n<\/ol>\n

    Steps 4, 5 and 6 are coordinated with the sample Python application.<\/p>\n

    \n

    Prerequisites<\/h2>\n

    Before you get started, complete the following prerequisites:<\/p>\n

      \n
    1. Create an AWS account<\/a>.<\/li>\n
    2. On your NVIDIA Jetson edge device, complete the following:\n
        \n
      1. Set up your edge device<\/a> (we have set IoT THING_NAME = l4vJetsonXavierNx when installing AWS IoT Greengrass V2<\/code>).<\/li>\n
      2. Clone the sample project containing the Python-based sample application (warmup-model.py<\/code> to load the model, and sample-client-file-mqtt.py<\/code> to run inferences). Load the Python modules. See the following code:<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
        \n
        git clone https:\/\/github.com\/aws-samples\/ds-peoplecounter-l4v-workshop.git\ncd ds-peoplecounter-l4v-workshop \npip3 install -r requirements.txt\ncd lab2\/inference_client  \n# Replace ENDPOINT variable in sample-client-file-mqtt.py with the \n# value on the AWS console AWS IoT->Things->l4JetsonXavierNX->Interact.  \n# Under HTTPS. It will be of type <name>-ats.iot.<region>.amazon.com \n<\/code><\/pre>\n<\/p><\/div>\n<\/p><\/div>\n
        Dataset and model training<\/h2>\n
        We use the printed circuit board dataset<\/a> to demonstrate the solution. The dataset contains normal and anomalous images. Here are a few sample images from the dataset.<\/p>\n
        The following image shows a normal printed circuit board.<\/p>\n
        \"\"<\/p>\n
        The following image shows a printed circuit board with scratches.<\/p>\n
        \"\"<\/p>\n
        The following image shows a printed circuit board with a soldering defect.<\/p>\n
        \"\"<\/p>\n
        To train a Lookout for Vision model, we follow the steps outlined in Amazon Lookout for Vision \u2013 New ML Service Simplifies Defect Detection for Manufacturing<\/a>. After you complete these steps, you can navigate to the project and the Models<\/strong> page to check the performance of the trained model. You can start the process of exporting the model to the target edge device any time after the model is trained.<\/p>\n
        \"\"<\/p>\n
        Compile and package the model as an AWS IoT Greengrass component<\/h2>\n
        In this section, we walk through the steps to compile the printed circuit board model to our target edge device and package the model as an AWS IoT Greengrass component.<\/p>\n
          \n
        1. On the Lookout for Vision console, choose your project.<\/li>\n
        2. In the navigation pane, choose Edge model packages<\/strong>.<\/li>\n
        3. Choose Create model packaging job<\/strong>.<\/li>\n<\/ol>\n
          \"\"<\/p>\n
            \n
          1. For Job name<\/strong>, enter a name.<\/li>\n
          2. For Job description<\/strong>, enter an optional description.<\/li>\n
          3. Choose Browse models<\/strong>.<\/li>\n<\/ol>\n
            \"\"<\/p>\n
              \n
            1. Select the model version (the printed circuit board model built in the previous section).<\/li>\n
            2. Choose Choose<\/strong>.<\/li>\n<\/ol>\n
              \"\"<\/p>\n
                \n
              1. Select Target device <\/strong>and enter the compiler options.<\/li>\n<\/ol>\n
                Our target device is on JetPack 4.5.1. See\u00a0this page<\/a>\u00a0for additional details on supported platforms. You can find the supported compiler options such as trt-ver<\/code> and cuda-ver<\/code> in the\u00a0NVIDIA JetPack 4.5.1 archive<\/a>.<\/p>\n
                \"\"<\/p>\n
                  \n
                1. Enter the details for Component name<\/strong>, Component description<\/strong> (optional), Component version<\/strong>, and Component location.<\/strong><\/li>\n<\/ol>\n
                  Amazon Lookout for Vision stores the component recipes and artifacts in this Amazon S3 location.<\/p>\n
                  \"\"<\/p>\n
                    \n
                  1. Choose Create model packaging job<\/strong>.<\/li>\n<\/ol>\n
                    \"\"<\/p>\n
                    You can see your job name and status showing as In progress<\/code>. The model packaging job may take a few minutes to complete.<\/p>\n
                    \"\"<\/p>\n
                    When the model packaging job is complete, the status shows as Success<\/code>.<\/p>\n
                    \"\"<\/p>\n
                      \n
                    1. Choose your job name (in our case it\u2019s ComponentCircuitBoard<\/code>) to see the job details.<\/li>\n<\/ol>\n
                      The Greengrass component and model artifacts have been created in your AWS account.<\/p>\n
                        \n
                      1. Choose Continue deployment to Greengrass<\/strong> to deploy the component to the target edge device.<\/li>\n<\/ol>\n
                        \"\"<\/p>\n
                        Deploy the model<\/h3>\n
                        In this section, we walk through the steps to deploy the printed circuit board model to the edge device using the AWS IoT Greengrass console.<\/p>\n
                          \n
                        1. Choose Deploy<\/strong> to initiate the deployment steps.<\/li>\n<\/ol>\n
                          \"\"<\/p>\n
                            \n
                          1. Select Core device<\/strong> (because the deployment is to a single device) and enter a name for Target name<\/strong>.<\/li>\n<\/ol>\n
                            The target name is the same name you used to name the core device during the AWS IoT Greengrass V2 installation process.<\/p>\n
                            \"\"<\/p>\n
                              \n
                            1. Choose your component. In our case, the component name is ComponentCircuitBoard<\/code>, which contains the circuit board model.<\/li>\n
                            2. Choose Next<\/strong>.<\/li>\n<\/ol>\n
                              \"\"<\/p>\n
                                \n
                              1. Configure the component (optional).<\/li>\n
                              2. Choose Next<\/strong>.<\/li>\n<\/ol>\n
                                \"\"<\/p>\n
                                  \n
                                1. Expand Deployment policies.<\/strong><\/li>\n<\/ol>\n
                                  \"\"<\/p>\n
                                    \n
                                  1. For Component update policy<\/strong>, select Notify components<\/strong>.<\/li>\n<\/ol>\n
                                    This allows the already deployed component (a prior version of the component) to defer an update until they are ready to update.<\/p>\n
                                      \n
                                    1. For Failure handling policy<\/strong>, select Don\u2019t roll back<\/strong>.<\/li>\n<\/ol>\n
                                      In case of a failure, this option allows us to investigate the errors in deployment.<\/p>\n
                                        \n
                                      1. Choose Next<\/strong>.<\/li>\n<\/ol>\n
                                        \"\"<\/p>\n
                                          \n
                                        1. Review the list of components that will be deployed on the target (edge) device.<\/li>\n
                                        2. Choose Next<\/strong>.<\/li>\n<\/ol>\n
                                          \"\"<\/p>\n
                                          You should see the message Deployment successfully created<\/code>.<\/p>\n
                                          \"\"<\/p>\n
                                            \n
                                          1. To validate the model deployment was successful, run the following command on your edge device:<\/li>\n<\/ol>\n
                                            \n
                                            sudo \/greengrass\/v2\/bin\/greengrass-cli component list<\/code><\/pre>\n<\/p><\/div>\n
                                            You should see a similar looking output running the ComponentCircuitBoard<\/code> lifecycle startup script:<\/p>\n
                                            \n
                                            \n
                                             Components currently running in Greengrass:\n \n Component Name: aws.iot.lookoutvision.EdgeAgent\n    Version: 0.1.34\n    State: RUNNING\n    Configuration: {\"Socket\":\"unix:\/\/\/tmp\/aws.iot.lookoutvision.EdgeAgent.sock\"}\n Component Name: ComponentCircuitBoard\n    Version: 1.0.0\n    State: RUNNING\n    Configuration: {\"Autostart\":false}\n<\/code><\/pre>\n<\/p><\/div>\n<\/p><\/div>\n
                                            Run inferences on the model<\/h3>\n
                                            We\u2019re now ready to run inferences on the model. On your edge device, run the following command to load the model:<\/p>\n
                                            \n
                                            # run command to load the model\n# This will load the model into running state \npython3 warmup-model.py\n<\/code><\/pre>\n<\/p><\/div>\n
                                            To generate inferences, run the following command with the source file name:<\/p>\n
                                            \n
                                            python3 sample-client-file-mqtt.py \/path\/to\/images<\/code><\/pre>\n<\/p><\/div>\n
                                            The following screenshot shows that the model correctly predicts the image as anomalous (bent pin) with a confidence score of 0.999766.<\/p>\n
                                            \"\"<\/p>\n
                                            The following screenshot shows that the model correctly predicts the image as anomalous (solder blob) with a confidence score of 0.7701461.<\/p>\n
                                            \"\"<\/p>\n
                                            The following screenshot shows that the model correctly predicts the image as normal with a confidence score of 0.9568462.<\/p>\n
                                            \"\"<\/p>\n
                                            The following screenshot shows that the inference data posted an MQTT topic in AWS IoT Core.<\/p>\n
                                            \"\"<\/p>\n
                                            Customer Stories<\/h2>\n
                                            With AWS IoT Greengrass and Amazon Lookout for Vision, you can now automate visual inspection with CV for processes like quality control and defect assessment \u2013 all on the edge and in real time. You can proactively identify problems such as parts damage (like dents, scratches, or poor welding), missing product components, or defects with repeating patterns, on the production line itself \u2013 saving you time and money. Customers like Tyson and Baxter are discovering the power of Amazon Lookout for Vision to increase quality and reduce operational costs by automating visual inspection.<\/p>\n
                                            \n
                                            \u201cOperational excellence is a key priority at Tyson Foods. Predictive maintenance is an essential asset for achieving this objective by continuously improving overall equipment effectiveness (OEE). In 2021, Tyson Foods launched a machine learning based computer vision project to identify failing product carriers during production to prevent them from impacting Team Member safety, operations, or product quality.<\/em><\/p>\n
                                            The models trained using Amazon Lookout for Vision performed well. The pin detection model achieved 95% accuracy across both classes. The Amazon Lookout for Vision model was tuned to perform at 99.1% accuracy for failing pin detection. By far the most exciting result of this project was the speedup in development time. Although this project utilizes two models and a more complex application code, it took 12% less developer time to complete. This project for monitoring the condition of the product carriers at Tyson Foods was completed in record time using AWS managed services such as Amazon Lookout for Vision.\u201d<\/em><\/p>\n
                                            Audrey Timmerman, Sr Applications Developer, Tyson Foods.<\/strong><\/p>\n<\/blockquote>\n
                                            \n
                                            \u201cWe use Amazon Lookout for Vision to automate inspection tasks and solve complex process management problems that can\u2019t be addressed by manual inspection or traditional machine vision alone. Lookout for Vision\u2019s cloud and edge capabilities provide us the ability to leverage computer vision and AI\/ML-based solutions at scale in a rapid and agile manner, helping us to drive efficiencies on the manufacturing shop floor and enhance our operator\u2019s productivity and experience.\u201d<\/em><\/p>\n
                                            K. Karan, Global Senior Director \u2013 Digital Transformation, Integrated Supply Chain, Baxter International Inc.<\/strong><\/p>\n<\/blockquote>\n
                                            Conclusion<\/h2>\n
                                            In this post, we described a typical scenario for industrial defect detection at the edge. We walked through the key components of the cloud and edge lifecycle with an end-to-end example using Lookout for Vision and AWS IoT Greengrass. With Lookout for Vision, we trained an anomaly detection model in the cloud using the printed circuit board dataset, compiled the model to a target architecture, and packaged the model as an AWS IoT Greengrass component. With AWS IoT Greengrass, we deployed the model to an edge device. We demonstrated a Python-based sample application that sources printed circuit board images from the edge device local file system, runs the inferences on the Lookout for Vision model at the edge using the gRPC interface, and sends the inference data to an MQTT topic in the AWS Cloud.<\/p>\n
                                            In a future post, we will show how to run inferences on a real-time stream of images using a GStreamer media pipeline.<\/p>\n
                                            Start your journey towards industrial anomaly detection and identification by visiting the Amazon Lookout for Vision<\/a> and AWS IoT Greengrass<\/a> resource pages.<\/p>\n
                                            \n
                                            About the Authors<\/h3>\n
                                            \"\"Amit Gupta<\/strong> is an AI Services Solutions Architect at AWS. He is passionate about enabling customers with well-architected machine learning solutions at scale.<\/p>\n
                                            \u00a0\"\"Ryan Vanderwerf<\/strong> is a partner solutions architect at Amazon Web Services. He previously provided Java virtual machine-focused consulting and project development as a software engineer at OCI on the Grails and Micronaut team. He was chief architect\/director of products at ReachForce, with a focus on software and system architecture for AWS Cloud SaaS solutions for marketing data management. Ryan has built several SaaS solutions in several domains such as financial, media, telecom, and e-learning companies since 1996.<\/p>\n
                                            \"\"Prathyusha Cheruku<\/strong> is an AI\/ML Computer Vision Product Manager at AWS. She focuses on building powerful, easy-to-use, no code\/ low code deep learning-based image and video analysis services for AWS customers.<\/p>\n
                                                   \n      <\/div>\n","protected":false},"excerpt":{"rendered":"
                                            https:\/\/aws.amazon.com\/blogs\/machine-learning\/amazon-lookout-for-vision-now-supports-visual-inspection-of-product-defects-at-the-edge\/<\/p>\n","protected":false},"author":0,"featured_media":1356,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1355"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1355"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1355\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1356"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1355"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1355"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1355"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}