{"id":1507,"date":"2022-01-27T17:51:01","date_gmt":"2022-01-27T17:51:01","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/27\/how-clearly-accurately-predicts-fraudulent-orders-using-amazon-fraud-detector\/"},"modified":"2022-01-27T17:51:01","modified_gmt":"2022-01-27T17:51:01","slug":"how-clearly-accurately-predicts-fraudulent-orders-using-amazon-fraud-detector","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/27\/how-clearly-accurately-predicts-fraudulent-orders-using-amazon-fraud-detector\/","title":{"rendered":"How Clearly accurately predicts fraudulent orders using Amazon Fraud Detector"},"content":{"rendered":"<div id=\"\">\n<p><em>This post was cowritten by Ziv Pollak, Machine Learning Team Lead, and Sarvi Loloei, Machine Learning Engineer at Clearly. The content and opinions in this post are those of the third-party authors and AWS is not responsible for the content or accuracy of this post.<\/em><\/p>\n<p>A pioneer in online shopping, Clearly launched their first site in 2000. Since then, we\u2019ve grown to become one of the biggest online eyewear retailers in the world, providing customers across Canada, the US, Australia, and New Zealand with glasses, sunglasses, contact lenses, and other eye health products. Through its mission to eliminate poor vision, Clearly strives to make eyewear affordable and accessible for everyone. Creating an optimized fraud detection platform is a key part of this wider vision.<\/p>\n<p>Identifying online fraud is one of the biggest challenges every online retail organization has\u2014hundreds of thousands of dollars are lost due to fraud every year. Product costs, shipping costs, and labor costs for handling fraudulent orders further increase the impact of fraud. Easy and fast fraud evaluation is also critical for maintaining high customer satisfaction rates. Transactions shouldn\u2019t be delayed due to lengthy fraud investigation cycles.<\/p>\n<p>In this post, we share how Clearly built an automated and orchestrated forecasting pipeline using <a href=\"https:\/\/aws.amazon.com\/step-functions\/\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Step Functions<\/a>, and used <a href=\"https:\/\/aws.amazon.com\/fraud-detector\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Fraud Detector<\/a> to train a machine learning (ML) model that can identify online fraudulent transactions and bring them to the attention of the billing operations team. This solution also collects metrics and logs, provides auditing, and is invoked automatically.<\/p>\n<p>With AWS services, Clearly deployed a serverless, well-architected solution in just a few weeks.<\/p>\n<h2>The challenge: Predicting fraud quickly and accurately<\/h2>\n<p>Clearly\u2019s existing solution was based on flagging transactions using hard-coded rules that weren\u2019t updated frequently enough to capture new fraud patterns. Once flagged, the transaction was manually reviewed by a member of the billing operations team.<\/p>\n<p>This existing process had major drawbacks:<\/p>\n<ul>\n<li><strong>Inflexible<\/strong> <strong>and inaccurate<\/strong> \u2013 The hard-coded rules to identify fraud transactions were difficult to update, meaning the team couldn\u2019t respond quickly to emerging fraud trends. The rules were unable to accurately identify many suspicious transactions.<\/li>\n<li><strong>Operationally intensive<\/strong> \u2013 The process couldn\u2019t scale to high sales volume events (like Black Friday), requiring the team to implement workarounds or accept higher fraud rates. Moreover, the high level of human involvement added significant cost to the product delivery process.<\/li>\n<li><strong>Delayed orders<\/strong> \u2013 The order fulfillment timeline was delayed by manual fraud reviews, leading to unhappy customers.<\/li>\n<\/ul>\n<p>Although our existing fraud identification process was a good starting point, it was neither accurate enough nor fast enough to meet the order fulfillment efficiencies that Clearly desired.<\/p>\n<p>Another major challenge we faced was the lack of a tenured ML team\u2014all members had been with the company less than a year when the project kicked off.<\/p>\n<h2>Overview of solution: Amazon Fraud Detector<\/h2>\n<p>Amazon Fraud Detector is a fully managed service that uses ML to deliver highly accurate fraud detection and requires no ML expertise. All we had to do was upload our data and follow a few straightforward steps. Amazon Fraud Detector automatically examined the data, identified meaningful patterns, and produced a fraud identification model capable of making predictions on new transactions.<\/p>\n<p>The following diagram illustrates our pipeline:<\/p>\n<p><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/26\/ML-7595-image001-new.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-32496 size-full\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/26\/ML-7595-image001-new.png\" alt=\"\" width=\"671\" height=\"514\"><\/a><\/p>\n<p>To operationalize the flow, we applied the following workflow:<\/p>\n<ol>\n<li><a href=\"https:\/\/aws.amazon.com\/eventbridge\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon EventBridge<\/a> calls the orchestration pipeline hourly to review all pending transactions.<\/li>\n<li>Step Functions helps manage the orchestration pipeline.<\/li>\n<li>An <a href=\"https:\/\/aws.amazon.com\/lambda\/\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Lambda<\/a> function calls <a href=\"https:\/\/aws.amazon.com\/athena\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Athena<\/a> APIs to retrieve and prepare the training data, stored on <a href=\"https:\/\/aws.amazon.com\/s3\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Simple Storage Service<\/a> (Amazon S3).<\/li>\n<li>An orchestrated pipeline of Lambda functions trains an Amazon Fraud Detector model and saves the model performance metrics to an S3 bucket.<\/li>\n<li><a href=\"https:\/\/docs.aws.amazon.com\/forecast\/latest\/dg\/howitworks-forecast.html\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Simple Notification Service<\/a> (Amazon SNS) notifies users when a problem occurs during the fraud detection process or when the process completes successfully.<\/li>\n<li>Business analysts build dashboards on <a href=\"https:\/\/aws.amazon.com\/quicksight\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon QuickSight<\/a>, which queries the fraud data from Amazon S3 using Athena, as we describe later in this post.<\/li>\n<\/ol>\n<p>We chose to use Amazon Fraud Detector for a few reasons:<\/p>\n<ul>\n<li>The service taps into years of expertise that Amazon has fighting fraud. This gave us a lot of confidence in the service\u2019s capabilities.<\/li>\n<li>The ease of use and implementation allowed us to quickly confirm we have the dataset we need to produce accurate results.<\/li>\n<li>Because the Clearly ML team was less than 1 year old, a fully managed service allowed us to deliver this project without needing deep technical ML skills and knowledge.<\/li>\n<\/ul>\n<h2>Results<\/h2>\n<p>Writing the prediction results into our existing data lake allows us to use QuickSight to build metrics and dashboards for senior leadership. This enables them to understand and use these results when making decisions on the next steps to meet our monthly marketing targets.<\/p>\n<p>We were able to present the forecast results on two levels, starting with overall business performance and then going deeper into needed performance per each line of business (contacts and glasses).<\/p>\n<p>Our dashboard includes the following information:<\/p>\n<ul>\n<li>Fraud per day per different lines of business<br \/><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image003.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-32389\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image003.png\" alt=\"\" width=\"1357\" height=\"275\"><\/a><\/li>\n<li>Revenue loss due to fraud transactions<br \/><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image005.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-32391\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image005.png\" alt=\"\" width=\"1033\" height=\"301\"><\/a><\/li>\n<li>Location of fraud transactions (identifying fraud hot spots)<br \/><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image007.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-32392\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image007.png\" alt=\"\" width=\"782\" height=\"367\"><\/a><\/li>\n<li>Fraud transactions impact by different coupon codes, which allows us to monitor for problematic coupon codes and take further actions to reduce the risk<br \/><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image009.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-32393\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/ML-7595image009.png\" alt=\"\" width=\"930\" height=\"238\"><\/a><\/li>\n<li>Fraud per hour, which allows us to plan and manage the billing operation team and make sure we have resources available to handle transaction volume when needed<\/li>\n<\/ul>\n<h2>Conclusions<\/h2>\n<p>Effective and accurate prediction of customer fraud is one of the biggest challenges in ML for retail today, and having a good understanding of our customers and their behavior is vital to Clearly\u2019s success. Amazon Fraud Detector provided a fully managed ML solution to easily create an accurate and reliable fraud prediction system with minimal overhead. Amazon Fraud Detector predictions have a high degree of accuracy and are simple to generate.<\/p>\n<p>\u201c<em>With leading ecommerce tools like <a href=\"https:\/\/www.clearly.ca\/glasses\/virtual-try-on-glasses\" target=\"_blank\" rel=\"noopener noreferrer\">Virtual Try On<\/a>, combined with our unparalleled customer service, we strive to help everyone see clearly in an affordable and effortless manner\u2014which means constantly looking for ways to innovate, improve, and streamline processes,<\/em>\u201d said Dr. Ziv Pollak, Machine Learning Team Leader. \u201c<em>Online fraud detection is one of the biggest challenges in machine learning in retail today. In just a few weeks, Amazon Fraud Detector helped us accurately and reliably identify fraud with a very high level of accuracy, and save thousands of dollars.<\/em>\u201d<\/p>\n<hr>\n<h3><strong>About the Author<\/strong><\/h3>\n<p><strong><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/08\/03\/image009.jpg\"><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-26793 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/08\/03\/image009.jpg\" alt=\"Dr Ziv Pollak \" width=\"100\" height=\"100\"><\/a>Dr. Ziv Pollak<\/strong> is an experienced technical leader who transforms the way organizations use machine learning to increase revenue, reduce costs, improve customer service, and ensure business success. He is currently leading the Machine Learning team at Clearly.<\/p>\n<p><strong><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/Dr.-Ziv-Pollak.jpg\"><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-32387 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2022\/01\/20\/Dr.-Ziv-Pollak.jpg\" alt=\"\" width=\"100\" height=\"130\"><\/a>Sarvi Loloei<\/strong> is an Associate Machine Learning Engineer at Clearly. Using AWS tools, she evaluates model effectiveness to drive business growth, increase revenue, and optimize productivity.<\/p>\n<p>       <!-- '\"` -->\n      <\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/aws.amazon.com\/blogs\/machine-learning\/how-clearly-accurately-predicts-fraudulent-orders-using-amazon-fraud-detector\/<\/p>\n","protected":false},"author":0,"featured_media":1508,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1507"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1507"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1507\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1508"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1507"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1507"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1507"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}