{"id":1083,"date":"2021-10-27T08:42:36","date_gmt":"2021-10-27T08:42:36","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/10\/27\/train-models-faster-with-an-automated-data-profiler-for-amazon-fraud-detector\/"},"modified":"2021-10-27T08:42:36","modified_gmt":"2021-10-27T08:42:36","slug":"train-models-faster-with-an-automated-data-profiler-for-amazon-fraud-detector","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/10\/27\/train-models-faster-with-an-automated-data-profiler-for-amazon-fraud-detector\/","title":{"rendered":"Train models faster with an automated data profiler for Amazon Fraud Detector"},"content":{"rendered":"
\n

Amazon Fraud Detector<\/a> is a fully managed service that makes it easy to identify potentially fraudulent online activities, such as the creation of fake accounts or online payment fraud. Amazon Fraud Detector uses machine learning (ML) under the hood and is based on over 20 years of fraud detection expertise from Amazon. It automatically identifies potentially fraudulent activity in milliseconds\u2014with no ML expertise required.<\/p>\n

To train a model in Amazon Fraud Detector, you need to supply a historical dataset. Amazon Fraud Detector doesn\u2019t require any data science knowledge to use; however, it does have certain requirements<\/a> on the data quality and formats to ensure the robustness of the ML models. You may sometimes encounter model training errors due to simple format and validation errors, which lead to extra time and effort to re-prepare the data and retrain the model. In addition, Amazon Fraud Detector requires you to define a variable type<\/a> for each variable in the dataset during model creation. It may be helpful to have suggestions on selecting Amazon Fraud Detector variable types based on your data statistics.<\/p>\n

In this post, we present an automated data profiler for Amazon Fraud Detector. It can generate an intuitive and comprehensive report of your dataset, which includes suggested Amazon Fraud Detector variable types for each variable in the dataset, and data quality issues that may potentially fail model training or hurt model performance. The data profiler also provides an option to reformat and transform the dataset to satisfy requirements in Amazon Fraud Detector, which can avoid some potential validation errors in model training. This automated data profiler is built with an AWS CloudFormation<\/a> stack, which you can easily launch with a few clicks, and it doesn\u2019t require any data science or programming knowledge.<\/p>\n

Overview of solution<\/h2>\n

The following diagram illustrates the architecture of the automated data profiler, which uses AWS Glue<\/a>, AWS Lambda<\/a>, Amazon Simple Storage Service<\/a> (Amazon S3), and AWS CloudFormation.<\/p>\n

\"\"<\/a><\/p>\n

You can launch the data profiler with the quick launch feature of AWS CloudFormation. The stack creates and triggers a Lambda function, which automatically triggers an AWS Glue job. The AWS Glue job reads your CSV data file, profiles and reformats your data, and saves the HTML report file and formatted copy of the CSV to an S3 bucket.<\/p>\n

The following screenshot shows a sample profiling report. You can also view the full sample report<\/a>.<\/p>\n

\"\"<\/a><\/p>\n

The sample report, synthetic dataset, and codes of the automated data profiler are available on GitHub<\/a>.<\/p>\n

Launch the data profiler<\/h2>\n

Follow these steps to launch the profiler:<\/p>\n

    \n
  1. Choose the following AWS CloudFormation quick launch link<\/a>.
    \"\"<\/a><\/li>\n<\/ol>\n

    This opens an AWS CloudFormation quick launch page.<\/p>\n

      \n
    1. Choose your Region to create all the resources in that Region.<\/li>\n
    2. For CSVFilePath<\/strong>, enter S3 path to your CSV file.<\/li>\n<\/ol>\n

      The output profiling report and formatted CSV file are saved under the same bucket.<\/p>\n

        \n
      1. For EventTimestampColumn<\/strong>, enter the header name of the event timestamp column.<\/li>\n<\/ol>\n

        This is a mandatory column required by Amazon Fraud Detector. The data formatter converts this header name to EVENT_TIMESTAMP<\/code>.<\/p>\n

          \n
        1. For LabelColumn<\/strong>, enter the header name of the label column.<\/li>\n<\/ol>\n

          This is a mandatory column required by Amazon Fraud Detector. The data formatter converts this header name to EVENT_LABEL<\/code>.<\/p>\n

            \n
          1. For FileDelimiter<\/strong>, enter the delimiter of your CSV file (by default, this is a comma).<\/li>\n
          2. For FormatCSV<\/strong>, choose whether you want to format the CSV file to the Amazon Fraud Detector required format (by default, this is Yes<\/strong>).<\/li>\n<\/ol>\n

            This transforms the header names, timestamp formats, and label formats. The formatted copy of your CSV data is saved in the same bucket as the input CSV.<\/p>\n

              \n
            1. For DropTimestampMissingRows<\/strong>, choose whether you want to drop rows with missing timestamp in the formatted copy of the CSV.<\/li>\n<\/ol>\n

              Events with a missing timestamp aren\u2019t used by Amazon Fraud Detector, and may cause validation errors, so we suggest setting this to Yes<\/strong>.<\/p>\n

                \n
              1. For DropLabelMissingRows<\/strong>, choose whether you want to drop rows with missing labels.<\/li>\n
              2. For ProfileCSV<\/strong>, choose whether you want to profile the CSV file (by default, this is Yes<\/strong>).<\/li>\n<\/ol>\n

                This generates a profiling report of your CSV data and saves it in the same bucket as the input CSV.<\/p>\n

                  \n
                1. For ReportSuffix (Optional)<\/strong>, specify a suffix for the report (the report is named report_<ReportSuffix><\/span>.html<\/code>).<\/li>\n
                2. For FeatureCorr<\/strong>, choose whether you want to show pair-wise feature correlation in the profiling report.<\/li>\n<\/ol>\n

                  The correlation shows for each pair of features, how much one feature depends on the other. Note that computing pair-wise feature correlation takes an additional 10\u201320 minutes, so the option is set to No<\/strong> by default.<\/p>\n

                    \n
                  1. For FraudLabels (Optional)<\/strong>, specify which label values should be considered as fraud.<\/li>\n<\/ol>\n

                    The report shows the distribution of mapped labels, namely fraud and non-fraud. You can specify multiple label values by separating with a comma, for example, suspicious<\/code>, fraud<\/code>. If you leave this option blank, the report shows the distribution of the original label values.<\/p>\n

                    The following example plots illustrate using FraudLabels=\u2019suspicious,fraud\u2019<\/code> (left) and empty FraudLabels<\/code> (right).
                    \"\"<\/a><\/p>\n

                    \"\"<\/a><\/p>\n

                    Wait a few minutes for the following resources to be created:<\/p>\n