{"id":684,"date":"2020-12-13T00:13:20","date_gmt":"2020-12-13T00:13:20","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/12\/13\/exploratory-data-analysis-feature-engineering-and-operationalizing-your-data-flow-into-your-ml-pipeline-with-amazon-sagemaker-data-wrangler\/"},"modified":"2020-12-13T00:13:20","modified_gmt":"2020-12-13T00:13:20","slug":"exploratory-data-analysis-feature-engineering-and-operationalizing-your-data-flow-into-your-ml-pipeline-with-amazon-sagemaker-data-wrangler","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/12\/13\/exploratory-data-analysis-feature-engineering-and-operationalizing-your-data-flow-into-your-ml-pipeline-with-amazon-sagemaker-data-wrangler\/","title":{"rendered":"Exploratory data analysis, feature engineering, and operationalizing your data flow into your ML pipeline with Amazon SageMaker Data Wrangler"},"content":{"rendered":"
\n

According to The State of Data Science 2020<\/a> survey, data management, exploratory data analysis (EDA), feature selection, and feature engineering accounts for more than 66% of a data scientist\u2019s time (see the following diagram).<\/p>\n

\"According<\/p>\n

The same survey highlights that the top three biggest roadblocks to deploying a model in production are managing dependencies and environments, security, and skill gaps (see the following diagram).<\/p>\n

\"The<\/p>\n

The survey posits that these struggles result in fewer than half (48%) of the respondents feeling able to illustrate the impact data science has on business outcomes.<\/p>\n

Enter Amazon SageMaker Data Wrangler<\/a>, the fastest and easiest way to prepare data for machine learning (ML). SageMaker Data Wrangler gives you the ability to use a visual interface to access data, perform EDA and feature engineering, and seamlessly operationalize your data flow by exporting it into an Amazon SageMaker<\/a> pipeline, Amazon SageMaker Data Wrangler job, Python file, or SageMaker feature group.<\/p>\n

SageMaker Data Wrangler also provides you with over 300 built-in transforms, custom transforms using a Python, PySpark or SparkSQL runtime, built-in data analysis such as common charts (like scatterplot or histogram), custom charts using the Altair library<\/a>, and useful model analysis capabilities such as feature importance, target leakage, and model explainability. Finally, SageMaker Data Wrangler creates a data flow file that can be versioned and shared across your teams for reproducibility.<\/p>\n

Solution overview<\/h2>\n

In this post, we use the retail demo store<\/a> example and generate<\/a> a sample dataset. We use three files: users.csv, items.csv, and interactions.csv. We first prepare the data in order to predict the customer segment based on past interactions. Our target is the field called persona<\/code>, which we later transform and rename to USER_SEGMENT<\/code>.<\/p>\n

The following code is a preview of the users dataset:<\/p>\n

\n
id,username,email,first_name,last_name,addresses,age,gender,persona\r\n1,user1,nathan.smith@example.com,Nathan,Smith,\"[{\"\"first_name\"\": \"\"Nathan\"\", \"\"last_name\"\": \"\"Smith\"\", \"\"address1\"\": \"\"049 Isaac Stravenue Apt. 770\"\", \"\"address2\"\": \"\"\"\", \"\"country\"\": \"\"US\"\", \"\"city\"\": \"\"Johnsonmouth\"\", \"\"state\"\": \"\"NY\"\", \"\"zipcode\"\": \"\"12758\"\", \"\"default\"\": true}]\",28,M,electronics_beauty_outdoors\r\n2,user2,kevin.martinez@example.com,Kevin,Martinez,\"[{\"\"first_name\"\": \"\"Kevin\"\", \"\"last_name\"\": \"\"Martinez\"\", \"\"address1\"\": \"\"074 Jennifer Flats Suite 538\"\", \"\"address2\"\": \"\"\"\", \"\"country\"\": \"\"US\"\", \"\"city\"\": \"\"East Christineview\"\", \"\"state\"\": \"\"MI\"\", \"\"zipcode\"\": \"\"49758\"\", \"\"default\"\": true}]\",19,M,electronics_beauty_outdoors<\/code><\/pre>\n<\/div>\n
The following code is a preview of the items dataset:<\/p>\n
\n
ITEM_ID,ITEM_URL,ITEM_SK,ITEM_NAME,ITEM_CATEGORY,ITEM_STYLE,ITEM_DESCRIPTION,ITEM_PRICE,ITEM_IMAGE,ITEM_FEATURED,ITEM_GENDER_AFFINITY\r\n36,http:\/\/dbq4nocqaarhp.cloudfront.net\/#\/product\/36,,Exercise Headphones,electronics,headphones,These stylishly red ear buds wrap securely around your ears making them perfect when exercising or on the go.,19.99,5.jpg,true,\r\n49,http:\/\/dbq4nocqaarhp.cloudfront.net\/#\/product\/49,,Light Brown Leather Lace-Up Boot,footwear,boot,Sturdy enough for the outdoors yet stylish to wear out on the town.,89.95,11.jpg,,<\/code><\/pre>\n<\/div>\n
The following code is a preview of the interactions dataset:<\/p>\n
\n
ITEM_ID,USER_ID,EVENT_TYPE,TIMESTAMP\r\n2,2539,ProductViewed,1589580300\r\n29,5575,ProductViewed,1589580305\r\n4,1964,ProductViewed,1589580309\r\n46,5291,ProductViewed,1589580309<\/code><\/pre>\n<\/div>\n
This post is not intended to be a step-by-step guide, but rather describe the process of preparing a training dataset and highlight some of the transforms and data analysis capabilities using SageMaker Data Wrangler. You can download the .flow files<\/a> if you want to download, upload, and retrace the full example in your SageMaker Studio environment.<\/p>\n
At a high level, we perform the following steps:\u00a0<\/strong><\/p>\n
    \n
  1. Connect to Amazon Simple Storage Service<\/a> (Amazon S3) and import the data.<\/li>\n
  2. Transform the data, including type casting, dropping unneeded columns, imputing missing values, label encoding, one hot encoding, and custom transformations to extract elements from a JSON formatted column.<\/li>\n
  3. Create table summaries and charts for data analysis. We use the quick model option to get a sense of which features are adding predictive power as we progress with our data preparation. We also use the built-in target leakage capability and get a report on any features that are at risk of leaking.<\/li>\n
  4. Create a data flow, in which we combine and join the three tables to perform further aggregations and data analysis.<\/li>\n
  5. Iterate by performing additional feature engineering or data analysis on the newly added data.<\/li>\n
  6. Export our workflow to a SageMaker Data Wrangler job.<\/li>\n<\/ol>\n
    Prerequisites<\/h2>\n
    Make sure you don\u2019t have any quota limits on the m5.4xlarge instance type part of your Studio application before creating a new data flow. For more information about prerequisites, see Getting Started with Data Wrangler<\/a>.<\/p>\n
    Importing the data<\/h2>\n
    We import our three CSV files from Amazon S3. SageMaker Data Wrangler supports CSV and Parquet files. It also allows you to sample the data in case the data is too large to fit in your studio application. The following screenshot shows a preview of the users dataset.<\/p>\n
    \"A<\/p>\n
    After importing our CSV files, our datasets look like the following screenshot in SageMaker Data Wrangler.<\/p>\n
    \"After<\/p>\n
    We can now add some transforms and perform data analysis.<\/p>\n
    Transforming the data<\/h2>\n
    For each table, we check the data types and make sure that it was inferred correctly.<\/p>\n
    Items table<\/h3>\n
    To perform transforms on the items table, complete the following steps:<\/p>\n
      \n
    1. On the SageMaker Data Wrangler UI, for the items table, choose +<\/strong>.<\/li>\n
    2. Choose Edit data types<\/strong>.<\/li>\n<\/ol>\n
      \"On<\/p>\n
      Most of the columns were inferred properly, except for one. The ITEM_FEATURED<\/code> column is missing values and should really be casted as a Boolean.<\/p>\n
      \"The<\/p>\n
      For the items table, we perform the following transformations:<\/p>\n