{"id":2040,"date":"2022-03-31T23:38:38","date_gmt":"2022-03-31T23:38:38","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/31\/prepare-data-from-databricks-for-machine-learning-using-amazon-sagemaker-data-wrangler\/"},"modified":"2022-03-31T23:38:38","modified_gmt":"2022-03-31T23:38:38","slug":"prepare-data-from-databricks-for-machine-learning-using-amazon-sagemaker-data-wrangler","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/31\/prepare-data-from-databricks-for-machine-learning-using-amazon-sagemaker-data-wrangler\/","title":{"rendered":"Prepare data from Databricks for machine learning using Amazon SageMaker Data Wrangler"},"content":{"rendered":"
\n

Data science and data engineering teams spend a significant portion of their time in the data preparation phase of a machine learning (ML) lifecycle performing data selection, cleaning, and transformation steps. It\u2019s a necessary and important step of any ML workflow in order to generate meaningful insights and predictions, because bad or low-quality data greatly reduces the relevance of the insights derived.<\/p>\n

Data engineering teams are traditionally responsible for the ingestion, consolidation, and transformation of raw data for downstream consumption. Data scientists often need to do additional processing on data for domain-specific ML use cases such as natural language and time series. For example, certain ML algorithms may be sensitive to missing values, sparse features, or outliers and require special consideration. Even in cases where the dataset is in a good shape, data scientists may want to transform the feature distributions or create new features in order to maximize the insights obtained from the models. To achieve these objectives, data scientists have to rely on data engineering teams to accommodate requested changes, resulting in dependency and delay in the model development process. Alternatively, data science teams may choose to perform data preparation and feature engineering internally using various programming paradigms. However, it requires an investment of time and effort in installation and configuration of libraries and frameworks, which isn\u2019t ideal because that time can be better spent optimizing model performance.<\/p>\n

Amazon SageMaker Data Wrangler<\/a> simplifies the data preparation and feature engineering process, reducing the time it takes to aggregate and prepare data for ML from weeks to minutes by providing a single visual interface for data scientists to select, clean, and explore their datasets. Data Wrangler offers over 300 built-in data transformations to help normalize, transform, and combine features without writing any code. You can import data from multiple data sources, such as Amazon Simple Storage Service (Amazon S3),<\/a> Amazon Athena<\/a>, Amazon Redshift<\/a>, and Snowflake<\/a>. You can now also use Databricks<\/a> as a data source in Data Wrangler to easily prepare data for ML.<\/p>\n

The Databricks Lakehouse Platform combines the best elements of data lakes and data warehouses to deliver the reliability, strong governance and performance of data warehouses with the openness, flexibility and machine learning support of data lakes. With Databricks as a data source for Data Wrangler, you can now quickly and easily connect to Databricks, interactively query data stored in Databricks using SQL, and preview data before importing. Additionally, you can join your data in Databricks with data stored in Amazon S3, and data queried through Amazon Athena, Amazon Redshift, and Snowflake to create the right dataset for your ML use case.<\/p>\n

In this post, we transform the Lending Club Loan dataset using Amazon SageMaker Data Wrangler for use in ML model training.<\/p>\n

Solution overview<\/h2>\n

The following diagram illustrates our solution architecture.<\/p>\n

\"\"<\/a><\/p>\n

The Lending Club Loan dataset contains complete loan data for all loans issued through 2007\u20132011, including the current loan status and latest payment information. It has 39,717 rows, 22 feature columns, and 3 target labels.<\/p>\n

To transform our data using Data Wrangler, we complete the following high-level steps:<\/p>\n

    \n
  1. Download and split the dataset.<\/li>\n
  2. Create a Data Wrangler flow.<\/li>\n
  3. Import data from Databricks to Data Wrangler.<\/li>\n
  4. Import data from Amazon S3 to Data Wrangler.<\/li>\n
  5. Join the data.<\/li>\n
  6. Apply transformations.<\/li>\n
  7. Export the dataset.<\/li>\n<\/ol>\n

    Prerequisites<\/strong><\/h2>\n

    The post assumes you have a running Databricks cluster. If your cluster is running on AWS, verify you have the following configured:<\/p>\n

    Databricks setup<\/strong><\/h3>\n

    Follow Secure access to S3 buckets using instance profiles<\/a> for the required AWS Identity and Access Management<\/a> (IAM) roles, S3 bucket policy, and Databricks cluster configuration. Ensure the Databricks cluster is configured with the proper Instance Profile<\/code>, selected under the advanced options, to access to the desired S3 bucket.<\/p>\n

    \"\"<\/a><\/p>\n

    After the Databricks cluster is up and running with required access to Amazon S3, you can fetch the JDBC URL<\/code> from your Databricks cluster to be used by Data Wrangler to connect to it.<\/p>\n

    Fetch the JDBC URL<\/strong><\/h3>\n

    To fetch the JDBC URL, complete the following steps:<\/p>\n

      \n
    1. In Databricks, navigate to the clusters UI.<\/li>\n
    2. Choose your cluster.<\/li>\n
    3. On the Configuration<\/strong> tab, choose Advanced options<\/strong>.<\/li>\n
    4. Under Advanced options<\/strong>, choose the JDBC\/ODBC<\/strong> tab.<\/li>\n
    5. Copy the JDBC URL.
      \"\"<\/a><\/li>\n<\/ol>\n

      Make sure to substitute your personal access token<\/a> in the URL.<\/p>\n

      Data Wrangler setup<\/strong><\/h3>\n

      This step assumes you have access to Amazon SageMaker, an instance of Amazon SageMaker Studio<\/a>, and a Studio user.<\/p>\n

      To allow access to the Databricks JDBC connection from Data Wrangler, the Studio user requires following permission:<\/p>\n