{"id":1954,"date":"2022-03-10T18:45:55","date_gmt":"2022-03-10T18:45:55","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/10\/load-and-transform-data-from-delta-lake-using-amazon-sagemaker-studio-and-apache-spark\/"},"modified":"2022-03-10T18:45:55","modified_gmt":"2022-03-10T18:45:55","slug":"load-and-transform-data-from-delta-lake-using-amazon-sagemaker-studio-and-apache-spark","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/10\/load-and-transform-data-from-delta-lake-using-amazon-sagemaker-studio-and-apache-spark\/","title":{"rendered":"Load and transform data from Delta Lake using Amazon SageMaker Studio and Apache Spark"},"content":{"rendered":"
\n

Data lakes have become the norm in the industry for storing critical business data. The primary rationale for a data lake is to land all types of data, from raw data to preprocessed and postprocessed data, and may include both structured and unstructured data formats. Having a centralized data store for all types of data allows modern big data applications to load, transform, and process whatever type of data is needed. Benefits include storing data as is without the need to first structure or transform it. Most importantly, data lakes allow controlled access to data from many different types of analytics and machine learning (ML) processes in order to guide better decision-making.<\/p>\n

Multiple vendors have created data lake architectures, including AWS Lake Formation<\/a>. In addition, open-source solutions allow companies to access, load, and share data easily. One of the options for storing data in the AWS Cloud is Delta Lake<\/a>. The Delta Lake library enables reads and writes in open-source Apache Parquet<\/a> file format, and provides capabilities like ACID transactions, scalable metadata handling, and unified streaming and batch data processing. Delta Lake offers a storage layer API that you can use to store data on top of an object-layer storage like Amazon Simple Storage Service<\/a> (Amazon S3).<\/p>\n

Data is at the heart of ML\u2014training a traditional supervised model is impossible without access to high-quality historical data, which is commonly stored in a data lake. Amazon SageMaker<\/a> is a fully managed service that provides a versatile workbench for building ML solutions and provides highly tailored tooling for data ingestion, data processing, model training, and model hosting. Apache Spark<\/a> is a workhorse of modern data processing with an extensive API for loading and manipulating data. SageMaker has the ability to prepare data at petabyte scale using Spark to enable ML workflows that run in a highly distributed manner. This post highlights how you can take advantage of the capabilities offered by Delta Lake using Amazon SageMaker Studio<\/a>.<\/p>\n

Solution overview<\/h2>\n

In this post, we describe how to use SageMaker Studio notebooks to easily load and transform data stored in the Delta Lake format. We use a standard Jupyter notebook to run Apache Spark commands that read and write table data in CSV and Parquet format. The open-source library delta-spark<\/a> allows you to directly access this data in its native format. This library allows you to take advantage of the many API operations to apply data transformations, make schema modifications, and use time-travel or as-of-timestamp queries to pull a particular version of the data.<\/p>\n

In our sample notebook, we load raw data into a Spark DataFrame, create a Delta table, query it, display audit history, demonstrate schema evolution, and show various methods for updating the table data. We use the DataFrame API<\/a> from the PySpark library to ingest and transform the dataset attributes. We use the delta-spark<\/code> library to read and write data in Delta Lake format and to manipulate the underlying table structure, referred to as the schema<\/em>.<\/p>\n

We use SageMaker Studio, the built-in IDE from SageMaker, to create and run Python code from a Jupyter notebook. We have created a GitHub repository<\/a> that contains this notebook<\/a> and other resources to run this sample on your own. The notebook demonstrates exactly how to interact with data stored in Delta Lake format, which permits tables to be accessed in-place without the need to replicate data across disparate datastores.<\/p>\n

For this example, we use a publicly available dataset from Lending Club<\/a> that represents customer loans data. We downloaded the accepted<\/code> data file (accepted_2007_to_2018Q4.csv.gz<\/code>), and selected a subset of the original attributes. This dataset is available under the Creative Commons (CCO) License<\/a>.<\/p>\n

Prerequisites<\/h2>\n

You must install a few prerequisites prior to using the delta-spark<\/code> functionality. To satisfy required dependencies, we have to install some libraries into our Studio environment, which runs as a Dockerized container and is accessed via a Jupyter Gateway app:<\/p>\n