{"id":887,"date":"2021-09-18T20:01:00","date_gmt":"2021-09-18T20:01:00","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/09\/18\/perform-interactive-data-engineering-and-data-science-workflows-from-amazon-sagemaker-studio-notebooks\/"},"modified":"2021-09-18T20:01:00","modified_gmt":"2021-09-18T20:01:00","slug":"perform-interactive-data-engineering-and-data-science-workflows-from-amazon-sagemaker-studio-notebooks","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/09\/18\/perform-interactive-data-engineering-and-data-science-workflows-from-amazon-sagemaker-studio-notebooks\/","title":{"rendered":"Perform interactive data engineering and data science workflows from Amazon SageMaker Studio notebooks"},"content":{"rendered":"
\n

Amazon SageMaker Studio<\/a> is the first fully integrated development environment (IDE) for machine learning (ML). With a single click, data scientists and developers can quickly spin up Studio notebooks to explore and prepare datasets to build, train, and deploy ML models in a single pane of glass.<\/p>\n

We\u2019re excited to announce a new set of capabilities that enable interactive Spark-based data processing from Studio notebooks. Data scientists and data engineers can now visually browse, discover, and connect to Spark data processing environments running on Amazon EMR<\/a>, right from your Studio notebooks in a few simple clicks. After you\u2019re connected, you can interactively query, explore and visualize data, and run Spark jobs to prepare data using the built-in SparkMagic notebook environments for Python and Scala.<\/p>\n

Analyzing, transforming, and preparing large amounts of data is a foundational step of any data science and ML workflow, and businesses are increasingly using Apache Spark for fast data preparation. Studio already offers purpose-built and best-in-class tooling such as Experiments<\/a>, Clarify<\/a>, and Model Monitor<\/a> for ML. The newly launched capability of easily accessing purpose-built Spark environments from Studio notebooks enables Studio to serve as a unified environment for data science and data engineering workflows. In this post, we present an example of predicting the sentiment of a movie review.<\/p>\n

We start with explaining how you can set up connecting Studio securely to an EMR cluster configured with various authentication methods. We provide CloudFormation templates to make it easy for you to deploy resources such as networking, EMR clusters, and Studio with a few simple clicks so that you can follow along with the examples in your own AWS account. We then demonstrate how you can use a Studio notebook to visually discover, authenticate with, and connect to an EMR cluster. After we\u2019re connected, we query a Hive table on Amazon EMR using SparkSQL and PyHive. We then locally preprocess and feature engineer the retrieved data, train an ML model, deploy it, and get predictions\u2014all from the Studio notebook.<\/p>\n

Solution overview<\/h2>\n

Studio runs on an environment managed by AWS. In this solution, the network access for the new Studio domain is configured as VPC Only. For more details on different connectivity methods, see Securing Amazon SageMaker Studio connectivity using a private VPC.<\/a> The elastic network interface created in the private subnet connects to required AWS services through VPC endpoints.<\/p>\n

The following diagram represents the different components used in this solution.<\/p>\n

\"\"<\/p>\n

For connecting to the EMR cluster, we walk through three authentication options. We use a separate AWS CloudFormation<\/a> template stack for each of these authentication scenarios.<\/p>\n

In each of the options, the CloudFormation template also does the following:<\/p>\n