{"id":614,"date":"2020-11-24T21:17:52","date_gmt":"2020-11-24T21:17:52","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/11\/24\/bringing-your-own-r-environment-to-amazon-sagemaker-studio\/"},"modified":"2020-11-24T21:17:52","modified_gmt":"2020-11-24T21:17:52","slug":"bringing-your-own-r-environment-to-amazon-sagemaker-studio","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/11\/24\/bringing-your-own-r-environment-to-amazon-sagemaker-studio\/","title":{"rendered":"Bringing your own R environment to Amazon SageMaker Studio"},"content":{"rendered":"<div id=\"\">\n<p><a href=\"https:\/\/aws.amazon.com\/sagemaker\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon SageMaker Studio<\/a> is the first fully integrated development environment (IDE) for machine learning (ML). With a single click, data scientists and developers can quickly spin up <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/notebooks.html\" target=\"_blank\" rel=\"noopener noreferrer\">SageMaker Studio notebooks<\/a> to explore datasets and build models. <a href=\"https:\/\/aws.amazon.com\/about-aws\/whats-new\/2020\/10\/amazon-sagemaker-studio-notebooks-now-support-custom-images\/\" target=\"_blank\" rel=\"noopener noreferrer\">On October 27, 2020, Amazon released a custom images feature <\/a>that allows you to launch SageMaker Studio notebooks with your own images.<\/p>\n<p>SageMaker Studio notebooks provide a set of built-in <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/notebooks-available-images.html\">images<\/a> for popular data science and ML frameworks and compute options to run notebooks. The built-in SageMaker images contain the <a href=\"https:\/\/sagemaker.readthedocs.io\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon SageMaker Python SDK<\/a> and the latest version of the backend runtime process, also called kernel. With the custom images feature, you can register custom built images and kernels, and make them available to all users sharing a SageMaker Studio domain. You can start by cloning and extending one of the <a href=\"https:\/\/github.com\/aws-samples\/sagemaker-studio-custom-image-samples\" target=\"_blank\" rel=\"noopener noreferrer\">example Docker files<\/a> provided by SageMaker, or <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/using-the-amazon-sagemaker-studio-image-build-cli-to-build-container-images-from-your-studio-notebooks\/\" target=\"_blank\" rel=\"noopener noreferrer\">build your own images<\/a> from scratch.<\/p>\n<p>This post focuses on adding a custom R image to SageMaker Studio so you can build and train your R models with SageMaker. After attaching the custom R image, you can select the image in Studio and use R to access the SDKs using the <a href=\"https:\/\/rstudio.github.io\/reticulate\/\" target=\"_blank\" rel=\"noopener noreferrer\">RStudio reticulate<\/a> package. For more information about R on SageMaker, see <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/coding-with-r-on-amazon-sagemaker-notebook-instances\/\" target=\"_blank\" rel=\"noopener noreferrer\">Coding with R on Amazon SageMaker notebook instances<\/a> and <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/r-guide.html\" target=\"_blank\" rel=\"noopener noreferrer\">R User Guide to Amazon SageMaker<\/a>.<\/p>\n<p>You can create images and image versions and attach image versions to your domain using the <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-attach.html\">S<\/a><a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-attach.html\" target=\"_blank\" rel=\"noopener noreferrer\">ageMaker Studio Control Panel<\/a>, the <a href=\"http:\/\/boto3.amazonaws.com\/v1\/documentation\/api\/latest\/reference\/services\/sagemaker.html\" target=\"_blank\" rel=\"noopener noreferrer\">AWS SDK for Python (Boto3)<\/a>, and the <a href=\"http:\/\/aws.amazon.com\/cli\">AWS Command Line Interface<\/a> (AWS CLI)\u2014for more information about CLI commands, see <a href=\"https:\/\/docs.aws.amazon.com\/cli\/latest\/reference\/sagemaker\/\" target=\"_blank\" rel=\"noopener noreferrer\">AWS CLI Command Reference<\/a>. This post explains both AWS CLI and SageMaker console UI methods to attach and detach images to a SageMaker Studio domain.<\/p>\n<h2>Prerequisites<\/h2>\n<p>Before getting started, you need to meet the following prerequisites:<\/p>\n<ul>\n<li>\n<a href=\"https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/cli-chap-install.html\" target=\"_blank\" rel=\"noopener noreferrer\">Install the AWS CLI on your local machine.<\/a> This post uses <a href=\"https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/install-cliv2.html\" target=\"_blank\" rel=\"noopener noreferrer\">AWS CLI version 2<\/a>. You should make necessary adjustments if you use a different AWS CLI version.<\/li>\n<li>Permissions to access the <a href=\"http:\/\/aws.amazon.com\/ecr\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Elastic Container Registry<\/a> (Amazon ECR). For more information, see <a href=\"https:\/\/docs.aws.amazon.com\/AmazonECR\/latest\/userguide\/ecr_managed_policies.html\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon ECR Managed Policies<\/a>.<\/li>\n<li>Install <a href=\"https:\/\/www.docker.com\/products\/docker-desktop\" target=\"_blank\" rel=\"noopener noreferrer\">Docker on your local machine<\/a>. For more information, see <a href=\"https:\/\/docs.docker.com\/get-started\/\">Orientation and setup<\/a>. This is necessary for building Docker images and pushing them to Amazon ECR.\n         <\/li>\n<li>Dockerfile for the R image. A sample Dockerfile is provided in this post that you can customize for your own specific case, but you can also use your own Dockerfile.\n<ul>\n<li>Building the R image from the Dockerfile installs dependencies that may be licensed under copyleft licenses such as GPLv3. You should review the license terms and make sure they are acceptable for your use case before proceeding to build this image.<\/li>\n<\/ul>\n<\/li>\n<li>An <a href=\"http:\/\/aws.amazon.com\/iam\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Identity and Access Management<\/a> (IAM) role that has the <code>AmazonSageMakerFullAccess<\/code> policy attached. If you have onboarded to SageMaker Studio, you can get the role from the <strong>Studio Summary<\/strong> section on the SageMaker Studio Control Panel.<\/li>\n<li>A SageMaker Studio domain. For instructions, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/gs-studio-onboard.html\" target=\"_blank\" rel=\"noopener noreferrer\">Onboard to Amazon SageMaker Studio<\/a>.<\/li>\n<\/ul>\n<h3>Creating your Dockerfile<\/h3>\n<p>Before attaching your image to Studio, you need to build a Docker image using a <a href=\"https:\/\/docs.docker.com\/engine\/reference\/builder\/\" target=\"_blank\" rel=\"noopener noreferrer\">Dockerfile<\/a>. You can build a customized Dockerfile using base images or other Docker image repositories, such as <a href=\"https:\/\/github.com\/jupyter\/docker-stacks\" target=\"_blank\" rel=\"noopener noreferrer\">Jupyter Docker-stacks repository<\/a>, and use or revise the ones that fit your specific need.<\/p>\n<p>SageMaker maintains a <a href=\"https:\/\/github.com\/aws-samples\/sagemaker-studio-custom-image-samples\" target=\"_blank\" rel=\"noopener noreferrer\">repository of sample Docker images <\/a>that you can use for common use cases (including R, Julia, Scala, and TensorFlow). This repository contains examples of Docker images that are valid custom images for <a href=\"https:\/\/jupyter-kernel-gateway.readthedocs.io\/en\/latest\/\" target=\"_blank\" rel=\"noopener noreferrer\">Jupyter KernelGateway Apps<\/a> in SageMaker Studio. These custom images enable you to bring your own packages, files, and kernels for use within SageMaker Studio.<\/p>\n<p>For more information about the specifications that apply to the container image that is represented by a SageMaker image version, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-specs.html\" target=\"_blank\" rel=\"noopener noreferrer\">Custom SageMaker image specifications<\/a>.<\/p>\n<p>For this post, we use the <a href=\"https:\/\/github.com\/aws-samples\/sagemaker-studio-custom-image-samples\/blob\/main\/examples\/r-image\/Dockerfile\" target=\"_blank\" rel=\"noopener noreferrer\">sample R Dockerfile<\/a>. This Dockerfile takes the base Python 3.6 image and installs R system library prerequisites, conda via Miniconda, and R packages and Python packages that are usable via <code>reticulate<\/code>. You can create a file named <code>Dockerfile<\/code> using the following script and copy it to your installation folder. You can customize this Dockerfile for your specific use case and install additional packages.<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\"># This project is licensed under the terms of the Modified BSD License \r\n# (also known as New or Revised or 3-Clause BSD), as follows:\r\n\r\n#    Copyright (c) 2001-2015, IPython Development Team\r\n#    Copyright (c) 2015-, Jupyter Development Team\r\n\r\n# All rights reserved.\r\n\r\nFROM python:3.6\r\n\r\nARG NB_USER=\"sagemaker-user\"\r\nARG NB_UID=\"1000\"\r\nARG NB_GID=\"100\"\r\n\r\n# Setup the \"sagemaker-user\" user with root privileges.\r\nRUN \r\n    apt-get update &amp;&amp; \r\n    apt-get install -y sudo &amp;&amp; \r\n    useradd -m -s \/bin\/bash -N -u $NB_UID $NB_USER &amp;&amp; \r\n    chmod g+w \/etc\/passwd &amp;&amp; \r\n    echo \"${NB_USER}    ALL=(ALL)    NOPASSWD:    ALL\" &gt;&gt; \/etc\/sudoers &amp;&amp; \r\n    # Prevent apt-get cache from being persisted to this layer.\r\n    rm -rf \/var\/lib\/apt\/lists\/*\r\n\r\nUSER $NB_UID\r\n\r\n# Make the default shell bash (vs \"sh\") for a better Jupyter terminal UX\r\nENV SHELL=\/bin\/bash \r\n    NB_USER=$NB_USER \r\n    NB_UID=$NB_UID \r\n    NB_GID=$NB_GID \r\n    HOME=\/home\/$NB_USER \r\n    MINICONDA_VERSION=4.6.14 \r\n    CONDA_VERSION=4.6.14 \r\n    MINICONDA_MD5=718259965f234088d785cad1fbd7de03 \r\n    CONDA_DIR=\/opt\/conda \r\n    PATH=$CONDA_DIR\/bin:${PATH}\r\n\r\n# Heavily inspired from https:\/\/github.com\/jupyter\/docker-stacks\/blob\/master\/r-notebook\/Dockerfile\r\n\r\nUSER root\r\n\r\n# R system library pre-requisites\r\nRUN apt-get update &amp;&amp; \r\n    apt-get install -y --no-install-recommends \r\n    fonts-dejavu \r\n    unixodbc \r\n    unixodbc-dev \r\n    r-cran-rodbc \r\n    gfortran \r\n    gcc &amp;&amp; \r\n    rm -rf \/var\/lib\/apt\/lists\/* &amp;&amp; \r\n    mkdir -p $CONDA_DIR &amp;&amp; \r\n    chown -R $NB_USER:$NB_GID $CONDA_DIR &amp;&amp; \r\n    # Fix for devtools https:\/\/github.com\/conda-forge\/r-devtools-feedstock\/issues\/4\r\n    ln -s \/bin\/tar \/bin\/gtar\r\n\r\nUSER $NB_UID\r\n\r\nENV PATH=$CONDA_DIR\/bin:${PATH}\r\n\r\n# Install conda via Miniconda\r\nRUN cd \/tmp &amp;&amp; \r\n    curl --silent --show-error --output miniconda-installer.sh https:\/\/repo.anaconda.com\/miniconda\/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh &amp;&amp; \r\n    echo \"${MINICONDA_MD5} *miniconda-installer.sh\" | md5sum -c - &amp;&amp; \r\n    \/bin\/bash miniconda-installer.sh -f -b -p $CONDA_DIR &amp;&amp; \r\n    rm miniconda-installer.sh &amp;&amp; \r\n    conda config --system --prepend channels conda-forge &amp;&amp; \r\n    conda config --system --set auto_update_conda false &amp;&amp; \r\n    conda config --system --set show_channel_urls true &amp;&amp; \r\n    conda install --quiet --yes conda=\"${CONDA_VERSION%.*}.*\" &amp;&amp; \r\n    conda update --all --quiet --yes &amp;&amp; \r\n    conda clean --all -f -y &amp;&amp; \r\n    rm -rf \/home\/$NB_USER\/.cache\/yarn\r\n\r\n\r\n# R packages and Python packages that are usable via \"reticulate\".\r\nRUN conda install --quiet --yes \r\n    'r-base=4.0.0' \r\n    'r-caret=6.*' \r\n    'r-crayon=1.3*' \r\n    'r-devtools=2.3*' \r\n    'r-forecast=8.12*' \r\n    'r-hexbin=1.28*' \r\n    'r-htmltools=0.4*' \r\n    'r-htmlwidgets=1.5*' \r\n    'r-irkernel=1.1*' \r\n    'r-rmarkdown=2.2*' \r\n    'r-rodbc=1.3*' \r\n    'r-rsqlite=2.2*' \r\n    'r-shiny=1.4*' \r\n    'r-tidyverse=1.3*' \r\n    'unixodbc=2.3.*' \r\n    'r-tidymodels=0.1*' \r\n    'r-reticulate=1.*' \r\n    &amp;&amp; \r\n    pip install --quiet --no-cache-dir \r\n    'boto3&gt;1.0&lt;2.0' \r\n    'sagemaker&gt;2.0&lt;3.0' &amp;&amp; \r\n    conda clean --all -f -y\r\n\r\nWORKDIR $HOME\r\nUSER $NB_UID<\/code><\/pre>\n<\/div>\n<h3>Setting up your installation folder<\/h3>\n<p>You need to create a folder on your local machine and add the following files in that folder:<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\">.\r\n\u251c\u2500\u2500 Dockerfile\r\n\u251c\u2500\u2500 app-image-config-input.json\r\n\u251c\u2500\u2500 create-and-attach-image.sh\r\n\u251c\u2500\u2500 create-domain-input.json\r\n\u2514\u2500\u2500 default-user-settings.json<\/code><\/pre>\n<\/div>\n<p>In the following scripts, the <a href=\"https:\/\/docs.aws.amazon.com\/general\/latest\/gr\/aws-arns-and-namespaces.html#arns-syntax\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Resource Names<\/a> (ARNs) should have a format similar to:<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\">arn:partition:service:region:account-id:resource-id\r\narn:partition:service:region:account-id:resource-type\/resource-id\r\narn:partition:service:region:account-id:resource-type:resource-id<\/code><\/pre>\n<\/div>\n<ol>\n<li>\n<code>Dockerfile<\/code> is the Dockerfile that you created in the previous step.<\/li>\n<\/ol>\n<ol start=\"2\">\n<li>Create a file named <code>app-image-config-input.json<\/code> with the following content:\n<div class=\"hide-language\">\n<pre><code class=\"lang-json\">{\r\n    \"AppImageConfigName\": \"custom-r-image-config\",\r\n    \"KernelGatewayImageConfig\": {\r\n        \"KernelSpecs\": [\r\n            {\r\n                \"Name\": \"ir\",\r\n                \"DisplayName\": \"R (Custom R Image)\"\r\n            }\r\n        ],\r\n        \"FileSystemConfig\": {\r\n            \"MountPath\": \"\/home\/sagemaker-user\",\r\n            \"DefaultUid\": 1000,\r\n            \"DefaultGid\": 100\r\n        }\r\n    }<\/code><\/pre>\n<\/div>\n<\/li>\n<\/ol>\n<ol start=\"3\">\n<li>Create a file named <code>default-user-settings.json<\/code> with the following content. If you\u2019re adding multiple custom images, add to the list of <code>CustomImages<\/code>.\n<div class=\"hide-language\">\n<pre><code class=\"lang-json\">{\r\n  \"DefaultUserSettings\": {\r\n    \"KernelGatewayAppSettings\": {\r\n      \"CustomImages\": [\r\n          {\r\n                   \"ImageName\": \"custom-r\",\r\n                   \"AppImageConfigName\": \"custom-r-image-config\"\r\n                }\r\n            ]\r\n        }\r\n    }\r\n}<\/code><\/pre>\n<\/div>\n<\/li>\n<\/ol>\n<ol start=\"4\">\n<li>Create one last file in your installation folder named <code>create-and-attach-image.sh<\/code> using the following bash script. The script runs the following in order:\n<ol type=\"a\">\n<li>Creates a repository named smstudio-custom in Amazon ECR and logs into that repository<\/li>\n<li>Builds an image using the Dockerfile\u00a0and attaches a tag to the image r<\/li>\n<li>Pushes the image to Amazon ECR<\/li>\n<li>Creates an image for SageMaker Studio and attaches the Amazon ECR image to that image<\/li>\n<li>Creates an <code>AppImageConfigfor<\/code> this image using <code><code>app-image-config-input.json<\/code><\/code>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\"># Replace with your AWS account ID and your Region, e.g. us-east-1, us-west-2\r\nACCOUNT_ID = <em><span>&lt;AWS ACCOUNT ID&gt;<\/span><\/em>\r\nREGION = <em><span>&lt;STUDIO DOMAIN REGION&gt;<\/span><\/em>\r\n\r\n# create a repository in ECR, and then login to ECR repository\r\naws --region ${REGION} ecr create-repository --repository-name smstudio-custom\r\naws ecr --region ${REGION} get-login-password | docker login --username AWS \r\n    --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com\/smstudio-custom\r\n\r\n# Build the docker image and push to Amazon ECR (modify image tags and name as required)\r\n$(aws ecr get-login --region ${REGION} --no-include-email)\r\ndocker build . -t smstudio-r -t ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com\/smstudio-custom:r\r\ndocker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com\/smstudio-custom:r\r\n\r\n# Using with SageMaker Studio\r\n## Create SageMaker Image with the image in ECR (modify image name as required)\r\nROLE_ARN = \"<span><em>&lt;YOUR EXECUTION ROLE ARN&gt;<\/em><\/span>\"\r\n\r\naws sagemaker create-image \r\n    --region ${REGION} \r\n    --image-name custom-r \r\n    --role-arn ${ROLE_ARN}\r\n\r\naws sagemaker create-image-version \r\n    --region ${REGION} \r\n    --image-name custom-r \r\n    --base-image ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com\/smstudio-custom:r\r\n\r\n## Create AppImageConfig for this image (modify AppImageConfigName and \r\n## KernelSpecs in app-image-config-input.json as needed)\r\n## note that 'file:\/\/' is required in the file path\r\naws sagemaker create-app-image-config \r\n    --region ${REGION} \r\n    --cli-input-json file:\/\/app-image-config-input.json<\/code><\/pre>\n<\/div>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<h2>Updating an existing SageMaker Studio domain with a custom image<\/h2>\n<p>If you already have a Studio domain, you don\u2019t need to create a new domain, and can easily update your existing domain by attaching the custom image. You can do this either using the <a href=\"https:\/\/docs.aws.amazon.com\/cli\/latest\/reference\/sagemaker\/index.html\" target=\"_blank\" rel=\"noopener noreferrer\">AWS CLI for Amazon SageMaker<\/a> or the <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-attach.html\" target=\"_blank\" rel=\"noopener noreferrer\">SageMaker Studio Control Panel<\/a> (which we discuss in the following sections). Before going to the next steps, make sure your domain is in Ready status, and get your Studio domain ID from the Studio Control Panel. The domain ID should be in <code>d-xxxxxxxx<\/code> format.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-18860\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/1-Studio-status.png\" alt=\"\" width=\"800\" height=\"338\"><\/p>\n<h3>Using the AWS CLI for SageMaker<\/h3>\n<p>In the terminal, navigate to your installation folder and run the following commands. This makes the bash scrip executable:<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-bash\">chmod +x create-and-attach-image.sh<\/code><\/pre>\n<\/div>\n<p>Then execute the following command in terminal:<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-bash\">.\/create-and-attach-image.sh<\/code><\/pre>\n<\/div>\n<p>After you successfully run the bash script, you need update your existing domain by executing the following command in the terminal. Make sure you provide your domain ID and Region.<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\">aws sagemaker update-domain --domain-id &lt;DOMAIN_ID&gt; \r\n    --region &lt;REGION_ID&gt; \r\n    --cli-input-json file:\/\/default-user-settings.json<\/code><\/pre>\n<\/div>\n<p>After executing this command, your domain status shows as <code>Updating<\/code> for a few seconds and then shows as <code>Ready<\/code> again. You can now open Studio.<\/p>\n<p>When in the Studio environment, you can use the Launcher to launch a new activity, and should see the <code>custom-r (latest)<\/code> image listed in the dropdown menu under <strong>Select a SageMaker image to launch your activity<\/strong>.<\/p>\n<h3>Using the SageMaker console<\/h3>\n<p>Alternatively, you can update your domain by <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-attach.html?icmpid=docs_sagemaker_console_studio\" target=\"_blank\" rel=\"noopener noreferrer\">attaching the image via the SageMaker console<\/a>. The image that you created is listed on the <strong>Images<\/strong> page on the console.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-18861\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/2-Images.png\" alt=\"\" width=\"800\" height=\"160\"><\/p>\n<ol>\n<li>To attach this image to your domain, on the <strong>SageMaker Studio Control Panel<\/strong>, under <strong>Custom images attached to domain<\/strong>, choose <strong>Attach image<\/strong>.<\/li>\n<li>For <strong>Image source<\/strong>, choose <strong>Existing image<\/strong>.<\/li>\n<li>Choose an existing image from the list.<\/li>\n<li>Choose a version of the image from the list.<\/li>\n<li>Choose <strong>Next<\/strong>.<\/li>\n<li>Choose the IAM role. For more information, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-create.html\" target=\"_blank\" rel=\"noopener noreferrer\">Create a custom SageMaker image (Console)<\/a>.<\/li>\n<li>Choose <strong>Next<\/strong>.<\/li>\n<li>Under <strong>Studio configuration<\/strong>, enter or change the following settings. For information about getting the kernel information from the image, see <a href=\"https:\/\/github.com\/aws-samples\/sagemaker-studio-custom-image-samples\/blob\/main\/DEVELOPMENT.md\" target=\"_blank\" rel=\"noopener noreferrer\">DEVELOPMENT<\/a> in the SageMaker Studio Custom Image Samples GitHub repo.\n<ol type=\"a\">\n<li>For <strong>EFS mount path<\/strong>, enter the path within the image to mount the user\u2019s <a href=\"https:\/\/aws.amazon.com\/efs\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Elastic File System<\/a> (Amazon EFS) home directory.<\/li>\n<li>For <strong>Kernel name<\/strong>, enter the name of an existing kernel in the image.<\/li>\n<li>(Optional) For <strong>Kernel display name<\/strong>, enter the display name for the kernel.<\/li>\n<li>Choose <strong>Add kernel<\/strong>.<\/li>\n<li>(Optional) For <strong>Configuration tags<\/strong>, choose <strong>Add new tag<\/strong> and add a configuration tag.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>For more information, see the <strong>Kernel discovery<\/strong> and <strong>User data<\/strong> sections of <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-specs.html\" target=\"_blank\" rel=\"noopener noreferrer\">Custom SageMaker image specifications<\/a>.<\/p>\n<ol start=\"9\">\n<li>Choose <strong>Submit<\/strong>.<\/li>\n<li>Wait for the image version to be attached to the domain.<\/li>\n<\/ol>\n<p>While attaching, your domain status is in <code>Updating<\/code><strong>. <\/strong>When attached, the version is displayed in the <strong>Custom images<\/strong> list and briefly highlighted, and your domain status shows as <code>Ready<\/code><strong>.<\/strong><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-18862\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/3-Attached-Image.png\" alt=\"\" width=\"800\" height=\"329\"><\/p>\n<p>The SageMaker image store automatically versions your images. You can select a pre-attached image and choose <strong>Detach <\/strong>to detach the image and all versions, or choose <strong>Attach image <\/strong>to attach a new version. There is no limit to the number of versions per image or the ability to detach images.<\/p>\n<h2>Using a custom image to create notebooks<\/h2>\n<p>When you\u2019re done updating your Studio domain with the custom image, you can use that image to create new notebooks. To do so, choose your custom image from the list of images in the Launcher. In this example, we use <code>custom-r<\/code><strong>. <\/strong>This shows the list of kernels that you can use to create notebooks. Create a new notebook with the R kernel.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-18863\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/4-R-kernel.png\" alt=\"\" width=\"800\" height=\"374\"><\/p>\n<p>If this is the first time you\u2019re using this kernel to create a notebook, it may take about a minute to start the kernel, and the <code>Kernel Starting<\/code> message appears on the lower left corner of your Studio. You can write R scripts while the kernel is starting but can only run your script after your kernel is ready. The notebook is created with a <code>default ml.t3.medium<\/code> instance attached to it. You can see <strong>R (Custom R Image) kernel <\/strong>and the instance type on the upper right corner of the notebook<strong>.<\/strong> You can <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/learn-how-to-select-ml-instances-on-the-fly-in-amazon-sagemaker-studio\/\" target=\"_blank\" rel=\"noopener noreferrer\">change ML instances on the fly in SageMaker Studio<\/a>. You can also right-size your instances for different workloads. For more information, see <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/right-sizing-resources-and-avoiding-unnecessary-costs-in-amazon-sagemaker\/\" target=\"_blank\" rel=\"noopener noreferrer\">Right-sizing resources and avoiding unnecessary costs in Amazon SageMaker<\/a>.<\/p>\n<p>To test the kernel, enter the following sample R script in the first cell and run the script. This script tests multiple aspects, including importing libraries, creating a SageMaker session, getting the IAM role, and importing data from public repositories.<\/p>\n<p>The abalone dataset in this post is from <em>Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http:\/\/archive.ics.uci.edu\/ml]. Irvine, CA: University of California, School of Information and Computer Science <\/em>(<a href=\"http:\/\/archive.ics.uci.edu\/ml\/datasets\/Abalone\" target=\"_blank\" rel=\"noopener noreferrer\">http:\/\/archive.ics.uci.edu\/ml\/datasets\/Abalone<\/a>)<em>.<\/em><\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\"># Simple script to test R Kernel in SageMaker Studio\r\n\r\n# Import reticulate, readr and sagemaker libraries\r\nlibrary(reticulate)\r\nlibrary(readr)\r\nsagemaker &lt;- import('sagemaker')\r\n\r\n# Create a sagemaker session\r\nsession &lt;- sagemaker$Session()\r\n\r\n# Get execution role\r\nrole_arn &lt;- sagemaker$get_execution_role()\r\n\r\n# Read a csv file from UCI public repository\r\n# Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http:\/\/archive.ics.uci.edu\/ml]. \r\n# Irvine, CA: University of California, School of Information and Computer Science\r\ndata_file &lt;- 'http:\/\/archive.ics.uci.edu\/ml\/machine-learning-databases\/abalone\/abalone.data'\r\n\r\n# Copy data to a dataframe, rename columns, and show dataframe head\r\nabalone &lt;- read_csv(file = data_file, col_names = FALSE, col_types = cols())\r\nnames(abalone) &lt;- c('sex', 'length', 'diameter', 'height', 'whole_weight', 'shucked_weight', 'viscera_weight', 'shell_weight', 'rings')\r\nhead(abalone)<\/code><\/pre>\n<\/div>\n<p>If the image is set up properly and the kernel is running, the output should look like the following screenshot.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-18864\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/5-R-notebook.png\" alt=\"\" width=\"800\" height=\"474\"><\/p>\n<h2>Listing, detaching, and deleting custom images<\/h2>\n<p>If you want to see the list of custom images attached to your Studio, you can either use the AWS CLI or go to SageMaker console to <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-sdk-view.html\" target=\"_blank\" rel=\"noopener noreferrer\">view the attached image in the Studio Control Panel<\/a>.<\/p>\n<h3>Using the AWS CLI for SageMaker<\/h3>\n<p>To view your list of custom images via the AWS CLI, enter the following command in the terminal (provide the Region in which you created your domain):<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\">aws sagemaker list-images --region <span><em>&lt;region-id&gt; <\/em><\/span><\/code><\/pre>\n<\/div>\n<p>The response includes the details for the attached custom images:<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-json\">{\r\n    \"Images\": [\r\n        {\r\n            \"CreationTime\": \"xxxxxxxxxxxx\",\r\n            \"ImageArn\": \"arn:aws:sagemaker:us-east-2:XXXXXXX:image\/custom-r\",\r\n            \"ImageName\": \"custom-r\",\r\n            \"ImageStatus\": \"CREATED\",\r\n            \"LastModifiedTime\": \"xxxxxxxxxxxxxx\"\r\n        },\r\n        ....\r\n    ]\r\n}<\/code><\/pre>\n<\/div>\n<p>If you want to detach or delete an attached image, you can do it on the SageMaker Studio Control Panel (see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-attach.html\" target=\"_blank\" rel=\"noopener noreferrer\">Detach a custom SageMaker image<\/a>). Alternatively, use the custom image name from your <code>default-user-settings.json<\/code> file and rerun the following command to update the domain by detaching the image:<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\">aws sagemaker update-domain --domain-id <span><em>&lt;YOUR DOMAIN ID&gt;<\/em><\/span> \r\n    --cli-input-json file:\/\/default-user-settings.json<\/code><\/pre>\n<\/div>\n<p>Then, delete the app image config:<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\">aws sagemaker delete-app-image-config \r\n    --app-image-config-name custom-r-image-config<\/code><\/pre>\n<\/div>\n<p>Delete the SageMaker image, which also deletes all image versions. The container images in Amazon ECR that are represented by the image versions are not deleted.<\/p>\n<div class=\"hide-language\">\n<pre><code class=\"lang-python\">aws sagemaker delete-image \r\n    --region &lt;region-id&gt; \r\n    --image-name custom-r<\/code><\/pre>\n<\/div>\n<p>After deleting the image, it will not be listed under custom images in SageMaker Studio. For more information, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/studio-byoi-sdk-cleanup.html\" target=\"_blank\" rel=\"noopener noreferrer\">Clean up resources<\/a>.<\/p>\n<h3>Using the SageMaker console<\/h3>\n<p>You can also detach (and delete) images from your domain via the Studio Control Panel UI. To do so, under <strong>Custom images attached to domain<\/strong>, select the image and choose <strong>Detach. <\/strong>You have the option to also delete all versions of the image from your domain. This detaches the image from the domain.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-18868\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/Missing-1-1.jpg\" alt=\"\" width=\"800\" height=\"414\"><\/p>\n<h2>Getting logs in Amazon CloudWatch<\/h2>\n<p>You can also get access to SageMaker Studio logs in <a href=\"https:\/\/aws.amazon.com\/cloudwatch\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon CloudWatch<\/a>, which you can use for troubleshooting your environment. The metrics are captured under the <code>\/aws\/sagemaker\/studio<\/code> namespace.<\/p>\n<p>To access the logs, on the CloudWatch console, choose <strong>CloudWatch Logs<\/strong>. On the <strong>Log groups<\/strong> page, enter the namespace to see logs associated with the Jupyter server and the kernel gateway.<\/p>\n<p>For more information, see <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/logging-cloudwatch.html\" target=\"_blank\" rel=\"noopener noreferrer\">Log Amazon SageMaker Events with Amazon CloudWatch<\/a>.<\/p>\n<h2>Conclusion<\/h2>\n<p>This post outlined the process of attaching a custom Docker image to your Studio domain to extend Studio\u2019s built-in images. We discussed how you can update an existing domain with a custom image using either the AWS CLI for SageMaker or the SageMaker console. We also explained how you can use the custom image to create notebooks with custom kernels.<\/p>\n<p>For more information, see the following resources:<\/p>\n<hr>\n<h3>About the Authors<\/h3>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-18872 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/Minaie.png\" alt=\"\" width=\"100\" height=\"100\"><strong>Nick Minaie<\/strong> is an Artificial Intelligence and Machine Learning (AI\/ML) Specialist Solution Architect, helping customers on their journey to well-architected machine learning solutions at scale. In his spare time, Nick enjoys family time, abstract painting, and exploring nature.<\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-18873 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/11\/24\/Sam-Liu.jpg\" alt=\"\" width=\"100\" height=\"122\"><strong>Sam Liu<\/strong> is a product manager at Amazon Web Services (AWS). His current focus is the infrastructure and tooling of machine learning and artificial intelligence. Beyond that, he has 10 years of experience building machine learning applications in various industries. In his spare time, he enjoys making short videos for technical education or animal protection.<\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/aws.amazon.com\/blogs\/machine-learning\/bringing-your-own-r-environment-to-amazon-sagemaker-studio\/<\/p>\n","protected":false},"author":0,"featured_media":615,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/614"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=614"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/614\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/615"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=614"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=614"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=614"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}