{"id":652,"date":"2020-12-02T17:45:59","date_gmt":"2020-12-02T17:45:59","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/12\/02\/private-package-installation-in-amazon-sagemaker-running-in-internet-free-mode\/"},"modified":"2020-12-02T17:45:59","modified_gmt":"2020-12-02T17:45:59","slug":"private-package-installation-in-amazon-sagemaker-running-in-internet-free-mode","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/12\/02\/private-package-installation-in-amazon-sagemaker-running-in-internet-free-mode\/","title":{"rendered":"Private package installation in Amazon SageMaker running in internet-free mode"},"content":{"rendered":"
\n

Amazon SageMaker Studio<\/a> notebooks and Amazon SageMaker<\/a> notebook instances are internet-enabled by default. However, many regulated industries, such as financial industries, healthcare, telecommunications, and others, require that network traffic traverses their own Amazon Virtual Private Cloud<\/a> (Amazon VPC) to restrict and control which traffic can go through public internet. Although you can disable direct internet access to Sagemaker Studio notebooks and notebook instances, you need to ensure that your data scientists can still gain access to popular packages. Therefore, you may choose to build your own isolated dev environments that contain your choice of packages and kernels.<\/p>\n

In this post, we learn how to set up such an environment for Amazon SageMaker notebook instances and SageMaker Studio. We also describe how to integrate this environment with AWS CodeArtifact<\/a>, which is a fully managed artifact repository that makes it easy for organizations of any size to securely store, publish, and share software packages used in your software development process.<\/p>\n

Solution overview<\/h2>\n

In this post, we cover the following steps:\u00a0<\/strong><\/p>\n

    \n
  1. Set up the Amazon SageMaker for internet-free mode.<\/li>\n
  2. Set up the Conda repository using Amazon Simple Storage Service<\/a> (Amazon S3). You create a bucket that hosts your Conda channels.<\/li>\n
  3. Set up the Python Package Index (PyPI) repository using CodeArtifact. You create a repository and set up AWS PrivateLink<\/a> endpoints for CodeArtifact.<\/li>\n
  4. Build an isolated dev environment with Amazon SageMaker notebook instances. In this step, you use the lifecycle configuration feature to build a custom Conda environment and configure your PyPI client.<\/li>\n
  5. Install packages in SageMaker Studio notebooks. In this last step, you can create a custom Amazon SageMaker image<\/a> and install the packages through Conda or pip client.<\/li>\n<\/ol>\n

    Setting up Amazon SageMaker for internet-free mode<\/h2>\n

    We assume that you have already set up a VPC that lets you provision a private, isolated section of the AWS Cloud where you can launch AWS resources in a virtual network. You use it to host Amazon SageMaker and other components of your data science environment. For more information about building secure environments or well-architected pillars, see the following whitepaper, Financial Services Industry Lens: AWS Well-Architected Framework<\/a>.<\/p>\n

    Creating an Amazon SageMaker notebook instance<\/h3>\n

    You can disable internet access for Amazon SageMaker notebooks, and also associate them to your secure VPC environment, which allows you to apply network-level control, such as access to resources through security groups, or to control ingress and egress traffic of data.<\/p>\n

      \n
    1. On the Amazon SageMaker console, choose Notebook instances<\/strong> in the navigation pane.<\/li>\n
    2. Choose Create notebook instance<\/strong>.<\/li>\n
    3. For IAM<\/strong> role, choose your role.<\/li>\n
    4. For VPC<\/strong>, choose your VPC.<\/strong>\n<\/li>\n
    5. For Subnet<\/strong>, choose your subnet.<\/li>\n
    6. For Security groups(s)<\/strong>, choose your security group.<\/li>\n
    7. For Direct internet access<\/strong>, select Disable \u2014 use VPC only<\/strong>.<\/li>\n
    8. Choose Create notebook instance<\/strong>.<\/li>\n<\/ol>\n

      \"\"<\/p>\n

        \n
      1. Connect to your notebook instance from your VPC instead of connecting over the public internet.<\/li>\n<\/ol>\n

        Amazon SageMaker notebook instances support VPC interface endpoints<\/a>. When you use a VPC interface endpoint, communication between your VPC and the notebook instance is conducted entirely and securely within the AWS network instead of the public internet. For instructions, see Creating an interface endpoint<\/a>.<\/p>\n

        Setting up SageMaker Studio<\/h3>\n

        Similar to Amazon SageMaker notebook instances, you can launch SageMaker Studio in a VPC<\/a> of your choice, and also disable direct internet access to add an additional layer of security.<\/p>\n

          \n
        1. On the Amazon SageMaker console, choose Amazon SageMaker Studio<\/strong> in the navigation pane.<\/li>\n
        2. Choose Standard setup.<\/strong>\n<\/li>\n
        3. To disable direct internet access, in the Network<\/strong> section, select the VPC only<\/strong> network access type for when you onboard to Studio or call the CreateDomain<\/a> API.<\/li>\n<\/ol>\n

          Doing so prevents Amazon SageMaker from providing internet access to your SageMaker Studio notebooks.<\/p>\n

          \"\"<\/p>\n

            \n
          1. Create interface endpoints (via AWS PrivateLink) to access the following (and other AWS services you may require):\n
              \n
            1. Amazon SageMaker API<\/li>\n
            2. Amazon SageMaker runtime<\/li>\n
            3. Amazon S3<\/li>\n
            4. \nAWS Security Token Service<\/a> (AWS STS)<\/li>\n
            5. Amazon CloudWatch<\/a><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

              Setting up a custom Conda repository using Amazon S3<\/h2>\n

              Amazon SageMaker notebooks come with multiple environments already installed. The different Jupyter kernels in Amazon SageMaker notebooks are separate Conda environments. If you want to use an external library in a specific kernel, you can install the library in the environment for that kernel. This is typically done using conda install<\/code>. When you use a conda<\/code> command to install a package, Conda environment searches a set of default channels, which are usually online or remote channels (URLs) that host the Conda packages. However, because we assume the notebook instances don\u2019t have internet access, we modify those Conda channel paths to a private repository where our packages are stored.<\/p>\n

                \n
              1. Build such custom channel is to create a bucket in Amazon S3.<\/li>\n
              2. Copy the packages into the bucket.<\/li>\n<\/ol>\n

                These packages can be either approved packages among the organization or the custom packages built using conda build<\/code>. These packages need to be indexed periodically or as soon as there is an update. The methods to index packages are out of scope of this post.<\/p>\n

                Because we set up the notebook to not allow direct internet access, the notebook can\u2019t connect to the S3 buckets that contain the channels unless you create a VPC endpoint.<\/p>\n

                  \n
                1. Create an Amazon S3 VPC endpoint to send the traffic through the VPC instead of the public internet.<\/li>\n<\/ol>\n

                  By creating a VPC endpoint, you allow your notebook instance to access the bucket where you stored the channels and its packages.<\/p>\n

                    \n
                  1. We recommend that you also create a custom resource-based bucket policy that allows only requests from your private VPC to access to your S3 buckets. For instructions, see Endpoints for Amazon S3<\/a>.<\/li>\n
                  2. Replace the default channels of the Conda environment in your Amazon SageMaker notebooks with your custom channel (we do that in the next step when we build the isolated dev environment):\n
                    \n
                    # remove default channel from the .condarc\r\nconda config --remove channels 'defaults'\r\n# add the conda channels to the .condarc file\r\nconda config --add channels 's3:\/\/user-conda-repository\/main\/'\r\nconda config --add channels 's3:\/\/user-conda-repository\/condaforge\/'<\/code><\/pre>\n<\/div>\n<\/li>\n<\/ol>\n
                    Setting up a custom PyPI repository using CodeArtifact<\/h2>\n
                    Data scientists typically use package managers such as pip, maven, npm, and others to install packages into their environments. By default, when you use pip to install a package, it downloads the package from the public PyPI repository.<\/a> To secure your environment, you can use private package management tools either on premises, such as Artifactory or Nexus, or on AWS, such as CodeArtifact. This allows you to allow access only to approved packages and perform safety checks. Alternatively, you may choose use a private PyPI mirror set up on Amazon Elastic Container Service<\/a> (Amazon ECS) or AWS Fargate<\/a> to mirror the public PyPI repository in your private environment. For more information on this approach, see Building Secure Environments<\/a>.<\/p>\n
                    If you want to use pip to install Python packages, you can use CodeArtifact to control access to and validate the safety of the Python packages. CodeArtifact is a managed artifact repository service to help developers and organizations securely store and share the software packages used in your development, build, and deployment processes. The CodeArtifact integration with AWS Identity and Access Management<\/a> (IAM), support for AWS CloudTrail<\/a>, and encryption with AWS Key Management Service<\/a> (AWS KMS) gives you visibility and the ability to control who has access to the packages.<\/p>\n
                    You can configure CodeArtifact to fetch software packages from public repositories such as PyPI. PyPI helps you find and install software developed and shared by the Python community. When you pull a package from PyPI, CodeArtifact automatically downloads and stores application dependencies from the public repositories, so recent versions are always available to you.<\/p>\n
                    Creating a repository for PyPI<\/h3>\n
                    You can create a repository using the CodeArtifact console or the AWS Command Line Interface (AWS CLI)<\/a>. Each repository is associated with the AWS account that you use when you create it. The following screenshot shows the view of choosing your AWS account on the CodeArtifact console.<\/p>\n
                    \"\"<\/p>\n
                    A repository can have one or more CodeArtifact repository associated with it as an upstream repository. It can facilitate two needs.<\/p>\n
                    Firstly, it allows a package manager client to access the packages contained in more than one repository using a single URL endpoint.<\/p>\n
                    Secondly, when you create a repository, it doesn\u2019t contain any packages. If an upstream repository has an external connection to a public repository, the repositories that are downstream from it can pull packages from that public repository. For example, the repository my-shared-python-repository<\/code> has an upstream repository named pypi-store<\/a>, which acts as an intermediate repository that connects your repository to an external connection (your PyPI<\/a> repository). In this case, a package manager that is connected to my-shared-python-repository<\/code> can pull packages from the PyPI public repository. The following screenshot shows this package flow.<\/p>\n
                    \"\"<\/p>\n
                    For instructions on creating a CodeArtifact repository, see Software Package Management with AWS CodeArtifact<\/a>.<\/p>\n
                    Because we disable internet access for the Amazon SageMaker notebooks, in the next section, we set up AWS PrivateLink endpoints to make sure all the traffic for installing the package in the notebooks traverses through the VPC.<\/p>\n
                    Setting up AWS PrivateLink endpoints for CodeArtifact<\/h3>\n
                    You can configure CodeArtifact to use an interface VPC endpoint to improve the security of your VPC. When you use an interface VPC endpoint, you don\u2019t need an internet gateway, NAT device, or virtual private gateway. To create VPC endpoints for CodeArtifact, you can use the AWS CLI or Amazon VPC console. For this post, we use the Amazon Elastic Compute Cloud<\/a> (Amazon EC2) create-vpc-endpoint<\/code> AWS CLI command. The following two VPC endpoints are required so that all requests to CodeArtifact are in the AWS network.<\/p>\n
                    The following command creates an endpoint to access CodeArtifact repositories:<\/p>\n
                    \n
                    aws ec2 create-vpc-endpoint --vpc-id vpcid --vpc-endpoint-type Interface \r\n  --service-name com.amazonaws.region.codeartifact.api --subnet-ids subnetid \r\n  --security-group-ids groupid<\/code><\/pre>\n<\/div>\n
                    The following command creates an endpoint to access package managers and build tools:<\/p>\n
                    \n
                    aws ec2 create-vpc-endpoint --vpc-id vpcid --vpc-endpoint-type Interface \r\n  --service-name com.amazonaws.region.codeartifact.repositories --subnet-ids subnetid \r\n  --security-group-ids groupid --private-dns-enabled<\/code><\/pre>\n<\/div>\n
                    CodeArtifact uses Amazon S3 to store package assets. To pull packages from CodeArtifact, you must create a gateway endpoint for Amazon S3. See the following code:<\/p>\n
                    \n
                    aws ec2 create-vpc-endpoint --vpc-id vpcid --service-name com.amazonaws.region.s3 \r\n  --route-table-ids routetableid<\/code><\/pre>\n<\/div>\n
                    Building your dev environment<\/h2>\n
                    Amazon SageMaker periodically updates the Python and dependency versions in the environments installed on the Amazon SageMaker notebook instances (when you stop and start) or in the images launched in SageMaker Studio. This might cause some incompatibility if you have your own managed package repositories and dependencies. You can freeze your dependencies in internet-free mode so that:<\/p>\n