{"id":632,"date":"2020-11-30T13:21:57","date_gmt":"2020-11-30T13:21:57","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/11\/30\/creating-amazon-sagemaker-studio-domains-and-user-profiles-using-aws-cloudformation\/"},"modified":"2020-11-30T13:21:57","modified_gmt":"2020-11-30T13:21:57","slug":"creating-amazon-sagemaker-studio-domains-and-user-profiles-using-aws-cloudformation","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/11\/30\/creating-amazon-sagemaker-studio-domains-and-user-profiles-using-aws-cloudformation\/","title":{"rendered":"Creating Amazon SageMaker Studio domains and user profiles using AWS CloudFormation"},"content":{"rendered":"
\n

Amazon SageMaker Studio<\/a> is the first fully integrated development environment (IDE) for machine learning (ML). It provides a single, web-based visual interface where you can perform all ML development steps required to build, train, tune, debug, deploy, and monitor models. In this post, we demonstrate how you can create a SageMaker Studio domain and user profile using AWS CloudFormation<\/a>. AWS CloudFormation gives you an easy way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycle by treating infrastructure as code.<\/p>\n

Because AWS CloudFormation isn\u2019t natively integrated with SageMaker Studio at the time of this writing, we use AWS CloudFormation to provision two AWS Lambda<\/a> functions and then invoke these functions to create, delete, and update the Studio domain and user profile. In the rest of this post, we walk through the Lambda function to create the Studio domain (the code for creating a Studio user profile works similarly) and then the CloudFormation template. All the code is available in the GitHub repo<\/a>.<\/p>\n

Lambda function for creating, deleting, and updating a Studio domain<\/h2>\n

In the Lambda function, the lambda_handler<\/code> calls one of the three functions, handle_create, handle_update<\/code>, and handle_delete<\/code>, to create, update, and delete the Studio domain, respectively. Because we invoke this function using an AWS CloudFormation custom resource<\/a>, the custom resource request type<\/a> is sent in the RequestType<\/code> field from AWS CloudFormation. RequestType<\/code> determines which function to call inside the lambda_handler<\/code> function. For example, when AWS CloudFormation detects any changes in the custom::StudioDomain<\/code> section of our CloudFormation template, the RequestType<\/code> is set to Update by AWS CloudFormation, and the handle_update<\/code> function is called. The following is the lambda_handler<\/code> code:<\/p>\n

\n
def lambda_handler(event, context):\r\n    try:\r\n        if event['RequestType'] == 'Create':\r\n            handle_create(event, context)\r\n        elif event['RequestType'] == 'Update':\r\n            handle_update(event, context)\r\n        elif event['RequestType'] == 'Delete':\r\n            handle_delete(event, context)\r\n    except ClientError as exception:\r\n        logging.error(exception)\r\n        cfnresponse.send(event, context, cfnresponse.FAILED,\r\n                         {}, error=str(exception))<\/code><\/pre>\n<\/div>\n
The three functions for creating, updating, and deleting the domain work similarly. For this post, we walk through the code responsible for creating a domain. When invoking the Lambda function through an AWS CloudFormation custom resource<\/a>, we pass key parameters that help define our Studio domain via the custom resource Properties<\/code>. We extract these parameters from the AWS CloudFormation event source in the Lambda function. In the handle_create<\/code> function, parameters are read in from the event and passed on to the create_studio_domain<\/code> function. See the following code for handle_create<\/code>:<\/p>\n
\n
def handle_create(event, context):\r\n    print(\"**Starting running the SageMaker workshop setup code\")\r\n    resource_config = event['ResourceProperties']\r\n    print(\"**Creating studio domain\")\r\n    response_data = create_studio_domain(resource_config)\r\n    cfnresponse.send(event, context, cfnresponse.SUCCESS,\r\n                     {}, physicalResourceId=response_data['DomainArn'])<\/code><\/pre>\n<\/div>\n
We use a boto3 SageMaker client to create Studio domains. For this post, we set the domain name, the VPC and subnet that Studio uses, and the SageMaker execution role for the Studio domain. After the create_domain<\/code> API is made, we check the creation status every 5 seconds. When creation is complete, we return the Amazon Resource Name (ARN) and the URL of the created domain. The amount of time that Lambda allows a function to run before stopping it is 3 seconds by default. Therefore, make sure that the timeout limit of your Lambda function is set appropriately. We set the timeout limit to 900 seconds. The following is the create_studio_domain<\/code> code (the functions for deleting and updating domains are also implemented using boto3 and constructed in a similar fashion):<\/p>\n
\n
client = boto3.client('sagemaker')\r\ndef create_studio_domain(config):\r\n    vpc_id = config['VPC']\r\n    subnet_ids = config['SubnetIds']\r\n    default_user_settings = config['DefaultUserSettings']\r\n    domain_name = config['DomainName']\r\n\r\n    response = client.create_domain(\r\n        DomainName=domain_name,\r\n        AuthMode='IAM',\r\n        DefaultUserSettings=default_user_settings,\r\n        SubnetIds=subnet_ids.split(','),\r\n        VpcId=vpc_id\r\n    )\r\n\r\n    domain_id = response['DomainArn'].split('\/')[-1]\r\n    created = False\r\n    while not created:\r\n        response = client.describe_domain(DomainId=domain_id)\r\n        time.sleep(5)\r\n        if response['Status'] == 'InService':\r\n            created = True\r\n\r\n    logging.info(\"**SageMaker domain created successfully: %s\", domain_id)\r\n    return response<\/code><\/pre>\n<\/div>\n
Finally, we zip the Python script, save it as domain_function.zip<\/code>, and upload it to Amazon Simple Storage Service<\/a> (Amazon S3).<\/p>\n
The Lambda function used for creating a user profile is constructed similarly. For more information, see the UserProfile_function.py<\/code> script in the GitHub repo<\/a>.<\/p>\n
CloudFormation template<\/h2>\n
In the CloudFormation template, we create an execution role for Lambda, an execution role for SageMaker Studio, and the Lambda function using the code explained in the previous section. We invoke this function by specifying it as the target of a customer resource. For more information about invoking a Lambda function with AWS CloudFormation, see Using AWS Lambda with AWS CloudFormation<\/a>.<\/p>\n
Lambda execution role<\/h3>\n
This role gives our Lambda function the permission to create an Amazon CloudWatch Logs<\/a> stream and write logs to CloudWatch. Because we create, delete, and update Studio domains in our function, we also grant this role the permission to do so. See the following code:<\/p>\n
\n
LambdaExecutionRole:\r\n    Type: \"AWS::IAM::Role\"\r\n    Properties:\r\n      AssumeRolePolicyDocument:\r\n        Version: 2012-10-17\r\n        Statement:\r\n          - Effect: Allow\r\n            Principal:\r\n              Service:\r\n                - lambda.amazonaws.com\r\n            Action:\r\n              - \"sts:AssumeRole\"\r\n      Path: \/\r\n\r\n  LambdaExecutionPolicy:\r\n    Type: AWS::IAM::ManagedPolicy\r\n    Properties:\r\n      Path: \/\r\n      PolicyDocument:\r\n        Version: 2012-10-17\r\n        Statement:\r\n          - Sid: CloudWatchLogsPermissions\r\n            Effect: Allow\r\n            Action:\r\n              - logs:CreateLogGroup\r\n              - logs:CreateLogStream\r\n              - logs:PutLogEvents\r\n            Resource: !Sub \"arn:${AWS::Partition}:logs:*:*:*\"\r\n          - Sid: SageMakerDomainPermission\r\n            Effect: Allow\r\n            Action:\r\n              - sagemaker:CreateDomain\r\n              - sagemaker:DescribeDomain\r\n              - sagemaker:DeleteDomain\r\n              - sagemaker:UpdateDomain\r\n              - sagemaker:CreateUserProfile\r\n              - sagemaker:UpdateUserProfile\r\n              - sagemaker:DeleteUserProfile\r\n              - sagemaker:DescribeUserProfile\r\n            Resource:\r\n              - !Sub \"arn:${AWS::Partition}:sagemaker:*:*:domain\/*\"\r\n              - !Sub \"arn:${AWS::Partition}:sagemaker:*:*:user-profile\/*\"\r\n          - Sid: SageMakerExecPassRole\r\n            Effect: Allow\r\n            Action:\r\n              - iam:PassRole\r\n            Resource: !GetAtt  SageMakerExecutionRole.Arn\r\n      Roles:\r\n        - !Ref LambdaExecutionRole<\/code><\/pre>\n<\/div>\n
SageMaker execution role<\/h3>\n
The following SageMaker execution role is attached to Studio (for demonstration purposes, we grant this role SageMakerFullAccess<\/a>):<\/p>\n
\n
SageMakerExecutionRole:\r\n    Type: \"AWS::IAM::Role\"\r\n    Properties:\r\n      AssumeRolePolicyDocument:\r\n        Version: 2012-10-17\r\n        Statement:\r\n          - Effect: Allow\r\n            Principal:\r\n              Service:\r\n                - sagemaker.amazonaws.com\r\n            Action:\r\n              - \"sts:AssumeRole\"\r\n      Path: \/\r\n      ManagedPolicyArns:\r\n        - arn:aws:iam::aws:policy\/AmazonSageMakerFullAccess<\/code><\/pre>\n<\/div>\n
Lambda function<\/h3>\n
The AWS::Lambda::Function<\/code> resource creates a Lambda function. To create a function, we need a deployment package<\/a> and an execution role<\/a>. The deployment package contains our function code (function.zip<\/code>). The execution role, which is the LambdaExecutionRole<\/code> created from previous step, grants the function permission to create a Lambda function. We also added the CfnResponseLayer<\/code> to our function\u2019s execution environment. CfnResponseLayer<\/code> enables the function to interact with an AWS CloudFormation custom resource. It contains a send method to send responses from Lambda to AWS CloudFormation. See the following code:<\/p>\n
\n
Resources:\r\n...\r\n StudioDomainFunction:\r\n    Type: AWS::Lambda::Function\r\n    Properties:\r\n      Handler: lambda_function.lambda_handler\r\n      Role: !GetAtt LambdaExecutionRole.Arn\r\n      Code:\r\n        S3Bucket: !Ref S3Bucket\r\n        S3Key: function.zip\r\n        S3ObjectVersion: !Ref S3ObjectVersion\r\n      Runtime: python3.8\r\n      Timeout: 900\r\n      Layers:\r\n        - !Ref CfnResponseLayer\r\n       \r\n  CfnResponseLayer:\r\n    Type: AWS::Lambda::LayerVersion\r\n    Properties:\r\n      CompatibleRuntimes:\r\n        - python3.8\r\n      Content:\r\n        S3Bucket: !Ref S3Bucket\r\n        S3Key: cfnResponse-layer.zip\r\n      Description: cfn-response layer\r\n      LayerName: cfn-response<\/code><\/pre>\n<\/div>\n
Invoking the Lambda function using an AWS CloudFormation custom resource<\/h3>\n
Custom resources provide a way for you to write custom provisioning logic in a CloudFormation template and have AWS CloudFormation run it during a stack operation, such as when you create, update, or delete a stack. For more information, see Custom resources<\/a>. We get the Lambda function\u2019s ARN created from previous step and pass it to AWS CloudFormation as our service token. This allows AWS CloudFormation to invoke the Lambda function. We pass parameters required for creating, updating, and deleting our domain under Properties<\/code>. See the following code:<\/p>\n
\n
StudioDomain:\r\n    Type: Custom::StudioDomain\r\n    Properties:\r\n      ServiceToken: !GetAtt StudioDomainFunction.Arn\r\n      VPC: !Ref VPCId\r\n      SubnetIds: !Ref SubnetIds\r\n      DomainName: \"MyDomainName\"\r\n      DefaultUserSettings:\r\n        ExecutionRole: !GetAtt SageMakerExecutionRole.Arn<\/code><\/pre>\n<\/div>\n
In the same fashion, we invoke the Lambda function for creating a user profile:<\/p>\n
\n
UserProfile:\r\n    Type: Custom::UserProfile\r\n    Properties:\r\n      ServiceToken: !GetAtt UserProfileFunction.Arn\r\n      DomainId: !GetAtt StudioDomain.DomainId\r\n      UserProfileName: !Ref UserProfileName\r\n      UserSettings:\r\n        ExecutionRole: !GetAtt SageMakerExecutionRole.Arn<\/code><\/pre>\n<\/div>\n
Conclusion<\/h2>\n
In this post, we walked through the steps of creating, deleting, and updating SageMaker Studio domains using AWS CloudFormation and Lambda. The sample files are available in the GitHub repo<\/a>. For information about creating Studio domain inside a VPC, see Securing Amazon SageMaker Studio connectivity using a private VPC<\/a>. For more information about SageMaker Studio, see Get Started with Amazon SageMaker Studio<\/a>.<\/p>\n
\n
About the Authors<\/h3>\n
\"\"Qingwei Li<\/strong> is a Machine Learning Specialist at Amazon Web Services. He received his Ph.D. in Operations Research after he broke his advisor\u2019s research grant account and failed to deliver the Nobel Prize he promised. Currently he helps customers in the financial service and insurance industry build machine learning solutions on AWS. In his spare time, he likes reading and teaching.<\/p>\n
\u00a0<\/p>\n
\u00a0<\/p>\n
\"\"Joseph Jegan<\/strong> is a Cloud Application Architect at Amazon Web Services. He helps AWS customers use AWS services to design scalable and secure applications. He has over 20 years of software development\u00a0experience prior to AWS, working on developing e-commerce platform for large retail customers. He is based out of New York metro and enjoys learning emerging cloud native technologies.<\/p>\n
\u00a0<\/p>\n
\u00a0<\/p>\n
\"\"David Ping<\/strong> is a Principal Machine Learning Solutions Architect and Sr. Manager of AI\/ML Solutions Architecture at Amazon Web Services. He helps enterprise customers build and operate machine learning solutions on AWS. In his spare time, David enjoys hiking and reading the latest machine learning articles.<\/p>\n
\u00a0<\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"
https:\/\/aws.amazon.com\/blogs\/machine-learning\/creating-amazon-sagemaker-studio-domains-and-user-profiles-using-aws-cloudformation\/<\/p>\n","protected":false},"author":0,"featured_media":633,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/632"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=632"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/632\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/633"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=632"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=632"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=632"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}