{"id":764,"date":"2021-01-20T03:01:55","date_gmt":"2021-01-20T03:01:55","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/01\/20\/automating-amazon-personalize-solution-using-the-aws-step-functions-data-science-sdk\/"},"modified":"2021-01-20T03:01:55","modified_gmt":"2021-01-20T03:01:55","slug":"automating-amazon-personalize-solution-using-the-aws-step-functions-data-science-sdk","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/01\/20\/automating-amazon-personalize-solution-using-the-aws-step-functions-data-science-sdk\/","title":{"rendered":"Automating Amazon Personalize solution using the AWS Step Functions Data Science SDK"},"content":{"rendered":"
\n

Machine learning (ML)-based recommender systems aren\u2019t a new concept across organizations such as retail, media and entertainment, and education, but developing such a system can be a resource-intensive task\u2014from data labelling, training and inference, to scaling. You also need to apply continuous integration, continuous deployment, and continuous training to your ML model, or MLOps. The MLOps model helps build code and integration across ML tools and frameworks. Moreover, building a recommender system requires managing and orchestrating multiple workflows, for example waiting for your training jobs to finish and trigger deployments.<\/p>\n

In this post, we show you how to use Amazon Personalize<\/a> to create a serverless recommender system for movie recommendations with no ML experience required. Creating an Amazon Personalize solution workflow involves multiple steps, such as preparing and importing the data<\/a>, choosing a recipe<\/a> and creating a solution<\/a> and finally generating recommendations. End-users or your data scientists can orchestrate these steps using AWS Lambda<\/a> functions and AWS Step Functions<\/a>. However, writing JSON code to manage such workflows can be complicated for your data scientists.<\/p>\n

You can orchestrate an Amazon Personalize workflow using the AWS Step Functions Data Science SDK for Python<\/a> using an Amazon SageMaker<\/a> Jupyter notebook. The AWS Step Functions Data Science SDK is an open-source library that allows data scientists to easily create workflows that process and publish ML models using Amazon SageMaker Jupyter notebooks and Step Functions. You can create multi-step ML workflows in Python that orchestrate AWS infrastructure at scale, without having to provision and integrate the AWS services separately.<\/p>\n

Solution and services overview<\/h2>\n

We have orchestrated the Amazon Personalize training workflow steps such as preparing and importing the data<\/a>, choosing a recipe<\/a> and creating a solution<\/a> using AWS Step functions and AWS lambda functions to create a below Amazon Personalize end to end automated workflow:<\/p>\n

\"AWS<\/p>\n

We walk you through the below steps using the accompanying Amazon SageMaker Jupyter notebook<\/a>. In order to create the above Amazon Personalize workflow, we are going to follow below detailed steps:<\/p>\n

    \n
  1. Complete the prerequisites of setting up permissions and preparing the dataset.<\/li>\n
  2. Set up AWS Lambda function and AWS Step function task states.<\/li>\n
  3. Add wait conditions to each AWS Step function task states.<\/li>\n
  4. Add a control flow and link the states.<\/li>\n
  5. Define and run the workflows by combining all the above steps.<\/li>\n
  6. Generate recommendations.<\/li>\n<\/ol>\n

    Prerequisites<\/h2>\n

    Before you build your workflow and generate recommendations, you must set up a notebook instance, AWS Identity and Access Management<\/a> (IAM) roles, and create an Amazon Simple Storage Service<\/a> (Amazon S3) bucket.<\/p>\n

    Creating a notebook instance in Amazon SageMaker<\/h3>\n

    Create an Amazon SageMaker notebook Instance by following instructions here<\/a>. For creating IAM role for your notebook, make sure your notebook IAM role has AmazonS3FullAccess and\u00a0AmazonPersonalizeFullAccess in order to access Amazon S3 and Amazon Personalize APIs through the Sagemaker notebook.<\/p>\n

      \n
    1. When the notebook is active, choose Open Jupyter.<\/li>\n
    2. On the Jupyter dashboard, choose New.<\/li>\n
    3. Choose Terminal.<\/li>\n
    4. In the terminal, enter the following code:\n
      \n
      \n
      cd SageMaker \r\ngit clone https:\/\/github.com\/aws-samples\/personalize-data-science-sdk-workflow.git<\/code><\/pre>\n<\/p><\/div>\n<\/p><\/div>\n<\/li>\n<\/ol>\n
        \n
      1. Open the notebook by choosing Personalize-Stepfunction-Workflow.ipynb in the root folder.<\/li>\n<\/ol>\n
        You\u2019re now ready to run the following steps through the notebook cells. You can find the complete notebook for this solution in the GitHub repo<\/a>.<\/p>\n
        Setting up Step Function Execution role<\/h3>\n
        You need a Step Functions execution role so that you can create and invoke workflows in Step Functions. For instructions, see Create a role for Step Functions<\/a> or go through the steps in the notebook Create an execution role for Step Functions<\/a>.<\/p>\n
        Attach an inline policy<\/a> to the role you created for notebook instance in previous step. Enter the JSON policy<\/strong> by copying the policy from the notebook.<\/p>\n
        Preparing your dataset<\/strong><\/h3>\n
        To prepare your dataset, complete the following steps:<\/p>\n
          \n
        1. Create an S3 bucket to store the training dataset and provide the bucket name and file name as \u2019movie-lens-100k.csv\u2019 in the notebook<\/a> step Setup S3 location and filename. See the following code:\n
          \n
          bucket = \"<SAMPLE BUCKET NAME<\/strong>><\/span><\/em>\" # replace with the name of your S3 bucket\r\nfilename = \"<SAMPLE FILE NAME<\/strong>><\/span><\/em>\" # replace with a name that you want to save the dataset under<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n<\/ol>\n
            \n
          1. Attach the policy to the S3 bucket by running the Attach policy to Amazon S3 bucket step in the notebook.<\/li>\n
          2. Create an IAM role for Amazon Personalize by running the Create Personalize Role step in the notebook<\/a> using Python boto3 create_role API<\/a>.<\/li>\n<\/ol>\n
              \n
            1. To preprocess the dataset and upload to Amazon S3, run the Data-Preparation step of the following notebook cell to download the movie-lens<\/a> dataset and select the movies that have a rating of 2 or above from the dataset:\n
              \n
              !wget -N http:\/\/files.grouplens.org\/datasets\/movielens\/ml-100k.zip\r\n!unzip -o ml-100k.zip\r\ndata = pd.read_csv('.\/ml-100k\/u.data', sep='t', names=['USER_ID', 'ITEM_ID', 'RATING', 'TIMESTAMP'])\r\npd.set_option('display.max_rows', 5)\r\ndata<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n<\/ol>\n
              The following screenshot shows your output.<\/p>\n
              \"The<\/p>\n
                \n
              1. Run the following code to upload your data to the S3 bucket that you created before:\n
                \n
                data = data[data['RATING'] > 2] # keep only movies rated 2 and above\r\ndata2 = data[['USER_ID', 'ITEM_ID', 'TIMESTAMP']] \r\ndata2.to_csv(filename, index=False)\r\n\r\nboto3.Session().resource('s3').Bucket(bucket).Object(filename).upload_file(filename)<\/code><\/pre>\n<\/p><\/div>\n<\/li>\n<\/ol>\n
                Set up AWS Lambda function and AWS Step function task states.<\/h2>\n
                After you complete the prerequisite steps, you would need do to below:<\/p>\n
                  \n
                1. Setup Lambda functions in the AWS Console: You need to setup AWS Lambda function for each Amazon Personalize tasks such as creating a dataset, choosing a recipe and creating a solution using Amazon Personalize AWS SDK for Python (Boto3)<\/a>. With the Python code provided in the Lambda GitHub repo<\/a>. This repository has python code for various lambda functions, you need to copy the code and create lambda functions following the steps in this documentation \u201cbuild a Lambda function on the Lambda console<\/a>.\u201d<\/li>\n<\/ol>\n
                  Note<\/strong>: While creating an execution role, make sure you provide AmazonPersonalizeFullAccess<\/code> along with AWSLambdaBasicExecutionRole<\/code> permission policy to the lambda role.<\/p>\n
                    \n
                  1. Orchestrate each of these lambda function as AWS Step function task states<\/a>: A\u00a0Task\u00a0state represents a single unit of work performed by a state machine. AWS Step Functions can invoke Lambda functions directly from a task state. For more information see Creating a Step Functions State Machine That Uses Lambda<\/a>. Below is a step function workflow diagram to orchestrate the lambda functions we are going to cover in this section:<\/li>\n<\/ol>\n
                    \"Below<\/p>\n
                    We already created Lambda functions for each of the Amazon Personalize steps using the Amazon Personalize AWS SDK for Python (Boto3)<\/a> with the Python code provided in the Lambda GitHub repo<\/a>. We deep dive and explain each step in this section. Alternatively, you can build a Lambda function on the Lambda console<\/a>.<\/p>\n
                    Creating a schema<\/h3>\n
                    Before you add a dataset to Amazon Personalize, you must define a schema for that dataset. For more information, see Datasets and Schemas<\/a>. First we would create a step function task state to create a schema using below code in the walkthrough notebook<\/a>.<\/p>\n
                    \n
                    lambda_state_schema = LambdaStep(\r\n    state_id=\"create schema\",\r\n    parameters={  \r\n        \"FunctionName\": \"stepfunction-create-schema\", #replace with the name of the function you created\r\n        \"Payload\": {  \r\n           \"input\": \"personalize-stepfunction-schema\"\r\n        }\r\n    },\r\n    result_path='$'    \r\n)<\/code><\/pre>\n<\/p><\/div>\n
                    The above step function is invoking the AWS lambda function Lambda function stepfunction-create-schema.py<\/a>. This lambda code snippet above is uses Personalize.create_schema() boto 3 APIs<\/a> to create schema.<\/p>\n
                    \n
                    import boto3\r\n\r\nclient = boto3.client('personalize')\r\ncreate_schema_response = personalize.create_schema(\r\n        name = event['input'],\r\n        schema = json.dumps(schema)\r\n    )<\/code><\/pre>\n<\/p><\/div>\n
                    This API creates an Amazon Personalize schema from the specified schema string. The schema you create must be in Avro JSON format.<\/p>\n
                    Creating a dataset group<\/h3>\n
                    After you create a schema, similarly, we are going to create a step function task states for creating a dataset group .A dataset group contains related datasets that supply data for training a model. A dataset group can contain at most three datasets, one for each type of dataset: Interactions, Items and Users.<\/p>\n
                    To train a model (create a solution), you need a dataset group that contains an Interactions dataset.<\/p>\n
                    Run the notebook steps to create state id \u201ccreate dataset\u201d below:<\/p>\n
                    \n
                    lambda_state_createdataset = LambdaStep(\r\n    state_id=\"create dataset\",\r\n    parameters={  \r\n        \"FunctionName\": \"stepfunctioncreatedataset\", #replace with the name of the function you created\r\n        \r\n        \"Payload\": {  \r\n           \"schemaArn.$\": '$.schemaArn',\r\n           \"datasetGroupArn.$\": '$.datasetGroupArn',        \r\n        } \r\n        \r\n        \r\n    },\r\n    result_path = '$'\r\n)<\/code><\/pre>\n<\/p><\/div>\n
                    The above step function task state is invoking the AWS lambda function stepfunctioncreatedatagroup.py<\/a>. Below is a code snippet of using personalize.create_dataset()<\/a> to create a dataset:<\/p>\n
                    \n
                    create_dataset_response = personalize.create_dataset(\r\n\r\nname = \"personalize-stepfunction-dataset\",\r\n\r\ndatasetType = dataset_type,\r\n\r\ndatasetGroupArn = event['datasetGroupArn'],\r\n\r\nschemaArn = event['schemaArn']\r\n\r\n)<\/code><\/pre>\n<\/p><\/div>\n
                    This API creates an empty dataset and adds it to the specified dataset group.<\/p>\n
                    Creating a dataset<\/h3>\n
                    In this step, you create a step function task state for automating an empty dataset and add it to the specified dataset group.<\/p>\n
                    Run through the notebook steps to create a dataset<\/a> step function. You can review the underlying AWS Lambda function stepfunctioncreatedataset.py<\/a>. We use the same API personalize.create_dataset<\/a> Python boto3 API to automate dataset creation which we used to create a dataset group in above step.<\/p>\n
                    Importing your data<\/h3>\n
                    When you complete Step 1: Creating a Dataset Group<\/a> and Step 2: Creating a Dataset and a Schema<\/a>, you\u2019re ready to import your training data into Amazon Personalize. When you import data, you can choose to import records in bulk, import records individually, or both, depending on your business requirements and the amount of historical data you have collected. If you have a large amount of historical records, we recommend you import data in bulk and add data incrementally as necessary.<\/p>\n
                    Run through the notebook steps<\/a> to create a step functions task state to import a dataset<\/a> with state_id=\u201dcreate dataset import job\u201d. Let\u2019s review the code snippet for underlying Lambda function stepfunction-createdatasetimportjob.py<\/a>.<\/p>\n
                    \n
                    create_dataset_import_job_response = personalize.create_dataset_import_job(\r\n\r\njobName = \"stepfunction-dataset-import-job\",\r\n\r\ndatasetArn = datasetArn,\r\n\r\ndataSource = {\r\n\r\n\"dataLocation\": \"s3:\/\/{}\/{}\".format(bucket, filename)\r\n\r\n},\r\n\r\nroleArn = roleArn\r\n\r\n)<\/code><\/pre>\n<\/p><\/div>\n
                    We are using personalize.create_dataset_import_job()<\/a> boto3 APIs to import the dataset in the above lambda code. This API creates a job that imports training data from your data source (an Amazon S3 bucket) to an Amazon Personalize dataset.<\/p>\n
                    Creating a solution<\/h3>\n
                    After you prepare and import the data<\/a>, you\u2019re ready to create a solution. A solution refers to the combination of an Amazon Personalize recipe, customized parameters, and one or more solution versions (trained models). A recipe<\/em> is an Amazon Personalize term specifying an appropriate algorithm to train for a given use case. After you create a solution with a solution version, you can create a campaign<\/a> to deploy the solution version and get recommendations. We will create a step function task states for choosing a recipe and creating a solution.<\/p>\n
                    Choosing a recipe, configuring a solution and creating a solution version<\/h3>\n
                    Run the notebook cell<\/a> Choose a recipe to create a step function task state to generate state ID state_id=\u201dselect recipe and create solution\u201d. This state is representing the AWS lambda function \u00a0stepfunction_select-recipe_create-solution.py<\/a>. Below is the code snippet:<\/p>\n
                    \n
                    list_recipes_response = personalize.list_recipes()\r\n\r\nrecipe_arn = \"arn:aws:personalize:::recipe\/aws-user-personalization\" # aws-user-personalization selected for demo purposes\r\n\r\n#list_recipes_response\r\n  create_solution_response = personalize.create_solution(\r\n\r\nname = \"stepfunction-solution\",\r\n\r\ndatasetGroupArn = event['dataset_group_arn'],\r\n\r\nrecipeArn = recipe_arn\r\n\r\n)<\/code><\/pre>\n<\/p><\/div>\n
                    We are using personalize.list_recipes()<\/a> boto3 APIs to return a list of available recipes and personalize.craete_solution()<\/a> boto 3 API to create the configuration for training a model. A trained model is known as a solution.<\/p>\n
                    We use the algorithm aws-user-personalization<\/code><\/a> for this post. For more information, see Choosing a Recipe<\/a> and \u00a0Configuring a Solution<\/a>.<\/p>\n
                    After you choose a recipe and configure your solution, you\u2019re ready to create a solution version, which refers to a trained ML model.<\/p>\n
                    Run the notebook cell<\/a> Create Solution Version to create a step function task state with the State ID state_id=\u201d create solution version\u201d and recipes using the underlying Lambda function stepfunction_create_solution_version.py<\/a>. Below is the lambda code snippet using personalize.create_solution_version()<\/a> boto3 API which trains or retrains an active solution.<\/p>\n
                    \n
                    create_solution_version_response = personalize.create_solution_version(\r\n\r\n\r\nsolutionArn = event['solution_arn']\r\n\r\n)<\/code><\/pre>\n<\/p><\/div>\n
                    A solution is created using the CreateSolution operation and must be in the ACTIVE state before calling CreateSolutionVersion . A new version of the solution is created every time you call this operation. For more information, see Creating a Solution Version<\/a>.<\/p>\n
                    Creating a campaign<\/h3>\n
                    A campaign is used to make recommendations for your users. You create a campaign by deploying a solution version.<\/p>\n
                    Run the notebook cell<\/a> Create Campaign to create step function task state with the state ID state_id=\u201d create campaign\u201d using the underlying Lambda function stepfunction_getsolution_metric_create_campaign.py<\/a>. This lambda is using below API<\/p>\n
                    \n
                    get_solution_metrics_response = personalize.get_solution_metrics(\r\n        solutionVersionArn = event['solution_version_arn']\r\n    )\r\n\r\n    create_campaign_response = personalize.create_campaign(\r\n        name = \"stepfunction-campaign\",\r\n        solutionVersionArn = event['solution_version_arn'],\r\n        minProvisionedTPS = 1\r\n    )<\/code><\/pre>\n<\/p><\/div>\n
                    personalize.get_solution_metrics()<\/a> gets the metrics for the specified solution version and personalize.create_campaign()<\/a> creates a campaign by deploying a solution version.<\/p>\n
                    In this section, we covered all the Step function task states and their AWS lambda functions needed to orchestrate the Amazon Personalize workflow. Go to next section to add the wait states to these step function states.<\/p>\n
                    Adding a wait state to your steps<\/h2>\n
                    In this section, we add wait states because these steps need to wait for previous steps to finish before running the next step. For example, you should create a dataset after you create a dataset group.<\/p>\n
                    Run the notebook steps from Wait for Schema to be ready to Wait for Campaign to be ACTIVE to make sure these Step Functions or Lambda steps are waiting for each step to run before they trigger the next step.<\/p>\n
                    The following code is an example of what a wait state looks like to wait for the schema to be created:<\/p>\n
                    \n
                    wait_state_schema = Wait(\r\n    state_id=\"Wait for create schema - 5 secs\",\r\n    seconds=5\r\n)<\/code><\/pre>\n<\/p><\/div>\n
                    We added a wait time of 5 seconds for the state ID, which means it should wait for 5 seconds before going to next state.<\/p>\n
                    Adding a control flow and linking states by using the choice state<\/h2>\n
                    The AWS Step Functions Data Science SDK\u2019s choice state<\/a> supports branching logic based on the outputs from previous steps. You can create dynamic and complex workflows by adding this state.<\/p>\n
                    After you define these steps, chain them together into a logical sequence. Run all the notebook steps to add choices while automating Amazon Personalize datasets, recipes, solutions, and the campaign.<\/p>\n
                    The following code is an example of the choice state for the create_campaign workflow:<\/p>\n
                    \n
                    create_campaign_choice_state = Choice(\r\n    state_id=\"Is the Campaign ready?\"\r\n)\r\ncreate_campaign_choice_state.add_choice(\r\n    rule=ChoiceRule.StringEquals(variable=lambda_state_campaign_status.output()['Payload']['status'], value='ACTIVE'),\r\n    next_step=Succeed(\"CampaignCreatedSuccessfully\")     \r\n)\r\ncreate_campaign_choice_state.add_choice(\r\n    ChoiceRule.StringEquals(variable=lambda_state_campaign_status.output()['Payload']['status'], value='CREATE PENDING'),\r\n    next_step=wait_state_campaign\r\n)\r\ncreate_campaign_choice_state.add_choice(\r\n    ChoiceRule.StringEquals(variable=lambda_state_campaign_status.output()['Payload']['status'], value='CREATE IN_PROGRESS'),\r\n    next_step=wait_state_campaign\r\n)\r\n\r\ncreate_campaign_choice_state.default_choice(next_step=Fail(\"CreateCampaignFailed\"))<\/code><\/pre>\n<\/p><\/div>\n
                    Defining and running workflows<\/h2>\n
                    You create a workflow that runs a group of Lambda functions (steps) in a specific order for example one Lambda function\u2019s output passes to the next Lambda function\u2019s input. We\u2019re now ready to define the workflow definition for each step in Amazon Personalize:<\/p>\n