{"id":1093,"date":"2021-10-28T08:40:18","date_gmt":"2021-10-28T08:40:18","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/10\/28\/enhance-your-machine-learning-development-by-using-a-modular-architecture-with-amazon-sagemaker-projects\/"},"modified":"2021-10-28T08:40:18","modified_gmt":"2021-10-28T08:40:18","slug":"enhance-your-machine-learning-development-by-using-a-modular-architecture-with-amazon-sagemaker-projects","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/10\/28\/enhance-your-machine-learning-development-by-using-a-modular-architecture-with-amazon-sagemaker-projects\/","title":{"rendered":"Enhance your machine learning development by using a modular architecture with Amazon SageMaker projects"},"content":{"rendered":"
\n

One of the main challenges in a machine learning (ML) project implementation is the variety and high number of development artifacts and tools used. This includes code in notebooks, modules for data processing and transformation, environment configuration, inference pipeline, and orchestration code. In production workloads, the ML model created within your development framework is almost never the end of the work, but is a part of a larger application or workflow.<\/p>\n

Another challenge is the varied nature of ML development activities performed by different user roles. For example, the DevOps engineer develops infrastructure components, such as CI\/CD automation, builds production inference pipelines, and configures security and networking. The data engineer is typically focused on data processing and transformation workflows. The data scientist or ML engineer delivers ML models and model building, training, and validation pipelines.<\/p>\n

These challenges call for an architecture and framework that facilitate separation of concerns by allowing each development role to work on their own part of the system, and hide the complexity of integration, security, and environment configuration.<\/p>\n

This post illustrates how to introduce a modular component-based architecture in your ML application by implementing reusable, self-contained, and consistent components with Amazon SageMaker<\/a>.<\/p>\n

Solution overview<\/h2>\n

As an example of an ML workflow that spans several development domains, the proposed solution implements a use case of an automated pipeline for data transformation, feature extraction, and ingestion into Amazon SageMaker Feature Store<\/a>.<\/p>\n

On a high level, the workflow comprises the following functional steps:<\/p>\n

    \n
  1. An upstream data ingestion component uploads data objects to an Amazon Simple Storage Service<\/a> (Amazon S3) bucket.<\/li>\n
  2. The data upload event launches a data processing and transformation process.<\/li>\n
  3. The data transformation process extracts, processes, and transforms features, and ingests them into a designated feature group<\/a> in Feature Store.
    \"\"<\/a><\/li>\n<\/ol>\n

    Terminology<\/h2>\n

    This section introduces the following important concepts and definitions.<\/p>\n

    ML component<\/h3>\n

    An ML component<\/em> is a construction unit that contains all the required resources, configuration, and workflows to perform a specific ML task. For example, the proposed data transformation and ingestion pipeline can be delivered as an ML component. ML components have a better integration capability to help you to implement reproducible, governed, and secure ML applications. An ML component can encapsulate all the boilerplate code required to properly set up data access permissions, security keys, tagging, naming, and logging requirements for all resources.<\/p>\n

    A process of implementing an ML component assumes that a dedicated DevOps or MLOps team performs the design, building, testing, and distribution of components. The recipients of ML components are data scientists, data engineers, and ML engineers.<\/p>\n

    This separation of development responsibilities brings higher agility, a faster time to market, and less manual heavy lifting, and results in a higher quality and consistency of your ML workflows.<\/p>\n

    Amazon SageMaker project<\/h3>\n

    SageMaker facilitates the development and distribution of ML components with SageMaker projects<\/a>.<\/p>\n

    A SageMaker project<\/em> is a self-sufficient collection of resources, which can be instantiated and used by the entitled users. A project contains all the resources, artifacts, source code, orchestration, and permissions that are needed to perform a designated ML task or workflow. For example, SageMaker provides MLOps project templates<\/a> to automate setup and implementation of MLOps for your applications.<\/p>\n

    You can implement a custom SageMaker project template<\/a> to deliver a packaged ML workflow, which can be distributed and provisioned via an Amazon SageMaker Studio<\/a> IDE.<\/p>\n

    When you implement custom reusable components with SageMaker projects, you can separate the development, testing, and distribution process for ML components from their employment, and follow MLOps best practices.<\/p>\n

    Product portfolio<\/h3>\n

    A project works together with two other AWS services, AWS Service Catalog<\/a> and AWS CloudFormation<\/a>, to provide an end-to-end, user-friendly integration in your SageMaker environment and Studio. You can combine multiple projects in a portfolio<\/em>. A SageMaker project is called product<\/em> in the portfolio scope. A product portfolio is delivered via AWS Service Catalog into Studio. You can control who can view and provision specific products by associating user roles with a designated portfolio.<\/p>\n

    Solution architecture<\/h2>\n

    The detailed component architecture of the solution is presented in the following diagram.
    \"\"<\/a><\/p>\n

    A product portfolio (1) defines the automated Feature Store data ingestion product (2) together with the associated user roles that are allowed to use the portfolio and the containing products. CloudFormation templates define both the product portfolio (1) and the product (2). A CloudFormation template (3) contains all the resources, source code, configuration, and permissions that are needed to provision the product in your SageMaker environment.<\/p>\n

    When AWS CloudFormation deploys the product, it creates a new SageMaker project (4).<\/p>\n

    The SageMaker project implements the feature ingestion workflow (5). The workflow contains an AWS Lambda<\/a> function, which is launched by an Amazon EventBridge <\/a>rule each time new objects are uploaded into a monitored S3 bucket. The Lambda function starts a SageMaker pipeline<\/a> (6), which is defined and provisioned as a part of the SageMaker project. The pipeline implements data transformation and ingestion in Feature Store.<\/p>\n

    The project also provisions CI\/CD automation (7) with an AWS CodeCommit<\/a> repository with source code, AWS CodeBuild<\/a> with a pipeline build script, and AWS CodePipeline<\/a> to orchestrate the build and deployment of the SageMaker pipeline (6).<\/p>\n

    ML pipeline<\/h3>\n

    This solution implements an ML pipeline by using Amazon SageMaker Pipelines<\/a>, an ML workflow creation and orchestration framework. The pipeline contains a single step with an Amazon SageMaker Data Wrangler<\/a> processor for data transformation and ingestion into a feature group in Feature Store. The following diagram shows a data processing pipeline implemented by this solution.<\/p>\n

    \"\"<\/a><\/p>\n

    Refer to Build, tune, and deploy an end-to-end churn prediction model using Amazon SageMaker Pipelines<\/a> for an example of how to build and use a SageMaker pipeline.<\/p>\n

    The rest of this post walks you through the implementation of a custom SageMaker project. We discuss how to do the following:<\/p>\n