{"id":730,"date":"2021-01-04T23:44:03","date_gmt":"2021-01-04T23:44:03","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2021\/01\/04\/building-a-secure-search-application-with-access-controls-using-amazon-kendra\/"},"modified":"2021-01-04T23:44:03","modified_gmt":"2021-01-04T23:44:03","slug":"building-a-secure-search-application-with-access-controls-using-amazon-kendra","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/01\/04\/building-a-secure-search-application-with-access-controls-using-amazon-kendra\/","title":{"rendered":"Building a secure search application with access controls using Amazon Kendra"},"content":{"rendered":"
\n

For many enterprises, critical business information is often stored as unstructured data scattered across multiple content repositories. Not only is it challenging for organizations to make this information available to employees when they need it, but it\u2019s also difficult to do so securely so relevant information is available to the right employees or employee groups.<\/p>\n

Amazon Kendra<\/a> is a highly accurate and easy-to-use intelligent search service powered by machine learning (ML). Amazon Kendra delivers secure search for enterprise applications and can make sure the results of a user\u2019s search query only include documents the user is authorized to read. In this post, we illustrate how to build an Amazon Kendra-powered search application supporting access controls that reflect the security model of an example organization.<\/p>\n

Amazon Kendra supports search filtering based on user access tokens that are provided by your search application, as well as document access control lists (ACLs) collected by the Amazon Kendra connectors. When user access tokens are applied, search results return links to the original document repositories and include a short description. Access control to the full document is still enforced by the original repository.<\/p>\n

In this post, we demonstrate token-based user access control in Amazon Kendra with Open ID. We use Amazon Cognito<\/a> user pools to authenticate users and provide Open ID tokens. You can use a similar approach with other Open ID providers.<\/p>\n

Application overview<\/h2>\n

This application is designed for guests and registered users to make search queries to a document repository, and results are returned only from those documents that are authorized for access by the user. Users are grouped based on their roles, and access control is at a group level. The following table outlines which documents each user is authorized to access for our use case. The documents being used in this example are a subset of AWS public documents.<\/p>\n\n\n\n\n\n\n\n\n
User<\/strong><\/td>\nRole<\/strong><\/td>\nGroup<\/strong><\/td>\nDocument Type Authorized for Access<\/strong><\/td>\n<\/tr>\n
<\/td>\nGuest<\/td>\n<\/td>\nBlogs<\/td>\n<\/tr>\n
Patricia<\/td>\nIT Architect<\/td>\nCustomer<\/td>\nBlogs, user guides<\/td>\n<\/tr>\n
James<\/td>\nSales Rep<\/td>\nSales<\/td>\nBlogs, user guides, case studies<\/td>\n<\/tr>\n
John<\/td>\nMarketing Exec<\/td>\nMarketing<\/td>\nBlogs, user guides, case studies, analyst reports<\/td>\n<\/tr>\n
Mary<\/td>\nSolutions Architect<\/td>\nSolutions Architect<\/td>\nBlogs, user guides, case studies, analyst reports, whitepapers<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Architecture<\/h2>\n

The following diagram illustrates our solution architecture.<\/p>\n

\"The<\/p>\n

The documents being queried are stored in an Amazon Simple Storage Service<\/a> (Amazon S3) bucket. Each document type has a separate folder: blogs<\/code>, case-studies<\/code>, analyst-reports<\/code>, user-guides<\/code>, and white-papers<\/code>. This folder structure is contained in a folder named Data<\/code>. Metadata files including the ACLs are included in a folder named Meta<\/code>.<\/p>\n

We use the Amazon Kendra S3 connector to configure this S3 bucket as the data source. When the data source is synced with the Amazon Kendra index, it crawls and indexes all documents as well as collects the ACLs and document attributes from the metadata files. For this example, we use a custom attribute DocumentType<\/code> to denote the type of the document.<\/p>\n

We use an Amazon Cognito user pool to authenticate registered users, and use an identity pool to authorize the application to use Amazon Kendra and Amazon S3. The user pool is configured as an Open ID provider in the Amazon Kendra index by configuring the signing URL of the user pool.<\/p>\n

When a registered user authenticates and logs in to the application to perform a query, the application sends the user\u2019s access token provided by the user pool to the Amazon Kendra index as a parameter in the query API call. For guest users, there is no authentication and therefore no access token is sent as a parameter to the query API. The results of a query API call without the access token parameter only return the documents without access control restrictions.<\/p>\n

When an Amazon Kendra index receives a query API call with a user access token, it decrypts the access token using the user pool signing URL and gets parameters such as cognito:username<\/code> and cognito:groups<\/code> associated with the user. The Amazon Kendra index filters the search results based on the stored ACLs and the information received in the user access token. These filtered results are returned in response to the query API call made by the application.<\/p>\n

The application, which the users can download with its source<\/a>, is written in ReactJS using components from the AWS Amplify<\/a> framework. We use the AWS Amplify console to implement the continuous integration and continuous deployment pipelines. We use an AWS CloudFormation<\/a> template to deploy the AWS infrastructure, which includes the following:<\/p>\n

In this post, we provide a step-by-step walkthrough to configure the backend infrastructure, build and deploy the application code, and use the application.<\/p>\n

Prerequisites<\/h2>\n

To complete the steps in this post, make sure you have the following:<\/p>\n

Preparing your S3 bucket as a data source<\/h2>\n

To prepare an S3 bucket as a data source, create an S3 bucket. In the terminal with the AWS CLI or AWS CloudShell<\/a>, run the following commands to upload the documents and the metadata to the data source bucket:<\/p>\n

\n
aws s3 cp s3:\/\/aws-ml-blog\/artifacts\/building-a-secure-search-application-with-access-controls-kendra\/docs.zip .\r\nunzip docs.zip\r\naws s3 cp Data\/ s3:\/\/<REPLACE-WITH-NAME-OF-S3-BUCKET><\/strong>\/Data\/ --recursive\r\naws s3 cp Meta\/ s3:\/\/<REPLACE-WITH-NAME-OF-S3-BUCKET><\/strong>\/Meta\/ --recursive\r\n<\/code><\/pre>\n<\/div>\n
Deploying the infrastructure as a CloudFormation stack<\/h2>\n
In a separate browser tab open the AWS Management Console<\/a>, and make sure that you are logged in to your AWS account. Click the button below to launch the CloudFormation stack to deploy the infrastructure.<\/p>\n
\"\"<\/a><\/p>\n
You should see a page similar to the image below:<\/p>\n
\"\"<\/p>\n
For S3DataSourceBucket<\/strong>, enter your data source bucket name without the s3:\/\/ prefix, select I acknowledge that AWS CloudFormation might create IAM resources with custom names,<\/strong> and then choose Create stack<\/strong>.<\/p>\n
Stack creation can take 30\u201345 minutes to complete. While you wait, you can look at the different tabs, such as Events<\/strong>, Resources<\/strong>, and Template<\/strong>. You can monitor the stack creation status on the Stack info<\/strong> tab.<\/p>\n
\"You<\/p>\n
When stack creation is complete, keep the Outputs<\/strong> tab open. We need values from the Outputs<\/strong> and Resources<\/strong> tabs in subsequent steps.<\/p>\n
Reviewing Amazon Kendra configuration and starting the data source sync<\/h2>\n
In the following steps, we configure Amazon Kendra to enable secure token access and start the data source sync to begin crawling and indexing documents.<\/p>\n
    \n
  1. On the Amazon Kendra console<\/a>, choose the index AuthKendraIndex<\/code>, which was created as part of the CloudFormation<\/code> stack.<\/li>\n<\/ol>\n
    \"On<\/p>\n
    Under User access control<\/strong>, token-based user access control is enabled, the signing key object is set to the Open ID provider URL of the Amazon Cognito user pool, and the user name and group are set to cognito:username<\/code> and cognito:groups<\/code>, respectively.<\/p>\n
    \"Under<\/p>\n
      \n
    1. In the navigation pane, choose Data sources<\/strong>.<\/li>\n
    2. On the Settings tab<\/strong>, you can see the data source bucket being configured.<\/li>\n
    3. Select the radio button for the data source and choose Sync now<\/strong>.<\/li>\n<\/ol>\n
      \"Choose<\/p>\n
      The data source sync can take 10\u201315 minutes to complete, but you don\u2019t have to wait to move to the next step.<\/p>\n
      Creating users and groups in the Amazon Cognito user pool<\/h2>\n
      In the terminal with the AWS CLI or AWS CloudShell<\/a>, run the following commands to create users and groups in the Amazon Cognito user pool to use for our application. You need to copy the contents of the Physical ID<\/strong> column in the UserPool<\/strong> row from the Resources<\/strong> tab of the CloudFormation stack. This is the user pool ID to use in the following steps. We set AmazonKendra@2020<\/code> as the temporary password for all the users. This password is required when logging in for the first time, and Amazon Cognito enforces a password reset.<\/p>\n
      \n
      USER_POOL_ID=<PASTE-USER-POOL-ID-HERE<\/strong>>\r\naws cognito-idp create-group --group-name customer --user-pool-id ${USER_POOL_ID}\r\naws cognito-idp create-group --group-name AWS-Sales --user-pool-id ${USER_POOL_ID}\r\naws cognito-idp create-group --group-name AWS-Marketing --user-pool-id ${USER_POOL_ID}\r\naws cognito-idp create-group --group-name AWS-SA --user-pool-id ${USER_POOL_ID}\r\naws cognito-idp admin-create-user --user-pool-id ${USER_POOL_ID} --username patricia --temporary-password AmazonKendra@2020\r\naws cognito-idp admin-create-user --user-pool-id ${USER_POOL_ID} --username james  --temporary-password AmazonKendra@2020\r\naws cognito-idp admin-create-user --user-pool-id ${USER_POOL_ID} --username john  --temporary-password AmazonKendra@2020\r\naws cognito-idp admin-create-user --user-pool-id ${USER_POOL_ID} --username mary  --temporary-password AmazonKendra@2020\r\naws cognito-idp admin-add-user-to-group --user-pool-id ${USER_POOL_ID} --username patricia --group-name customer\r\naws cognito-idp admin-add-user-to-group --user-pool-id ${USER_POOL_ID} --username james --group-name AWS-Sales\r\naws cognito-idp admin-add-user-to-group --user-pool-id ${USER_POOL_ID} --username john --group-name AWS-Marketing\r\naws cognito-idp admin-add-user-to-group --user-pool-id ${USER_POOL_ID} --username mary --group-name AWS-SA\r\n<\/code><\/pre>\n<\/div>\n
      Building and deploying the app<\/h2>\n
      Now we build and deploy the app using the following steps:<\/p>\n
        \n
      1. On the AWS Amplify<\/a> console, choose the app AWSKendraAuthApp<\/code>.<\/li>\n
      2. Choose Run build<\/strong>.<\/li>\n<\/ol>\n
        \"Choose<\/p>\n
        You can monitor the build progress on the console.<\/p>\n
        \"You<\/p>\n
        Let the build continue and complete the steps: Provision, Build, Deploy, and Verify. After this, the application is deployed and ready to use.<\/p>\n
        You can browse through the source code by opening up the CodeCommit repository. The important file to look at is src\/App.tsx<\/code>.<\/p>\n
          \n
        1. Choose the link on the left to start the application in a new browser tab.<\/li>\n<\/ol>\n
          \"Choose<\/p>\n
          Trial run<\/h2>\n
          We can now take a trial run of our app.<\/p>\n
            \n
          1. On the login page, sign in with the username patricia<\/code> and the temporary password AmazonKendra@2020<\/code>.<\/li>\n<\/ol>\n
            \"On<\/p>\n
            Amazon Cognito requires you to reset your password the first time you log in. After you log in, you can see the search field.<\/p>\n
            \"Amazon<\/p>\n
              \n
            1. In the search field, enter a query, such as what is serverless<\/code>?<\/li>\n
            2. Expand Filter search results<\/strong> to see different document types.<\/li>\n<\/ol>\n
              You can select different document types to filter the search results.<\/p>\n
              \"You<\/p>\n
                \n
              1. Sign out and repeat this process for other users that are created in the Cognito user pool, namely, james<\/code>, john<\/code>, and mary<\/code>.<\/li>\n<\/ol>\n
                You can also choose Continue as Guest <\/strong>to use the app without authenticating. However, this option only shows results from blogs.<\/p>\n
                \"You<\/p>\n
                You can return back to the login screen by choosing Welcome Guest! Click here to sign up or sign in<\/strong>.<\/p>\n
                Using the application<\/h2>\n
                You can use the application we developed by making a few search queries logged in as different users. To experience how access control works, issue the same query from different user accounts and observe the difference in the search results. The following users get results from different sources:<\/p>\n