{"id":330,"date":"2020-10-02T01:18:48","date_gmt":"2020-10-02T01:18:48","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/10\/02\/creating-a-multi-department-enterprise-search-using-custom-attributes-in-amazon-kendra\/"},"modified":"2020-10-02T01:18:48","modified_gmt":"2020-10-02T01:18:48","slug":"creating-a-multi-department-enterprise-search-using-custom-attributes-in-amazon-kendra","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/10\/02\/creating-a-multi-department-enterprise-search-using-custom-attributes-in-amazon-kendra\/","title":{"rendered":"Creating a multi-department enterprise search using custom attributes in Amazon Kendra"},"content":{"rendered":"
\n

An enterprise typically houses multiple departments such as engineering, finance, legal, and marketing, creating a growing number of documents and content that employees need to access. Creating a search experience that intuitively delivers the right information according to an employee\u2019s role, and the department is critical to driving productivity and ensuring security.<\/p>\n

Amazon Kendra<\/a>\u00a0is a highly accurate and easy-to-use enterprise search service powered by machine learning. Amazon Kendra delivers powerful natural language search capabilities to your websites and applications. These capabilities help your end-users easily find the information they need within the vast amount of content spread across your company.<\/p>\n

With Amazon Kendra, you can index the content from multiple departments and data sources into one Amazon Kendra index. To tailor the search experience by user role and department, you can add metadata to your documents and FAQs using Kendra\u2019s built-in attributes and custom attributes and apply user context filters.<\/p>\n

For search queries issued from a specific department\u2019s webpage, you can set Kendra to only return content from that department filtered by the employee\u2019s access level. For example, an associate role may only access a subset of restricted documents. In contrast, the department manager might have access to all the documents.<\/p>\n

This post provides a solution to indexing content from multiple departments into one Amazon Kendra index. To manage content access, the organization can create restrictions based on an employee\u2019s role and department or provide page-level filtering of search results. It demonstrates how content is filtered based on the web page location and individual user groups.<\/p>\n

Solution architecture<\/h2>\n

The following architecture is comprised of two primary components: document ingestion into Amazon Kendra and document query using Amazon API Gateway<\/a>.<\/p>\n

\"Architecture<\/p>\n

The preceding diagram depicts a fictitious enterprise environment with two departments: Marketing and Legal. Each department has its own webpage on their internal website. Every department has two employee groups: associates and managers. Managers are entitled to see all the documents, but associates can only see a subset.<\/p>\n

When employees from Marketing issue a search query on their department page, they only see the documents they are entitled to within their department (pink documents without the key). In contrast, the Marketing Manager sees all Marketing documents (all pink documents).<\/p>\n

When employees from Legal search on a Marketing department page, they don\u2019t see any documents. When all employees search on the internal website\u2019s main page, they see the public documents common to all departments (yellow).<\/p>\n

The following table shows the types of documents an employee gets for the various query combinations of webpage, department, and access roles.<\/p>\n

\"Access<\/p>\n

Ingesting documents into Amazon Kendra<\/h2>\n

The document ingestion step consists of ingesting content and metadata from different departments\u2019 specific S3 buckets, indexed by Amazon Kendra. Content can\u00a0comprise structured data like FAQs and unstructured content like HTML, Microsoft PowerPoint, Microsoft Word, plain text, and PDF documents. For ingesting FAQ documents into Amazon Kendra, you can provide the questions, answers, and optional custom and access control attributes either in a CSV or JSON format.<\/p>\n

You can add metadata to your documents and FAQs using the built-in attributes in Amazon Kendra, custom attributes, and user context filters. You can filter content using a combination of these custom attributes and user context filters. For this post, we index each document and FAQ with:<\/p>\n

    \n
  1. Built-in attribute _category<\/code> to represent the web page.<\/li>\n
  2. User context filter attribute for the employee access level.<\/li>\n
  3. Custom attribute department<\/code> representing the employee department.<\/li>\n<\/ol>\n

    The following code is an example of the FAQ document for the Marketing webpage:<\/p>\n

    \n
    {\r\n\"SchemaVersion\": 1,\r\n\"FaqDocuments\": [{\r\n\"Question\": \"What is the reimbursement policy for business related expenses?\",\r\n\"Answer\": \"All expenses must be submitted within 2 weeks.\",\r\n\"Attributes\": {\r\n\"_category\": \"page_marketing\",\r\n\"department\": \"marketing\"\r\n},\r\n\"AccessControlList\": [{\r\n\"Name\": \"associate\",\r\n\"Type\": \"GROUP\",\r\n\"Access\": \"ALLOW\"\r\n},{\r\n\"Name\": \"manager\",\r\n\"Type\": \"GROUP\",\r\n\"Access\": \"ALLOW\"\r\n}\r\n]\r\n},\r\n{\r\n\"Question\": \"What are the manager guidelines for employee promotions ?\",\r\n\"Answer\": \"Guidelines for employee promotions can be found on the manager portal.\",\r\n\"Attributes\": {\r\n\"_category\": \"page_marketing\",\r\n\"department\": \"marketing\"\r\n},\r\n\"AccessControlList\": [{\r\n\"Name\": \"manager\",\r\n\"Type\": \"GROUP\",\r\n\"Access\": \"ALLOW\"\r\n}]\r\n}\r\n]\r\n}\r\n<\/code><\/pre>\n<\/div>\n
    The following code is an example of the metadata document for the Legal webpage:<\/p>\n
    \n
    {\r\n\"DocumentId\": \"doc1\",\r\n\"Title\": \"What is the immigration policy?\",\r\n\"ContentType\": \"PLAIN_TEXT\",\r\n\"Attributes\": {\r\n\"_category\": \"page_legal\",\r\n\"department\": \"legal\"\r\n},\r\n\"AccessControlList\": [\r\n{\r\n\"Name\": \"associate\",\r\n\"Type\": \"GROUP\",\r\n\"Access\": \"ALLOW\"\r\n}\r\n]\r\n}<\/code><\/pre>\n<\/div>\n
    Document search by department<\/h2>\n
    The search capability is exposed to the client application using an API Gateway endpoint. The API accepts an optional path parameter for the webpage on which the query was issued. If the query comes from the Marketing-specific page, the query looks like \/search\/dept\/marketing<\/code>. For a comprehensive website search covering all the departments, you will leave out the path parameter. The query looks like \/search<\/code>.\u00a0<\/strong>Every API request also has two header values:\u00a0<\/strong>EMP_ROLE<\/code>, representing the employee access level, and EMP_DEPT<\/code>, representing the department name. In this post, we don\u2019t describe how to authenticate users. We assume that you populate these two header values after authenticating the user with\u00a0Amazon Cognito<\/a>\u00a0or your custom solutions.<\/p>\n
    The\u00a0AWS Lambda<\/a>\u00a0function that serves the API Gateway parses the path parameters and headers and issues an Amazon Kendra query call with AttributeFilters<\/code> set to the category name from the path parameter (if present), the employee access level, and department from the headers. Amazon Kendra returns the FAQs and documents for that particular category and filters them by the employee access level and department. The Lambda function constructs a response with these search results and sends the FAQ and document search results back to the client application.<\/p>\n
    Deploying the AWS CloudFormation template<\/h2>\n
      \n
    1. You can deploy this architecture using the provided AWS CloudFormation<\/a> template in us-east-1<\/code>. Please click to get started.<\/li>\n<\/ol>\n
      \"CloudFormation<\/a><\/p>\n
        \n
      1. Choose Next<\/strong>.<\/li>\n
      2. Provide a stack name and choose Next<\/strong>.<\/li>\n
      3. In the\u00a0Capabilities and transforms\u00a0<\/strong>section, select all three check-boxes to provide acknowledgment to AWS CloudFormation to create IAM resources and expand the template. \"Acknowledgement\n<\/li>\n
      4. Choose\u00a0Create stack<\/strong>.<\/strong>\n<\/li>\n<\/ol>\n
        This process might take 15 minutes or more to complete and creates the following resources:<\/p>\n