{"id":1541,"date":"2022-02-09T18:02:21","date_gmt":"2022-02-09T18:02:21","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/02\/09\/extract-entities-from-insurance-documents-using-amazon-comprehend-named-entity-recognition\/"},"modified":"2022-02-09T18:02:21","modified_gmt":"2022-02-09T18:02:21","slug":"extract-entities-from-insurance-documents-using-amazon-comprehend-named-entity-recognition","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/02\/09\/extract-entities-from-insurance-documents-using-amazon-comprehend-named-entity-recognition\/","title":{"rendered":"Extract entities from insurance documents using Amazon Comprehend named entity recognition"},"content":{"rendered":"
\n

Intelligent document processing (IDP) is a common use case for customers on AWS. You can utilize\u00a0Amazon Comprehend<\/a>\u00a0and\u00a0Amazon Textract<\/a>\u00a0for a variety of use cases ranging from document extraction, data classification, and entity extraction. One specific industry that uses IDP is insurance. They use IDP to automate data extraction for common use cases such as claims intake, policy servicing, quoting, payments, and next best actions. However, in some cases, an office receives a document with complex, label-less information. This is normally difficult for optical character recognition (OCR) software to capture, and identifying relationships and key entities becomes a challenge. The solution is often requires manual human entry to ensure high accuracy.<\/p>\n

In this post, we demonstrate how you can use named entity recognition<\/a> (NER) for documents in their native formats in Amazon Comprehend to address these challenges.<\/p>\n

Solution overview<\/h2>\n

In an insurance scenario, an insurer might receive a demand letter from an attorney\u2019s office. The demand letter includes information such as what law office is sending the letter, who their client is, and what actions are required to satisfy their requests, as shown in the following example:
\"\"<\/a><\/p>\n

Because of the varied locations that this information could be found in a demand letter, these documents are often forwarded to an individual adjuster, who takes the time to read through the letter to determine all the necessary information required to proceed with a claim. The document may have multiple names, addresses, and requests that each need to be classified. If the client is mixed up with the beneficiary, or the addresses are switched, delays could add up and negative consequences could impact the company and customers. Because there are often small differences between categories like addresses and names, the documents are often processed by humans rather than using an IDP approach.<\/p>\n

The preceding example document has many instances of overlapping entity values (entities that share similar properties but aren\u2019t related). Examples of this are the address of the law office vs. the address of the insurance company or the names of the different individuals (attorney name, beneficiary, policy holder). Additionally, there is positional information (where the entity is positioned within the document) that a traditional text-only algorithm might miss. Therefore, traditional recognition techniques may not meet requirements.<\/p>\n

In this post, we use named entity recognition in Amazon Comprehend to solve these challenges. The benefit of using this method is that the custom entity recognition model uses both the natural language and positional information of the text to accurately extract custom entities that may otherwise be impacted when flattening a document, as demonstrated in our preceding example of overlapping entity values. For this post, we use an AWS artificially created dataset of legal requisition and demand letters for life insurance, but you can use this approach across any industry and document that may benefit from spatial data in custom NER training. The following diagram depicts the solution architecture:
\"\"<\/a><\/p>\n

We implement the solution with the following high-level steps:<\/p>\n

    \n
  1. Clone the repository containing the sample dataset.<\/li>\n
  2. Create an Amazon Simple Storage Service<\/a> (Amazon S3) bucket.<\/li>\n
  3. Create and train your custom entity recognition model.<\/li>\n
  4. Use the model by running an asynchronous batch job.<\/li>\n<\/ol>\n

    Prerequisites<\/h2>\n

    You need to complete the following prerequisites to use this solution:<\/p>\n

      \n
    1. Install<\/a> Python 3.8.x.<\/li>\n
    2. Make sure you have pip installed<\/a>.<\/li>\n
    3. Install and configure<\/a> the AWS Command Line Interface<\/a> (AWS CLI).<\/li>\n
    4. Configure<\/a> your AWS credentials.<\/li>\n<\/ol>\n

      Annotate your documents<\/h2>\n

      To train a custom entity recognition model that can be used on your PDF, Word, and plain text documents, you need to first annotate PDF documents using a custom Amazon SageMaker Ground Truth<\/a> annotation template that is provided by Amazon Comprehend. For instructions, see Custom document annotation for extracting named entities in documents using Amazon Comprehend<\/a>.<\/p>\n

      We recommend a minimum of 250 documents and 100 annotations per entity to ensure good quality predictions. With more training data, you\u2019re more likely to produce a higher-quality model.<\/p>\n

      When you\u2019ve finished annotating, you can train a custom entity recognition model and use it to extract custom entities from PDF, Word, and plain text documents for batch (asynchronous) processing.<\/p>\n

      For this post, we have already labeled our sample dataset, and you don\u2019t have to annotate the documents provided<\/strong>. However, if you want to use your own documents or adjust the entities, you have to annotate the documents. For instructions, see Custom document annotation for extracting named entities in documents using Amazon Comprehend<\/a>.<\/p>\n

      We extract the following entities (which are case sensitive):<\/p>\n