{"id":404,"date":"2020-10-15T22:15:43","date_gmt":"2020-10-15T22:15:43","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/10\/15\/building-a-medical-image-search-platform-on-aws\/"},"modified":"2020-10-15T22:15:43","modified_gmt":"2020-10-15T22:15:43","slug":"building-a-medical-image-search-platform-on-aws","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/10\/15\/building-a-medical-image-search-platform-on-aws\/","title":{"rendered":"Building a medical image search platform on AWS"},"content":{"rendered":"
\n

Improving radiologist efficiency and preventing burnout is a primary goal for healthcare providers. A nationwide study published in Mayo Clinic Proceedings<\/em> in 2015 showed radiologist burnout percentage at a concerning 61% [1]. In additon, the report concludes that \u201cburnout and satisfaction with work-life balance in US physicians worsened from 2011 to 2014. More than half of US physicians are now experiencing professional burnout.\u201d[2] As technologists, we\u2019re looking for ways to put new and innovative solutions in the hands of physicians to make them more efficient, reduce burnout, and improve care quality.<\/p>\n

To reduce burnout and improve value-based care through data-driven decision-making, Artificial Intelligence (AI) can be used to unlock the information trapped in the vast amount of unstructured data (e.g. images, texts, and voice) and create clinically actionable knowledge base. AWS AI services can derive insights and relationships from free-form medical reports, automate the knowledge sharing process, and eventually improve personalized care experience.<\/p>\n

In this post, we use Convolutional Neural Networks (CNN) as a feature extractor to convert medical images into a one-dimensional feature vector with a size of 1024. We call this process medical image embedding<\/em>. Then we index the image feature vector using the K-nearest neighbors (KNN) algorithm<\/a> in Amazon Elasticsearch Service<\/a> (Amazon ES) to build a similarity-based image retrieval system. Additionally, we use the AWS managed natural language processing (NLP) service Amazon Comprehend Medical<\/a> to perform named entity recognition (NER)<\/a> against free text clinical reports. The detected named entities are also linked to medical ontology, ICD-10-CM, to enable simple aggregation and distribution analysis. The presented solution also includes a front-end React web application and backend GraphQL API managed by AWS Amplify<\/a> and AWS AppSync<\/a>, and authentication is handled by Amazon Cognito<\/a>.<\/p>\n

After deploying this working solution, the end-users (healthcare providers) can search through a repository of unstructured free text and medical images, conduct analytical operations, and use it in medical training and clinical decision support. This eliminates the need to manually analyze all the images and reports and get to the most relevant ones. Using a system like this improves the provider\u2019s efficiency. The following graphic shows an example end result of the deployed application.<\/p>\n

\u00a0<\/p>\n

Dataset and architecture<\/h2>\n

We use the MIMIC CXR<\/a> dataset to demonstrate how this working solution can benefit healthcare providers, in particular, radiologists. MIMIC CXR is a publicly available database of chest X-ray images in DICOM format and the associated radiology reports as free text files[3]. The methods for data collection and the data structures in this dataset have been well documented and are very detailed [3]. Also, this is a restricted-access resource. To access the files, you must be a registered user<\/a> and sign the data use agreement<\/a>. The following sections provide more details on the components of the architecture.<\/p>\n

The following diagram illustrates the solution architecture.<\/p>\n

\"\"<\/p>\n

The architecture is comprised of the offline data transformation and online query components. The offline data transformation step, the unstructured data, including free texts and image files, is converted into structured data.<\/p>\n

Electronic Heath Record (EHR) radiology reports as free text are processed using Amazon Comprehend Medical, an NLP service that uses machine learning to extract relevant medical information from unstructured text, such as medical conditions including clinical signs, diagnosis, and symptoms. The named entities are identified and mapped to structured vocabularies, such as ICD-10 Clinical Modifications (CMs) ontology. The unstructured text plus structured named entities are stored in Amazon ES to enable free text search and term aggregations.<\/p>\n

The medical images from Picture Archiving and Communication System (PACS) are converted into vector representations using a pretrained deep learning model deployed in an Amazon Elastic Container Service<\/a> (Amazon ECS) AWS Fargate<\/a> cluster. Similar visual search on AWS <\/a>has been published previously for online retail product image search. It used an Amazon SageMaker<\/a> built-in KNN algorithm<\/a> for similarity search, which supports different index types and distance metrics.<\/p>\n

We took advantage of the KNN for Amazon ES<\/a> to find the k<\/em> closest<\/em> images<\/em> from a feature space as demonstrated on the GitHub repo<\/a>. KNN search is supported in Amazon ES version 7.4+. The container running on the ECS Fargate cluster reads medical images in DICOM format, carries out image embedding using a pretrained model, and saves a PNG thumbnail in an Amazon Simple Storage Service<\/a> (Amazon S3) bucket, which serves as the storage for AWS Amplify React<\/a> web application. It also parses out the DICOM image metadata and saves them in Amazon DynamoDB<\/a>. The image vectors are saved in an Elasticsearch cluster and are used for the KNN visual search, which is implemented in an AWS Lambda<\/a> function.<\/p>\n

The unstructured data from EHR and PACS needs to be transferred to Amazon S3 to trigger the serverless data processing pipeline through the Lambda functions. You can achieve this data transfer by using AWS Storage Gateway<\/a> or AWS DataSync<\/a>, which is out of the scope of this post. The online query API, including the GraphQL schemas and resolvers, was developed in AWS AppSync. The front-end web application was developed using the Amplify React framework, which can be deployed using the Amplify CLI. The detailed AWS CloudFormation<\/a> templates and sample code are available in the Github repo<\/a>.<\/p>\n

Solution overview<\/h2>\n

To deploy the solution, you complete the following steps:<\/p>\n

    \n
  1. Deploy the Amplify React web application for online search.<\/li>\n
  2. Deploy the image-embedding container to AWS Fargate.<\/li>\n
  3. Deploy the data-processing pipeline and AWS AppSync API.<\/li>\n<\/ol>\n

    Deploying the Amplify React web application<\/h2>\n

    The first step creates the Amplify React web application, as shown in the following diagram.<\/p>\n

    \"\"<\/p>\n

      \n
    1. \nInstall<\/a> and configure<\/a> the AWS Command Line Interface<\/a> (AWS CLI).<\/li>\n
    2. Install the AWS Amplify CLI<\/a>.<\/li>\n
    3. Clone the code base<\/a> with stepwise instructions.<\/li>\n
    4. Go to your code base folder and initialize the Amplify app using the command amplify init<\/code>. You must answer a series of questions, like the name of the Amplify app.<\/li>\n<\/ol>\n

      After this step, you have the following changes in your local and cloud environments:<\/p>\n