{"id":1926,"date":"2022-03-03T18:05:41","date_gmt":"2022-03-03T18:05:41","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/03\/enable-the-visually-impaired-to-hear-documents-using-amazon-textract-and-amazon-polly\/"},"modified":"2022-03-03T18:05:41","modified_gmt":"2022-03-03T18:05:41","slug":"enable-the-visually-impaired-to-hear-documents-using-amazon-textract-and-amazon-polly","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/03\/03\/enable-the-visually-impaired-to-hear-documents-using-amazon-textract-and-amazon-polly\/","title":{"rendered":"Enable the visually impaired to hear documents using Amazon Textract and Amazon Polly"},"content":{"rendered":"
\n

At the 2021 AWS re:Invent conference in Las Vegas, we demoed Read For Me<\/a> at the AWS Builders Fair\u2014a website that helps the visually impaired hear documents.<\/p>\n

For better quality, view the video here<\/a>.<\/p>\n

Adaptive technology and accessibility features are often expensive, if they\u2019re available at all. Audio books help the visually impaired read. Audio description makes movies accessible. But what do you do when the content isn\u2019t already digitized?<\/p>\n

This post focuses on the AWS AI services Amazon Textract<\/a> and Amazon Polly<\/a>, which empower those with impaired vision. Read For Me was co-developed by Jack Marchetti, who is visually impaired.<\/p>\n

Solution overview<\/h2>\n

Through an event-driven, serverless architecture and a combination of multiple AI services, we can create natural-sounding audio files in multiple languages from a picture of a document, or any image with text. For example, a letter from the IRS, a holiday card from family, or even the opening titles to a film.<\/p>\n

The following Reference Architecture<\/a>, published in the AWS Architecture Center<\/a> shows the workflow of a user taking a picture with their phone and playing an MP3 of the content found within that document.<\/p>\n

\"\"<\/a><\/p>\n

The workflow includes the following steps:<\/p>\n

    \n
  1. Static content (HTML, CSS, JavaScript) is hosted on AWS Amplify<\/a>.<\/li>\n
  2. Temporary access is granted for anonymous users to backend services via an Amazon Cognito<\/a> identity pool.<\/li>\n
  3. The image files are stored in Amazon Simple Storage Service<\/a> (Amazon S3).<\/li>\n
  4. A user makes a POST request through Amazon API Gateway<\/a> to the audio service, which proxies to an express AWS Step Functions<\/a> workflow.<\/li>\n
  5. The Step Functions workflow includes the following steps:\n
      \n
    1. Amazon Textract<\/a> extracts text from the image.<\/li>\n
    2. Amazon Comprehend<\/a> detects the language of the text.<\/li>\n
    3. If the target language differs from the detected language, Amazon Translate<\/a> translates to the target language.<\/li>\n
    4. Amazon Polly<\/a> creates an audio file as output using the text.<\/li>\n<\/ol>\n<\/li>\n
    5. The AWS Step Functions workflow creates an audio file as output and stores it in Amazon S3 in MP3 format.<\/li>\n
    6. A pre-signed URL with the location of the audio file stored in Amazon S3 is sent back to the user\u2019s browser through API Gateway. The user\u2019s mobile device plays the audio file using the pre-signed URL.<\/li>\n<\/ol>\n

      In the following sections, we discuss the reasons for why we chose the specific services, architecture pattern, and service features for this solution.<\/p>\n

      AWS AI services<\/h2>\n

      Several AI services are wired together to power Read For Me:<\/p>\n