{"id":320,"date":"2020-09-30T21:37:43","date_gmt":"2020-09-30T21:37:43","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/30\/building-custom-language-models-to-supercharge-speech-to-text-performance-for-amazon-transcribe\/"},"modified":"2020-09-30T21:37:43","modified_gmt":"2020-09-30T21:37:43","slug":"building-custom-language-models-to-supercharge-speech-to-text-performance-for-amazon-transcribe","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/30\/building-custom-language-models-to-supercharge-speech-to-text-performance-for-amazon-transcribe\/","title":{"rendered":"Building custom language models to supercharge speech-to-text performance for Amazon Transcribe"},"content":{"rendered":"
\n

Amazon Transcribe<\/a> is a fully-managed automatic speech recognition service (ASR) that makes it easy to add speech-to-text capabilities to voice-enabled applications. As our service grows, so does the diversity of our customer base, which now spans domains such as insurance, finance, law, real estate, media, hospitality, and more. Naturally, customers in different market segments have asked Amazon Transcribe for more customization options to further enhance transcription performance.<\/p>\n

We\u2019re excited to introduce Custom Language Models (CLM)<\/a>. The new feature allows you to submit a corpus of text data to train custom language models that target domain-specific use cases. Using CLM is easy because it capitalizes on existing data that you already possess (such as marketing assets, website content, and training manuals).<\/p>\n

In this post, we show you how to best use your available data to train a custom language model tailored for your speech-to-text<\/a> use case. Although our walkthrough uses a transcription example from the video gaming industry, you can use CLM to enhance custom speech recognition for any domain of your choosing. This post assumes that you\u2019re already familiar with how to use Amazon Transcribe, and focuses on demonstrating how to use the new CLM feature. Additional resources for using the service are available at the end.<\/p>\n

Establishing context for evaluating CLM transcription performance<\/h2>\n

To evaluate how powerful CLM can be in terms of enhancing transcription accuracy, there are few steps we want to take. First we need to establish a baseline. To do that, we recorded an audio sample<\/a> that contains speech content and lingo commonly found in video game chats. We also have a human-generated transcription to benchmark the ground truth. This ground truth transcript serves as the reference transcript we use to compare the general transcription output of Amazon Transcribe and the output of CLM. The following is a partial snippet of this reference transcript.<\/p>\n

\n

The 2020 holiday season is right around the corner. And with the way that the year\u2019s been going, we can all hope for a little excitement around the next-gen video game consoles coming out soon. So, what\u2019s the difference in hardware specs between the upcoming Playstation 5 and Xbox Series X? Well, let\u2019s take a look under the hoods of each next-gen gaming console. The PS5 features an AMD Zen 2 CPU with up to 3.5 GHz frequency. It sports an AMD Radeon GPU that touts 10.3 teraflops, running up to 2.23 GHz. Memory and storage respectively dial in at 16 GB and 825 GB. The PS5 supports both PS The PS5 supports both 4K and 8K resolution screens. The Xbox Series X also features an AMD Zen 2 CPU, but clocks in at 3.8 GHz instead. The console boasts a similar AMD custom GPU with 12 teraflops and 1.825 GHz.<\/p>\n

Memory is the same as that of the PS5\u2019s, coming in at 16 GB. But the default storage is where the system has an edge, bringing out a massive 1 TB hard drive. Like the PS5, the Series X also supports 4K and 8K resolution screens as well. Those of course, are just the numbers. Therefore, it remains to be seen exactly how the performance plays out in practice. It\u2019s worth noting that both systems have incorporated ray-tracing technology, something that\u2019s used to make light and shadows look better in-game. Both systems also offer 3D audio output for immersive experiences.<\/p>\n<\/blockquote>\n

Next, we want to run the sample audio through Amazon Transcribe using its generic speech engine and compare the text output to the ground truth transcript. We\u2019ve produced a partial snippet of the side-by-side comparison and highlighted errors for visibility. Compared to the ground truth transcript, the default Amazon Transcribe transcript showed a Word Error Rate (WER) of 31.87%. This WER shouldn\u2019t be interpreted as a full representation of the Amazon Transcribe service performance. It is just one instance for a very specific example audio. Note that accuracy is a measure of 100 minus WER. So the lower the WER, the higher the accuracy. For more information about calculating WER, see Word Error Rate<\/a> on Wikipedia.<\/p>\n

*Note that this WER is not a representation of the Amazon Transcribe service performance. It is just one instance for a very specific and limited test example. All figures are for single-case and limited scope illustration purposes.<\/em><\/p>\n

The following text is the human-generated ground-truth reference transcript:<\/p>\n

\n

The 2020 holiday season is right around the corner. And with the way that the year\u2019s been going, we can all hope for a little excitement around the next-gen video game consoles coming out soon. So, what\u2019s the difference in hardware specs<\/strong><\/span> between the upcoming Playstation 5 and Xbox Series X? Well, let\u2019s take a look under the hoods of each next-gen gaming console<\/strong><\/span>. The PS5 features an AMD Zen 2 CPU<\/strong> <\/span>with up to 3.5 GHz frequency. It sports an AMD Radeon<\/strong><\/span> GPU that touts<\/strong> <\/span>10.3 teraflops, running up to 2.23 GHz. Memory and storage respectively dial in<\/strong><\/span> at 16 GB and 825 GB. The PS5 supports both PS The PS5 supports both 4K and 8K resolution<\/strong> <\/span>screens. Meanwhile, the Xbox Series X also features an AMD Zen 2 CPU<\/strong><\/span>, but clocks in at 3.8 GHz<\/strong><\/span> instead. The console boasts a similar AMD custom<\/strong><\/span> GPU with 12 teraflops and 1.825 GHz. Memory is the same as that of the PS5\u2019s, coming in at<\/strong><\/span> 16 GB. But the default storage is where the system has an edge, bringing out a massive 1 TB hard drive. Like the PS5, the Series X<\/strong><\/span> also supports 4K and 8K resolution screens as well. Those of course, are just the numbers. Therefore, it remains to be seen exactly how the performance plays out in practice. It\u2019s worth noting that both systems have incorporated ray-tracing technology, something that\u2019s used to make light and shadows look better in-game. Both systems also offer 3D audio output for immersive experiences.<\/p>\n<\/blockquote>\n

The following text is the machine-generated transcript by the Amazon Transcribe generic speech engine:<\/p>\n

\n

the 2020 holiday season is right around the corner. And with the way that the years been going, we can all hope for a little excitement around the next Gen video game consoles coming out soon. So what\u2019s the difference in heart respects<\/strong><\/span> between the upcoming PlayStation five and Xbox Series X? Well, let\u2019s take a look under the hood of each of these Consul\u2019s<\/strong><\/span>. The PS five features in a M descend to CPU<\/strong> <\/span>with up to 3.5 gigahertz frequency is sports and AM the radio on<\/strong><\/span> GPU that tells<\/strong> <\/span>10.3 teraflops running up to 2.23 gigahertz memory and storage, respectively. Dahlin<\/strong> <\/span>at 16 gigabytes in 825 gigabytes. The PS five supports both PS. The PS five supports both four K and A K resolutions<\/strong><\/span>. Meanwhile, the Xbox Series X also features an AM descend to CPU<\/strong><\/span>, but clocks in at three point take. It hurts<\/strong><\/span> instead, the cost. Almost a similar AMG custom<\/strong><\/span> GPU, with 12 teraflops and 1.825 gigahertz memory, is the same as out of the PS. Fives come in at<\/strong> <\/span>16 gigabytes, but the default storage is where the system has an edge. Bring out a massive one terabyte hard drive. Like the PS five. The Siri\u2019s X<\/strong> <\/span>also supports four K and eight K resolution screens as well. Those, of course, are just the numbers. Therefore, it remains to be seen exactly how the performance plays out. In practice. It\u2019s worth noting that both systems have incorporated Ray tracing technology, something that\u2019s used to make light and shadows look better in game. Both systems also offer three D audio output for immersive experiences.<\/p>\n<\/blockquote>\n

Although the Amazon Transcribe generic speech model has done a decent job of transcribing the audio, it\u2019s more likely that a tailored custom model can yield higher transcription accuracy.<\/p>\n

Solution overview<\/h2>\n

In this post, we walk you through how to train a custom language model and evaluate how its transcription output compares to the reference transcript. You complete the following high-level steps:<\/p>\n

    \n
  1. Prepare your training data.<\/li>\n
  2. Train your CLM.<\/li>\n
  3. Use your CLM to transcribe audio.<\/li>\n
  4. Evaluate the results by comparing the CLM transcript against the generic model transcript\u2019s accuracy.<\/li>\n<\/ol>\n

    Preparing your training data<\/h3>\n

    Before we begin, it\u2019s important to distinguish between training data<\/em> and tuning data<\/em>.<\/p>\n

    Training data for CLM typically includes text data that is specific to your domain. Some examples of training data could include relevant text content from your website, training manuals, sales and marketing collateral, or other text sources.<\/p>\n

    Meanwhile, human-annotated audio transcripts of actual phone calls or media content that are directly relevant to your use case can be used as tuning data. Ideally, both training and tuning data ought to be domain-specific, but in practice you may only have a small amount of audio transcriptions available. In that case, transcriptions should be used as tuning data. If more transcription data is available, it can and should be used as part of the training set as well. For more information about the difference between training and tuning data, see Improving Domain-Specific Transcription Accuracy with Custom Language Models<\/a>.<\/p>\n

    Like many domains, video gaming has its own set of technical jargon, syntax, and speech dynamics that may make the use of a general speech recognition engine suboptimal. To build a custom model, we first need data that\u2019s representative of the domain. For this use case, we want free form text from the video gaming industry. We use a variety of publicly available information from a variety of sources about video gaming. For convenience, we\u2019ve compiled that training and tuning set, which you can download<\/a> to follow along. Keep in mind that the nature, quality, and quantity of your training data has a dramatic impact on the resultant custom model you build. All else equal, it\u2019s better to have more data than less.<\/p>\n

    As a general guideline, your training and tuning datasets should meet the following parameters:<\/p>\n