{"id":1191,"date":"2021-11-12T08:33:11","date_gmt":"2021-11-12T08:33:11","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/12\/customize-amazon-translate-output-to-meet-your-domain-and-organization-specific-vocabulary\/"},"modified":"2021-11-12T08:33:11","modified_gmt":"2021-11-12T08:33:11","slug":"customize-amazon-translate-output-to-meet-your-domain-and-organization-specific-vocabulary","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2021\/11\/12\/customize-amazon-translate-output-to-meet-your-domain-and-organization-specific-vocabulary\/","title":{"rendered":"Customize Amazon Translate output to meet your domain and organization specific vocabulary"},"content":{"rendered":"
\n

Amazon Translate<\/a> is a neural machine translation service that delivers fast, high-quality, affordable, and customizable language translation. When you translate from one language to another, you want your machine translation to be accurate, fluent, and most importantly contextual. Customization is key in keeping your machine translation contextual. Amazon Translate provides multiple capabilities for customization to achieve the best machine translation. One such capability is custom terminology<\/a>. Custom terminology enables you to customize your translation output such that your domain and organization specific vocabulary such as brand names, character names, model names, and other unique content (named entities) are translated exactly the way you need. To use the custom terminology feature, you create a terminology using a terminology file in a CSV or TMX file format and specify this custom terminology as a parameter in an Amazon Translate real-time translation<\/a> or asynchronous batch processing<\/a> request.<\/p>\n

Amazon Translate now supports multi-directional custom terminology. You no longer have to create multiple terminology CSV files with each one differing only in the first column to indicate the source language, include additional preprocessing logic to identify the dominant language, and choose the correct terminology file for the translation request. You can now use a single custom terminology for multiple source and target language combinations. Even when you set the source language to be detected automatically, Amazon Translate uses Amazon Comprehend<\/a> to determine the dominant language of the source material, uses it as the source language, and translates the text using the terms specified in the custom terminology. For additional details on custom terminology, refer to Customizing Your Translations with Custom Terminology<\/a>.<\/p>\n

In this post, we walk you through the step-by-step process of how to use custom terminology and get a customized machine translated output securely.<\/p>\n

Solution overview<\/h2>\n

To customize your translation for terms that are unique to your industry domain or organization, you define these terms in a terminology file in CSV or TMX file format. The terms within the custom terminology are considered case-sensitive, and Amazon Translate identifies an exact match between a terminology entry and a string in the source text when their case matches.<\/p>\n

For our use case, we have our data in CSV format, and the name of the file is custom_terminology.csv<\/code>. The data in the file should also be UTF-8<\/a> encoded. The following table summarizes the contents of the file.<\/p>\n\n\n\n\n\n\n\n\n
en<\/td>\nes<\/td>\nfr<\/td>\nhi<\/td>\nta<\/td>\n<\/tr>\n
Echo<\/td>\nEcho<\/td>\nEcho<\/td>\nEcho<\/td>\nEcho<\/td>\n<\/tr>\n
Show<\/td>\nShow<\/td>\nShow<\/td>\nShow<\/td>\nShow<\/td>\n<\/tr>\n
Amazon<\/td>\nAmazon<\/td>\nAmazon<\/td>\nAmazon<\/td>\nAmazon<\/td>\n<\/tr>\n
Alexa<\/td>\nAlexa<\/td>\nAlexa<\/td>\nAlexa<\/td>\nAlexa<\/td>\n<\/tr>\n
AZ2<\/td>\nAZ2<\/td>\nAZ2<\/td>\nAZ2<\/td>\nAZ2<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Import terminology<\/h2>\n

First, we import our multi-directional custom terminology using the custom_terminology.csv<\/code> file. In the following sections, we show you how to import your terminology via the AWS Management Console<\/a>, AWS Command Line Interface<\/a> (AWS CLI), or with the Amazon Translate SDK (Python Boto3).<\/p>\n

Amazon Translate console<\/h3>\n

To import the terminology via the console, complete the following steps:<\/p>\n

    \n
  1. On the Amazon Translate console, in the navigation pane, choose Custom terminology<\/strong>.<\/li>\n
  2. Choose Create terminology<\/strong>.<\/li>\n<\/ol>\n

    \"\"<\/a><\/p>\n

      \n
    1. For Name<\/strong>, enter an appropriate name, for example CustomTerminologyDemo<\/code>.<\/li>\n
    2. For Terminology file<\/strong>, upload the custom_terminology.csv<\/code> file.<\/li>\n
    3. For Terminology file data format<\/strong>, choose CSV<\/strong>, since we uploaded a CSV file.<\/li>\n
    4. For Directionality<\/strong>, choose Multi-directional<\/strong>.<\/li>\n
    5. For Encryption key<\/strong>, for the purpose of this post, we leave it as default, an AWS owned and managed key. You can select any appropriate key.<\/li>\n<\/ol>\n

      \u00a0<\/strong>Your data is always secure with Amazon Translate. It\u2019s encrypted using an AWS owned encryption key using AWS Key Management Service<\/a> (AWS KMS) by default. You can encrypt it using a key from your current account or use a key from a different account.<\/p>\n

        \n
      1. Choose Create Terminology<\/strong>.<\/li>\n<\/ol>\n

        \"\"<\/a><\/p>\n

        Your custom terminology is now listed on the Custom terminology<\/strong> page.<\/p>\n

        \"\"<\/a><\/p>\n

        AWS CLI<\/h3>\n

        The following AWS CLI commands are formatted for Unix, Linux, and macOS. For Windows, replace the backslash (<\/code>) Unix continuation character at the end of each line with a caret (^<\/code>).<\/p>\n

        You can call the import-terminology<\/code> AWS CLI command to create a custom terminology resource:<\/p>\n

        \n
        aws translate import-terminology \n--region us-east-1 \n--name CustomTerminologyDemo \n--description \"Multi-Directional custom terminology in AWS Translate\" \n--merge-strategy OVERWRITE \n--data-file fileb:\/\/custom_terminology.csv \n--terminology-data Format=CSV,Directionality=MULTI \n<\/code><\/pre>\n<\/p><\/div>\n
        You get a response like the following snippet:<\/p>\n
        \n
        {\n    \"TerminologyProperties\": {\n        \"Name\": \"CustomTerminologyDemo\",\n        \"Description\": \"Multi-Directional custom terminology in AWS Translate\",\n        \"Arn\": \"arn:aws:translate:us-east-1:123456789012:terminology\/CustomTerminologyDemo\/LATEST\",\n        \"Directionality\": \"MULTI\"\n        \"SourceLanguageCode\": \"en\",\n        \"TargetLanguageCodes\": [\n            \"hi\",\n            \"fr\",\n            \"ta\",\n            \"es\"\n        ],\n        \"SizeBytes\": 136,\n        \"TermCount\": 20, \n        \"CreatedAt\": \"2021-10-12T15:29:51.294000-04:00\",\n        \"LastUpdatedAt\": \"2021-10-12T15:29:51.458000-04:00\"\n    }\n}\n<\/code><\/pre>\n<\/p><\/div>\n
        You can use the list-terminologies<\/code> command to list all the custom terminology created:<\/p>\n
        \n
        \n
        aws translate get-terminology --name CustomTerminologyDemo \u2013-region us-east-1<\/code><\/pre>\n<\/p><\/div>\n
        The response looks like the following:<\/p>\n
        \n
        {\n    \"TerminologyPropertiesList\": [\n        {\n            \"Name\": \"CustomTerminologyDemo\",\n            \"Arn\": \"arn:aws:translate:us-east-1:123456789012:terminology\/CustomTerminologyDemo\/LATEST\",\n            \"SourceLanguageCode\": \"en\",\n            \"TargetLanguageCodes\": [\n            \"hi\",\n            \"ta\",\n            \"fr\",\n            \"es\"\n            ],\n            \"SizeBytes\": 157,\n            \"TermCount\": 20,\n            \"CreatedAt\": \"2021-10-12T15:29:51.294000-04:00\",\n            \"LastUpdatedAt\": \"2021-10-12T15:29:51.458000-04:00\",\n \t\t\"Directionality\": \"MULTI\",\n\t\t\"Format\": \"CSV\"\t\t   \n        }\n    ]\n}\n<\/code><\/pre>\n<\/p><\/div>\n
        You can use the get-terminology<\/code> command to get the details of a specific custom terminology resource:<\/p>\n<\/p><\/div>\n
        \n
        aws translate get-terminology --name CustomTerminologyDemo --terminology-data-format CSV \u2013region us-east-1<\/code><\/pre>\n<\/p><\/div>\n
        The response looks like the following:<\/p>\n
        \n
        \n
        {\n    \"TerminologyProperties\": {\n        \"Name\": \"CustomTerminologyDemo\",\n        \"Description\": \"Custom terminology in AWS Translate\",\n        \"Arn\": \"arn:aws:translate:us-east-1:123456789012:terminology\/CustomTerminologyDemo\/LATEST\",\n        \"Format\": \"CSV\",\n        \"Directionality\": \"MULTI\"\n        \"SourceLanguageCode\": \"en\",\n        \"TargetLanguageCodes\": [\n            \"hi\",\n            \"fr\",\n            \"ta\",\n            \"es\"\n        ],\n        \"SizeBytes\": 136,\n        \"TermCount\": 20, \n        \"CreatedAt\": \"2021-10-12T15:29:51.294000-04:00\",\n        \"LastUpdatedAt\": \"2021-10-12T15:29:51.458000-04:00\"\n    },\n    \"TerminologyDataLocation\": {\n        \"RepositoryType\": \"S3\",\n        \"Location\": \"https:\/\/aws-translate-terminology-prod-us-east-1.s3.us-east-1.amazonaws.com\/123456789012\/CustomTerminologyDemo\/LATEST\/c5c307b8-30f3-4704-8e39-ca4e9330ff6f\/CSV\/Custom_terminology.csv?X-Amz-Security-Token=1111222233334444aaaaeeeefffff&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211022T150354Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1800&X-Amz-Credential=ASIA1a2b3c4d5e6f%2F20211022%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=aaaabbbb11112222\"\n    }\n}\n<\/code><\/pre>\n<\/p><\/div>\n<\/p><\/div>\n
        To delete a custom terminology resource, you can use the delete-terminology<\/code> command:<\/p>\n
        \n
        aws translate delete-terminology --name CustomTerminologyDemo \u2013region us-east-1<\/code><\/pre>\n<\/p><\/div>\n
        Amazon Translate SDK (Python Boto3)<\/h3>\n
        The following Python 3 code creates a custom terminology, lists all the custom terminology, and uses the terminology resource part of the real-time translation call:<\/p>\n
        \n
        import boto3\nimport json\n\ntranslate = boto3.client('translate')\n\nwith open('custom_terminology.csv', 'rb') as ct_file:\n    translate.import_terminology(\n        Name='CustomTerminology_boto3',\n        MergeStrategy='OVERWRITE',\n        Description='Terminology for Demo through boto3',\n        TerminologyData={\n            'File': ct_file.read(),\n            'Format': 'CSV'\n            'Directionality': 'MULTI'\n        }\n    )\n\nresponse = translate.list_terminologies()\nterminology_names = [tag[\"Name\"] for tag in response[\"TerminologyPropertiesList\"]]\nprint(str(terminology_names))\n\nresponse = translate.get_terminology(\n    Name='CustomTerminology_boto3',\n    TerminologyDataFormat='CSV'\n)\nprint(\"Name:{}\".format(response[\"TerminologyProperties\"][\"Name\"]))\nprint(\"Description:{}\".format(response[\"TerminologyProperties\"][\"Description\"]))\nprint(\"ARN:{}\".format(response[\"TerminologyProperties\"][\"Arn\"]))\nprint(\"Directionality:{}\".format(response[\"TerminologyProperties\"][\"Directionality\"]))\n\nSOURCE_TEXT = (\"Amazon a pr\u00e9sent\u00e9 aujourd'hui Echo Show 15, un nouvel ajout \u00e0 la famille Echo Show qui est con\u00e7u pour \u00eatre le c\u0153ur num\u00e9rique de votre maison\")\n\nOUTPUT_LANG_CODE = 'en'\n\nresult = translate.translate_text(\n    Text=SOURCE_TEXT,\n    TerminologyNames=['CustomTerminology_boto3'], \n    SourceLanguageCode='auto',\n    TargetLanguageCode=OUTPUT_LANG_CODE\n)\n\nprint(\"Translated Text:{}\".format(result['TranslatedText']))\n<\/code><\/pre>\n<\/p><\/div>\n
        Running the Python code prints the following result:<\/p>\n
        \n
        python translate_custom_terminology.py\n \n['CustomTerminology_boto3']\nName: CustomTerminology_boto3\nDescription: Terminology for Demo through boto3\nARN: arn:aws:translate:us-east-1:123456789012:terminology\/CustomTerminology_boto3\/LATEST\nDirectionality: MULTI\n\nTranslated Text: Amazon today introduced Echo Show 15, a new addition to the Echo Show family that is designed to be the digital heart of your home.\n<\/code><\/pre>\n<\/p><\/div>\n
        Real-time translation using multi-directional custom terminology<\/h2>\n
        In this section, we demonstrate two use cases using multi-directional custom terminology for real-time translation in Amazon Translate.<\/p>\n
        Scenario 1: Multi-directional custom terminology<\/h3>\n
        For a basic demonstration of using multi-directional custom terminology with real-time translation, we use the following sample text in Spanish to be translated to French.<\/p>\n
        Amazon ha presentado hoy el Echo Show 15, una nueva incorporaci\u00f3n a la familia Echo Show que est\u00e1 dise\u00f1ada para ser el coraz\u00f3n digital de tu hogar. Con una pantalla Full HD de 15,6 pulgadas y 1080p, el Echo Show 15 puede fijarse en la pared o colocarse sobre un soporte compatible, ya sea en orientaci\u00f3n vertical u horizontal, y est\u00e1 dise\u00f1ado para ayudarte a mantenerte organizado, conectado y entretenido. El Echo Show 15 est\u00e1 fabricado con el procesador Amazon AZ2 Neural Edge de \u00faltima generaci\u00f3n, una pantalla de inicio redise\u00f1ada con m\u00e1s opciones de personalizaci\u00f3n, nuevas funcionalidades de personalizaci\u00f3n con ID Visual, y experiencias de Alexa totalmente nuevas.<\/em><\/p>\n
        On the Amazon Translate console, complete the following steps:<\/p>\n
          \n
        1. Choose Spanish (es)<\/strong> as the Source language<\/strong>.<\/li>\n
        2. Choose French (fr)<\/strong> as the Target Language<\/strong>.<\/li>\n
        3. In the Additional settings<\/strong> section, turn Custom terminology<\/strong><\/li>\n
        4. Choose CustomTerminologyDemo<\/strong> as the terminology.<\/li>\n
        5. Enter the provided sample text in the Source Language<\/strong> text area.<\/li>\n<\/ol>\n
          The following screenshot shows the translated text with custom terminology applied.<\/p>\n
          \"\"<\/a><\/p>\n
          Spanish wasn\u2019t the first column in the terminology file we uploaded, but with multi-directional terminology support, Amazon Translate was able to use the supplied terminology file to customize the translation.<\/p>\n
          Scenario 2: Automatically detect source language<\/h3>\n
          In this use case, we demonstrate the capability in Amazon Translate to automatically detect the source language and use the supplied terminology file to customize the translation. We use the following sample text in French and translate it to Hindi:<\/p>\n
          Aujourd\u2019hui, Amazon pr\u00e9sente Echo Show 15, dernier-n\u00e9 de la gamme Echo Show, imagin\u00e9 pour \u00eatre le c\u0153ur num\u00e9rique de votre domicile. Avec un \u00e9cran Full HD 1080p de 15,6\u2019\u2019, Echo Show 15 peut \u00eatre fix\u00e9 au mur ou pos\u00e9 sur un support compatible, en orientation portrait ou paysage, et est con\u00e7u pour vous aider \u00e0 rester organis\u00e9\u00b7e, connect\u00e9\u00b7e et diverti\u00b7e. Echo Show 15 est \u00e9quip\u00e9 du processeur Amazon AZ2 Neural Edge de nouvelle g\u00e9n\u00e9ration, d\u2019un \u00e9cran d\u2019accueil repens\u00e9 avec plus d\u2019options et de nouvelles fonctionnalit\u00e9s de personnalisation gr\u00e2ce \u00e0 l\u2019identifiant facial, et b\u00e9n\u00e9ficie de toutes nouvelles exp\u00e9riences Alexa.<\/em><\/p>\n
          First let\u2019s demonstrate the translation without custom terminology.<\/p>\n
            \n
          1. Choose Source language <\/strong>as Auto (auto).<\/strong><\/li>\n
          2. Choose Hindi (hi)<\/strong> as the Target Language.<\/strong><\/li>\n
          3. Enter the provided text in the Source Language<\/strong> text area.<\/li>\n<\/ol>\n
            The following screenshot shows the translated text.<\/p>\n
            \"\"<\/a><\/p>\n
            Words like Amazon, Echo, Show, AZ2, and Alexa have been translated into Devanagari script.<\/p>\n
            Let\u2019s perform the same translation using our multi-directional custom terminology.<\/p>\n
              \n
            1. Choose Source language <\/strong>as Auto (auto).<\/strong><\/li>\n
            2. Choose Hindi (hi)<\/strong> as the Target Language.<\/strong><\/li>\n
            3. In the Additional settings<\/strong> section, turn Custom terminology<\/strong><\/li>\n
            4. Choose CustomTerminologyDemo<\/strong> as the terminology.<\/li>\n
            5. Enter the provided text in the Source Language<\/strong> text area.<\/li>\n<\/ol>\n
              The following screenshot shows the translated text with custom terminology applied.<\/p>\n
              \"\"<\/a><\/p>\n
              The source language was automatically detected as French, and with the multi-directional custom terminology support, Amazon Translate was able to use the supplied terminology file to customize the translation and retain the Latin script for words like Amazon, Echo, Show, AZ2, and Alexa.<\/p>\n
              Conclusion<\/h2>\n
              When you use custom terminology with translation requests, you can make sure that your unique content, such as brand names, character names, and model names, is translated exactly the way you need it, regardless of context and the Amazon Translate algorithm\u2019s decision. In addition, with multi-directional custom terminology, the management overhead of maintaining multiple terminologies is drastically reduced, and you can use a single terminology to translate to and from a specific language. For more information about how to get the best translation quality when using custom terminology, see Best Practices<\/a>.<\/p>\n
              \n
              About the Authors<\/h3>\n
              \"\"Siva Rajamani<\/strong> is a Boston-based Enterprise Solutions Architect at AWS. He enjoys working closely with customers and supporting their digital transformation and AWS adoption journey. His core areas of focus are serverless, application integration, and security. Outside of work, he enjoys outdoors activities and watching documentaries.<\/p>\n
              \"\"Sudhanshu Malhotra<\/strong> is a Boston-based Enterprise Solutions Architect for AWS. He\u2019s a technology enthusiast who enjoys helping customers find innovative solutions to complex business challenges. His core areas of focus are DevOps, machine learning, and security. When he\u2019s not working with customers on their journey to the cloud, he enjoys reading, hiking, and exploring new cuisines.<\/p>\n
              \"\"Watson G. Srivathsan<\/strong> is the Sr. Product Manager for Amazon Translate, AWS\u2019s natural language processing service. On weekends you will find him exploring the outdoors in the Pacific Northwest.<\/p>\n
                     \n      <\/div>\n","protected":false},"excerpt":{"rendered":"
              https:\/\/aws.amazon.com\/blogs\/machine-learning\/customize-amazon-translate-output-to-meet-your-domain-and-organization-specific-vocabulary\/<\/p>\n","protected":false},"author":0,"featured_media":1192,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1191"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1191"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1191\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1192"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1191"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1191"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1191"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}