{"id":1443,"date":"2022-01-05T20:23:30","date_gmt":"2022-01-05T20:23:30","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/05\/how-to-approach-conversation-design-with-amazon-lex-building-and-testing-part-3\/"},"modified":"2022-01-05T20:23:30","modified_gmt":"2022-01-05T20:23:30","slug":"how-to-approach-conversation-design-with-amazon-lex-building-and-testing-part-3","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2022\/01\/05\/how-to-approach-conversation-design-with-amazon-lex-building-and-testing-part-3\/","title":{"rendered":"How to approach conversation design with Amazon Lex: Building and testing (Part 3)"},"content":{"rendered":"<div id=\"\">\n<p><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/12\/17\/ML-5750-image001.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-31877\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/12\/17\/ML-5750-image001.png\" alt=\"\" width=\"1429\" height=\"360\"><\/a><\/p>\n<p>In parts <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/part-1-approach-conversation-design-the-basics\/\" target=\"_blank\" rel=\"noopener noreferrer\">one<\/a> and <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/part-2-how-to-approach-conversation-design-getting-started-with-amazon-lex\/\" target=\"_blank\" rel=\"noopener noreferrer\">two<\/a> of our guide to conversation design with <a href=\"https:\/\/aws.amazon.com\/lex\/\" target=\"_blank\" rel=\"noopener noreferrer\">Amazon Lex<\/a>, we discussed how to gather requirements for your conversational AI application and draft conversational flows. In this post, we help you bring all the pieces together. You\u2019ll learn how draft an interaction model to deliver natural conversational experiences, and how to test and tune your application.<\/p>\n<h2>Draft an interaction model<\/h2>\n<p>In the <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/part-2-how-to-approach-conversation-design-getting-started-with-amazon-lex\/\" target=\"_blank\" rel=\"noopener noreferrer\">second post<\/a> of this series, you identified some use cases that you wanted to automate and wrote sample interactions between a user and your application. In this post, we use these use cases to build an Amazon Lex framework, called an interaction model, but first, let\u2019s review some important definitions.<\/p>\n<p>An <em>interaction model<\/em> defines the tasks that your application can help a user accomplish in a structure that a bot can understand. In your interaction model, you define the five major components for Amazon Lex: intents, sample utterances, slot names, slot values, and slot synonyms. You can make an interaction model (also called a <em>bot definition<\/em>) in whatever format works best for you and your team, such as in a spreadsheet or directly on the Amazon Lex console.<\/p>\n<p>We start by defining intents. An <em>intent<\/em> is a user\u2019s goal\u2014what is the user trying to accomplish when they interact with your bot? Continuing with the banking bot example, a user goal might be something like blocking a card or opening a bank account. For example, in the following dialog, the user\u2019s intent is to set up a recurring payment.<\/p>\n<table border=\"1px\">\n<tbody>\n<tr>\n<td><span>Turn<\/span><\/td>\n<td><span>Participant<\/span><\/td>\n<td><span>Dialog<\/span><\/td>\n<td><span>Notes<\/span><\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>Amazon Lex<\/td>\n<td>Thanks for contacting Your Favorite Loan Company. What can I help you with today?<\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>Customer<\/td>\n<td>I wanna set up a recurring payment.<\/td>\n<\/tr>\n<tr>\n<td>2<\/td>\n<td>Amazon Lex<\/td>\n<td>Your current balance is [$200.00]. That includes an overdue balance of [$100.00]. A payment of [$200.00] is due on October 15th. To set up a recurring payment, you\u2019ll need the routing number and the account number. Do you have that information ready?<\/td>\n<td>Can customers change their debit date? Or only the account?<\/td>\n<\/tr>\n<tr>\n<td>2<\/td>\n<td>Customer<\/td>\n<td>No<\/td>\n<\/tr>\n<tr>\n<td>3<\/td>\n<td>Amazon Lex<\/td>\n<td>Okay. I can wait. Let me know when you have the information.<\/td>\n<\/tr>\n<tr>\n<td>3<\/td>\n<td>Customer<\/td>\n<td>\n<p>\u2026<\/p>\n<p>I\u2019m ready.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>4<\/td>\n<td>Amazon Lex<\/td>\n<td>To start, what\u2019s the 9-digit routing number?<\/td>\n<\/tr>\n<tr>\n<td>4<\/td>\n<td>Customer<\/td>\n<td>999-999-999<\/td>\n<\/tr>\n<tr>\n<td>5<\/td>\n<td>Amazon Lex<\/td>\n<td>And what\u2019s the account number?<\/td>\n<\/tr>\n<tr>\n<td>5<\/td>\n<td>Customer<\/td>\n<td>1234 567 890<\/td>\n<\/tr>\n<tr>\n<td>\u2026<\/td>\n<td>\u2026<\/td>\n<td>\u2026<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Each intent should come directly from a use case that you previously established in the earlier steps. We go into more detail about how to define intents later in this post.<\/p>\n<p>Continuing on, a <em>prompt label<\/em> is a value defined by the application\u2019s designer that maps to something the bot says.<\/p>\n<p>A <em>sample utterance<\/em> is something the user says to the bot that is defined in the interaction model to help the bot classify customer intent. For example, if you\u2019re creating an intent for opening a bank account, you\u2019d likely want to include utterances like \u201copen an account,\u201d \u201chelp with opening an account,\u201d or \u201cHow can I open a bank account?\u201d The idea behind sample utterances is that by defining a class of utterances with similar semantic content, the bot can use these to make an educated guess about what the user\u2019s goal is. Even if you don\u2019t define every possible utterance (and you shouldn\u2019t), the bot can guess what the user is trying to do.<\/p>\n<p>A <em>slot<\/em> is a piece of information that the user provides in order to accomplish their goal. For example, if a customer wants to open a bank account, we need to know the type of account. We can use a slot to collect those account types, and name it something that builders will understand, like <code>AccountType<\/code>. Slots can be either required or optional, depending on the use case. For example, you might need a required slot like <code>BirthDate<\/code> to authenticate your user, but collect an optional slot type of <code>AccountType<\/code> to disambiguate between the different accounts a user might have. <em>Slot values<\/em> are the pieces of information that you want the bot to recognize as a slot, like \u201cchecking\u201d or \u201csavings.\u201d <em>Synonyms<\/em> are alternate ways of saying a slot value, like \u201cISA\u201d or \u201cdeposit account.\u201d<\/p>\n<p>Finally, a <em>slot corresponding utterance<\/em> is an utterance that contains a slot value, but doesn\u2019t contain an intent, such as \u201cto my savings account\u201d or \u201cit\u2019s for my savings account.\u201d In these utterances, you can\u2019t tell what the user is trying to accomplish without the context of the rest of the conversation, but they do contain valuable slot information that you need the bot to capture.<\/p>\n<p>The bot also has some available actions, <code>ElicitIntent<\/code> and <code>ElicitSlot<\/code>, which mean that the bot is either trying to capture the user\u2019s intent or the bot is looking to gather some slot information.<\/p>\n<p>Now that you\u2019ve defined the values for all these components and put all those pieces together, you\u2019ve created the first draft of an interaction model. Here\u2019s an example, complete with the bot\u2019s available actions.<\/p>\n<p><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/12\/17\/ML-5750image003.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-31878\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/12\/17\/ML-5750image003.png\" alt=\"\" width=\"1276\" height=\"500\"><\/a><\/p>\n<h2>Turn user stories into intents and slots<\/h2>\n<p>From your user stories, you\u2019ve identified the use cases that you want your application to be able to help your users fulfill, such as blocking a bank card due to fraud or opening a new credit card. Make a complete list of all use cases that you developed. Now, it\u2019s time to work backwards from the use cases to create user intents.<\/p>\n<p>Start by getting a group of people together from all different teams of your organization\u2014business analysts, technical pros, and leadership team members should all be present. Ask each person to create a list of possible things that they might reasonably say to a human agent or to an AI application for help with their use case. For example, if your use case is to open an account, you might list things like \u201cI\u2019d like to open a bank account,\u201d \u201cCan you help me open a new account?\u201d or \u201cOpening a savings account.\u201d Be flexible with what you write. Have each person write 10\u201320 utterances per use case. Keep in mind the variety available in human language:<\/p>\n<ul>\n<li><strong>Verb variation<\/strong> \u2013 Open, start, begin, get started, establish<\/li>\n<li><strong>Noun variation<\/strong> \u2013 Account, savings, first credit card, new customer<\/li>\n<li><strong>Phrase or full sentence<\/strong> \u2013 Open account, I\u2019d like to open an account<\/li>\n<li><strong>Statements versus questions<\/strong> \u2013 I want to open an account, Can you help me with getting started with a new account?<\/li>\n<li><strong>Implicit understanding<\/strong> \u2013 I\u2019m a new customer, Help for new customers<\/li>\n<li><strong>Tone (formal or informal) <\/strong>\u2013 I need some assistance with opening a new high-yield savings account, I wanna open a new card<\/li>\n<\/ul>\n<p>Now, compare with each other. Combine all the utterances into a team-wide list, and organize them with the most frequent utterances first. You can use these as a head start on your sample utterances. Try to classify each utterance into a single use case. You might think that this seems easy or obvious, since you just created these utterances directly from a list of use cases, but you might be surprised by how ambiguous human language can be.<\/p>\n<p>Now that you have your utterances and your use cases, decide on which ones you want to turn into intents for your bot. Again, this requires input from your team to complete successfully, but here are some basic strategies. For each use case that you created utterances for and classified utterances for, you should turn that into an intent. If you find that you\u2019re running into lots of ambiguity and having trouble classifying utterances, you should make a judgment call with your team about how you want to handle those tricky cases. You can merge use cases into a single intent if you find that there is too much similarity between the utterances, or you can split use cases up into more fine-grained intents if a single intent has too much variety in the utterances to classify them successfully.<\/p>\n<p>Another strategy for dealing with these ambiguous utterances is to use slots. If you have an assortment of similarly defined intents, like <code>OpenACreditCard<\/code> and <code>OpenADebitCard<\/code>, you might find that utterances like \u201copen a card\u201d cause confusion in the model. After all, as a human being, it\u2019s tough to decide just from that utterance whether the card is credit or debit card without more information. You can use slots to help by defining them in the model as a required piece of information, so that the bot looks for the words \u201ccredit\u201d or \u201cdebit\u201d in the utterance. Then, if those slots aren\u2019t filled, use that information to surface a disambiguation prompt like \u201cWould you like to open a new credit card or a new debit card?\u201d to help get the necessary information. You should keep a running list of utterances that are difficult to classify and use them for testing to see how users navigate these tricky situations.<\/p>\n<p>Remember that design is an iterative process and that no single interaction model will be perfect on the first try. This is why we continue with the next steps of prototyping and testing in order to build a successful conversational application.<\/p>\n<h2>Prototype your design<\/h2>\n<p>Given the often ambiguous nature of designing a conversational AI system, prototyping your design is crucial. Prototyping is a great way to gather meaningful feedback from real users in realistic contexts. In a design prototype, you want to build a simple way to test your design and gather feedback, without investing too much time building the software, because the design isn\u2019t even finalized yet.<\/p>\n<p>Following our example from earlier, we can build out a simple prototype to evaluate our user experience and amend our design as needed. Let\u2019s build a mini-prototype with two intents: <code>ReportCreditCardFraudIntent<\/code> and <code>OpenANewCreditCardAccountIntent<\/code>.<\/p>\n<table border=\"1px\">\n<tbody>\n<tr>\n<td><span>ReportCreditCardFraudIntent<\/span><\/td>\n<\/tr>\n<tr>\n<td>Unknown charge on my account<\/td>\n<\/tr>\n<tr>\n<td>I think someone stole my card<\/td>\n<\/tr>\n<tr>\n<td>Credit card fraud department<\/td>\n<\/tr>\n<tr>\n<td>Fraudulent charges on my account<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<table border=\"1px\">\n<tbody>\n<tr>\n<td><span>OpenANewCreditCardAccountIntent<\/span><\/td>\n<\/tr>\n<tr>\n<td>Open a new account<\/td>\n<\/tr>\n<tr>\n<td>Help with opening a credit card<\/td>\n<\/tr>\n<tr>\n<td>Open a credit card account<\/td>\n<\/tr>\n<tr>\n<td>I want to open a credit card<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Before we even build these intents on the Amazon Lex console, we can make a prototype to make sure that we\u2019re covering the most common utterances that a user might say. One simple way to do this would be to engage a few potential end-users, provide them with a scenario like pretending their card was stolen, and have them provide a few utterances. You can match this against what you\u2019ve outlined and collected with your team, and use this data to help enhance your design. You might find that users are very unlikely to just say \u201ccredit card\u201d at the open menu, or you might find that it\u2019s the most common utterance. Gathering information from a likely pool of users helps you understand your customers better to make your design more robust.<\/p>\n<p>The preceding example is a quick way to test your initial designs without much code. Other examples for prototyping your design would be Wizard of Oz (where the designer plays the role of the bot opposite a user who doesn\u2019t necessarily know they\u2019re talking to a human) or visual prototypes to help visualize the best experience (like a video simulating a chat window).<\/p>\n<h2>Test and tune your bot<\/h2>\n<p>Now that you\u2019ve gathered all the different elements of your design, and the experience has been built and integrated, you can start testing.<\/p>\n<p>The first step is to test against the design documentation you\u2019ve put together (the sample dialogs, conversation flows, and interaction model). Thoroughly test all the different intents, slots, slot values, paths, and error handling flows that you\u2019ve designed, going step by step through each one. The following is an example list of things to test:<\/p>\n<ul>\n<li><strong>Intent classification<\/strong> \u2013 Is the bot correctly predicting the intent for all utterances?<\/li>\n<li><strong>Slot values<\/strong> \u2013 Is the bot correctly recognizing all the possible slot values? For example, if you\u2019re using a slot with phone numbers over voice, does the bot recognize both \u201cone zero zero\u201d and \u201cone hundred\u201d as valid inputs?<\/li>\n<li><strong>Error handling<\/strong> \u2013 Are there places in the flow where you get stuck in a loop? Does the bot correctly recover if some kind of error occurs?<\/li>\n<li><strong>Prompts<\/strong> \u2013 Are the prompts eliciting the expected response? Is the wording clear and understandable for all users?<\/li>\n<\/ul>\n<p>The following is a sample test plan for a call center bot that you can use to guide your own testing.<\/p>\n<table border=\"1px\">\n<tbody>\n<tr>\n<td width=\"129\"><span><strong>Test ID<\/strong><\/span><\/td>\n<td width=\"120\"><span><strong>Scenario<\/strong><\/span><\/td>\n<td width=\"128\"><span><strong>Steps to test<\/strong><\/span><\/td>\n<td width=\"102\"><span><strong>Utterance<\/strong><\/span><\/td>\n<td width=\"115\"><span><strong>Successful?<\/strong><\/span><\/td>\n<\/tr>\n<tr>\n<td width=\"129\"><strong>Sample_100<\/strong><\/td>\n<td rowspan=\"3\" width=\"120\">You notice a fraudulent charge on your account<\/td>\n<td width=\"128\">Call number<\/td>\n<td width=\"102\"><\/td>\n<td width=\"115\">yes<\/td>\n<\/tr>\n<tr>\n<td width=\"129\"><strong>Sample_100<\/strong><\/td>\n<td width=\"128\">Say \u201ccredit card fraud\u201d<\/td>\n<td width=\"102\">Credit card fraud<\/td>\n<td width=\"115\">yes<\/td>\n<\/tr>\n<tr>\n<td width=\"129\"><strong>Sample_100<\/strong><\/td>\n<td width=\"128\">Say or enter date of birth when prompted<\/td>\n<td width=\"102\">January 1, 1980<\/td>\n<td width=\"115\">yes<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>After testing, you may find that your bot requires some tuning. Go through your interaction model and add in any commonly missed utterances, new intents or slots, or change the wording in problematic prompts that are losing users along the way. This is a great place to explore an<a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/using-a-test-framework-to-design-better-experiences-with-amazon-lex\/\" target=\"_blank\" rel=\"noopener noreferrer\"> automated testing framework <\/a>to expedite the testing process, but manual testing offers different insights about the user experience that can help alert you to any usability defects before launch.<\/p>\n<p>Finally, you should also provide your users with a way to test what you\u2019ve built against the business requirements that you defined in part one of this series. You need to make sure that before you launch your application to production it handles all customer requests and fulfills the business requirements that you received. Before beginning user testing, define the test plan with all stakeholders so it\u2019s clear to everyone on the team how you define success. Make sure that at this point, you\u2019ve developed your application in an environment that is as close as possible to the production environment, so that any feedback from this testing provides insight for production. Provide testers with the test plan and clearly document the results, so that it\u2019s easy to use the data from testing to make decisions about how best to move forward.<\/p>\n<p>After you\u2019ve launched your application, the work isn\u2019t done! Design is an iterative experience and continually requires fresh perspectives to improve. As part of the business requirements, you should define how you\u2019ll monitor the health of the system in order to identify issues, such as missed utterances. For example, you might want to explore an <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/building-a-real-time-conversational-analytics-platform-for-amazon-lex-bots\/\" target=\"_blank\" rel=\"noopener noreferrer\">analytics framework dashboard<\/a> or a <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/building-a-business-intelligence-dashboard-for-your-amazon-lex-bots\/\" target=\"_blank\" rel=\"noopener noreferrer\">business intelligence dashboard<\/a> to help spot gaps in utterance coverage or places where users exit early. Use this information to improve your interaction model, test the new design, and ultimately, tune your application.<\/p>\n<h2>Conclusion<\/h2>\n<p>In this series, we covered all the important basics for creating a great conversational experience using Amazon Lex. We encourage you to test and iterate through your design multiple times to ensure the best possible customer experience. Keeping these best practices in mind, we hope you explore all the different and creative ways that humans interface with the technology around us.<\/p>\n<p>And remember that we at <a href=\"https:\/\/aws.amazon.com\/professional-services\/\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Professional Services<\/a> and our extensive <a href=\"https:\/\/aws.amazon.com\/partners\/\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Partner Network<\/a> are available to help you and your team through the process. Whether you\u2019re only in need of consultation and advice, or whether you need full access to a designer, our goal is to help you achieve the best conversational interface for you and your customers.<\/p>\n<hr>\n<h3>About the Authors<\/h3>\n<p><strong><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/09\/15\/Nancy-Clarke.jpg\"><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-28180 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/09\/15\/Nancy-Clarke.jpg\" alt=\"\" width=\"100\" height=\"133\"><\/a>Nancy Clarke<\/strong>\u00a0is a Conversation Designer with the AWS Professional Services Natural Language AI team. When she\u2019s not at her desk, you\u2019ll find her gardening, hiking, or re-reading the Lord of the Rings for the billionth time.<\/p>\n<p><strong><a href=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/09\/15\/Rosie-Connolly.png\"><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-28198 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2021\/09\/15\/Rosie-Connolly.png\" alt=\"\" width=\"100\" height=\"89\"><\/a>Rosie Connolly<\/strong> is a Conversation Designer with the AWS Professional Services Natural Language AI team. A linguist by training, she has worked with language in some form for over 15 years. When she\u2019s not working with customers, she enjoys running, reading, and dreaming of her future on American Ninja Warrior.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-16190 alignleft\" src=\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2020\/09\/23\/ClaireMitchell.jpg\" alt=\"\" width=\"100\" height=\"133\"><strong>Claire Mitchell\u00a0<\/strong>is a Design Strategy Lead with the AWS Professional Services AWS Professional Services Emerging Technologies Intelligence Practice\u2014Solutions team. Occasionally she spends time exploring speculative design practices, textiles, and playing the drums.<\/p>\n<p>       <!-- '\"` -->\n      <\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/aws.amazon.com\/blogs\/machine-learning\/part-3-how-to-approach-conversation-design-with-amazon-lex-building-and-testing\/<\/p>\n","protected":false},"author":0,"featured_media":1444,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1443"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=1443"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/1443\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/1444"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=1443"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=1443"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=1443"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}