Current status on NordAxon's solution Emely, a tool for practicing speaking interactively

This article was originally <a href="https://medium.com/@nordaxon/emely-speaking-practice-with-focus-on-interaction-3c79ec2a5468">published on medium</a> on the 14 September 2022.<h2>Introduction</h2>There are many tools available for learning new languages, where the focus usually lies on learning words, grammar, and pronunciation. During 2020, Helsingborg AMF together with <a href="https://www.nordaxon.com/">NordAxon</a> identified additional barriers to learning a new language related to interaction. One of the best ways to learn a new language is to speak it and interact with others that are also trying to learn or that are already fluent speakers. This can be very challenging for people that have limited access to conversation partners, limited time to seek these out, or simply do not have the courage to do so.Having a job is an efficient way to learn a local language since it often involves a lot of interaction, but to get a job it is usually required that you are already a fluent speaker. The idea brought together by Helsingborg AMF and NordAxon is to make interactive conversation practice more accessible, in the same way as other parts of language practice have been made accessible already.<img class="image_resized" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/blog%2FEmely-Interactive%20Speaking%20Practice%20using%20ML-powered%20Conversational%20Agent%2Femely-screen.jpg?alt=media&amp;token=6473b5ab-1a19-47f2-8a7d-3d9cf43ee355" /><h2> </h2><h2>Emely</h2>Emely is a tool for conversation practice made available as a web application. The first iterations of Emely have been focused on job interview practice and common conversations on varying levels of difficulty. The user interacts with Emely through a chat-like interface, where the user records their response, and Emely in turn responds through speech synthesis via the microphone, as well as displaying the response in the chat as text.Since Swedish is not yet a “high resource” language within the field of NLP (Natural Language Processing), we are for now using English as an intermediary language for certain parts of the functionality (“high resource” in this context means that there would be many useful datasets and previous modelling work performed in Swedish). In the image below we show the architecture of the conversation flow that is used in the current version of Emely. A response is generated through the steps 1-6 indicated in the image. The steps are:<ol><li>Speech recognition – turns the user input into text.</li><li>Translation – we make use of translation from Swedish to English.</li><li>Dialog control – we control the flow of dialogs differently based on scenarios, and how the conversation progresses.</li><li>Generative Dialog Model – a machine learning model generates an answer using user input, previous conversation history, as well as additional information from the dialog control.</li><li>Translation – the response from Emely is translated back into Swedish.</li><li>Speech synthesis – text is transformed to speech which is then played on the user’s device.</li></ol><img class="image_resized" src="https://ik.imagekit.io/nordaxon040/tr:w-1000/v0/b/nordaxon-webapp.appspot.com/o/blog%2FEmely%20%E2%80%93Speaking%20Practice%20with%20focus%20on%20Interaction%2Farchitecture.png?alt=media&amp;token=189af527-f4e6-47dc-9c8b-16bae6e18675" />In addition to these steps, we analyze conversations, considering both how well a user is performing, and how well Emely is performing. The analysis of user replies can be visualized in the user interface, and further analyzed for improving language skills by yourself, in collaboration with a language coach/teacher, or in the case of job interviews a job coach.<img class="image_resized" src="https://firebasestorage.googleapis.com/v0/b/nordaxon-webapp.appspot.com/o/blog%2FEmely-Interactive%20Speaking%20Practice%20using%20ML-powered%20Conversational%20Agent%2Femely-screen2.jpg?alt=media&amp;token=26f6ce9c-8c64-4f5c-92c5-b95c41d77d4b" /><h2> </h2><h2>Job Interviews</h2>Large amounts of resources in Swedish municipalities are spent on helping people being able to find a job. Language skills is a large part of this task, where job coaches are working together with language instructors and teachers to help people improve their interactive language skills.Having job interviews is very different to having common conversations. One big difference is that the conversations are usually only being controlled by the interviewer, who is asking many questions and follow-up questions. To make a good impression at a job interview it is in most cases not enough to only respond to questions that are easily expected, but there might be an even higher weight on how you can answer on follow-up questions. Also, an interview can switch topics fast, but are usually centered around the person being interviewed.The Interview module in Emely was developed in collaboration with job coaches at Helsingborg AMF, ensuring that the above aspects were taken into consideration. The main idea in the current version of Emely is to steer interviews in the “dialog control” and utilize a custom generative dialog model that is specialized in asking follow-up questions. Therefore, the model has been trained using a large dataset with examples of good follow-up questions. An example of user responses and “labels” (a label is an example of what we want the model to respond in this specific context) from the training dataset is shown below:Emely:                            How do you think a good boss should be?User response:            A good manager is clear and coaching, gives responsibility and lets his employees come up with ideas and views.Emely:                           How should a boss not be?User response:            A manager should not be afraid of conflict or inaccessible.Emely:                           What can happen then? <h2>Research</h2>Several of the components in Emely is currently subject to research in the field of NLP. NordAxon is both actively engaged in this research relating to some of these components, as well as following the state of the art in NLP closely.One topic where NordAxon has been involved in research is in the quality assurance of generative dialog models. Quality assurance in this case is mainly focused on analyzing the quality of the generated responses and conversations. In other words, how can we determine that Emely is responding reasonably within the context, grammatically correct, as well as having overall good conversations.An initial framework for evaluating conversations was developed in a collaboration between NordAxon, Research Institutes of Sweden (RISE), and Lund University. A research article from this collaboration can be found <a href="https://arxiv.org/abs/2203.15414">here</a>, and an initial implementation has been made available as open source <a href="https://github.com/NordAxon/GDM-testing">here</a>. <h2>Emely on H22</h2>In the summer of 2022, Helsingborg held an innovation exhibition named H22 City Expo. The exhibition is described as “An international event that focused on future welfare and urban development solutions and how we can create a smarter, more sustainable, and thoughtful city – together” [<a href="https://www.h22cityexpo.se/">https://www.h22cityexpo.se/</a>]. A main attraction on the exhibition was speaking with Emely.During H22 City Expo more than 600 visitors initiated over 2.500 conversations with Emely. This meant that visitors spent in total 86 hours talking with Emely, sending 10.742 replies. Even the Swedish crown princess <a href="https://www.mynewsdesk.com/se/helsingborg/pressreleases/3-smarta-roliga-och-fiffiga-loesningar-3186558">Victoria had a quick chat with Emely</a>.The H22 City Expo was therefore an excellent opportunity for us to evaluate the current version of Emely and analyze user interactions. For example, it has helped to improve performance, UI/UX, and handling of the very varying quality of messages that is received at an exhibition. <h2>Prospects</h2>Moving forward, we are looking to expand the scope and usage of Emely in the following ways:<ul><li>User group – who could benefit from interactive language practice?</li><li>Practice scenario – which scenarios are relevant for practicing interactively?</li><li>Language – which languages other than Swedish could be useful to use with Emely?</li></ul>If you are interested in the functionality of Emely, or how Emely can be applied to practice speaking interactively in other scenarios, please get in touch with us at info.nordaxon.com.