You have probably heard of LLMs – Large Language Models – by now, or at least ChatGPT. If you haven’t, well, you should check it out, it’s pretty cool! But what is it like to use LLMs in a launched product? What is prompt engineering and what challenges are there?

Spoiler alert: The challenges are many and complicated !!!

In this article we are going to look at these challenges from the perspective of working on Emely – NordAxon’s chatbot for language learners. We will discuss what we have learned from our experiences and show the internal tools we have developed to make our prompt engineering work easier.

But first, a brief overview of what Emely is and how it works. If you’re curious about the background and how it started, check out our earlier blog post about Emely.

What is Emely?

Being new – especially to a language – can be daunting. For some it’s an anxious experience. Others may dive headfirst into new situations. Even so, if you don’t know the language, then how will people know how good your jokes are?

When you’re new in Sweden, there are language courses available called Svenska för Invandrare (SFI; Swedish for Immigrants), but the hours are limited. Wouldn’t it be nice to be able to practice hearing Swedish, and speak Swedish – on your time?

Enter Emely – a conversational agent, or chatbot in common parlance. It communicates in text and speech, and users can reply with speech or text as well. The interface mirrors the design of familiar chat apps, such as WhatsApp, to enhance user comfort and ease of use. We deliver the Emely app as a browser application, so it can be used on a regular desktop or laptop computer as well as on your mobile phone.

Emely's start page after logging in. Users can click “Översätt” to translate into their native language.

Emely started as a collaboration with the city of Helsingborg to help their citizens practice job interviews. Today, we have four main versions of Emely; job interview, SFI class, preschool teacher, and fika (see below for explanation). Our customers now include not only local government but also companies with staff that are new in Sweden. We call these versions “personas”, and each one is individually tuned to provide a context or situation suitable for different groups. For some of these personas we have multiple sub scenarios.

In the Interview persona, Emely plays the role of a job interviewer. We have dozens of common jobs to pick from, and for each the purpose is for the user to practice being grilled in an interview. A challenge here is to find a balance between bot responses being chatty and giving feedback, while at the same time being somewhat strict and professional.
Fika, if you’re unfamiliar with the word, is a Swedish concept that means – roughly – to meet someone for a coffee and chat. Ideally, cake and/or biscuits are involved. Fika can happen at work or between friends or even mean a romantic date so it really depends on the context. In other words, the Fika persona is designed to facilitate the practice of Swedish in a relaxed manner.
SFI, as explained above, are official language learner classes for people new in Sweden so here we have multiple short and focussed scenarios like going shopping or visiting the doctor.
Finally, Pre-school is a persona meant for preschool staff where they can practice speaking to parents and colleagues.

For all personas, there are prompt engineering challenges in balancing language level, coherency, and fun during a conversation. We will come back to this in more detail later when we talk about prompting.

An advantage of our use case is that we don’t need to worry too much about LLM issues like hallucinations. We just want dialogue in Swedish, which we get 99% of the time (but see below about emojis) and it’s grammatically correct almost as often (but we do run an extra check for grammar and currently collecting data on how often that’s used).

Behind the scenes

The backend architecture of Emely has not changed significantly since our last blog article. The main difference is that today we use OpenAI’s GPT3.5 Turbo (called GPT henceforth). Since GPT handles Swedish surprisingly well, we no longer need to rely on translation of messages. This was required previously as most models were primarily trained on English or languages far bigger than Swedish. We rely on Azure Text-to-Speech and Speech-to-Text which according to our studies works better than other services for Swedish. However, since many of our users are Swedish beginners, pronunciation can be difficult to interpret. We have an ongoing project to train our own models for Speech-to-Text for Swedish dialects and beginners, which will be the topic for another future blog post, so watch this space!

More specifically, we use the Azure OpenAI service as our main model API. Azure can host GPT models in Sweden, enabling lower response times than OpenAI’s own API. We chose GPT because it’s well trained to follow instructions and provide non-toxic answers, with the latter being very important for our use case. Using another model, like Mistral, would be more costly, and require more hands-on coding to build content filters for responses. Further, we found that while GPT4 provides slightly better-quality answers than GPT3.5, the difference does not motivate the 20x difference in price. In other words, GPT3.5 is the better choice for us with regards to price/performance.

Currently Azure OpenAI uses a content filter that we cannot turn off. Only large organisations with their own Microsoft representative are able to. We can only set the content filter to high, medium, or low. We have set the filter to low, but we suspect the filter translates everything into English – and not very well. This suspicion arises from the common occurrence that perfectly reasonable Swedish is blocked by the Azure content filter. For example, the sentence “Jag har passat min kusin” (I babysat my cousin) will get blocked 100% of the time due to “sexual content”. We suspect it is being translated into something like “I made a pass on my cousin”. For this reason, we catch when Azure blocks our request due to content filtering and ask for a reply from the OpenAI API instead.

A downside of using GPT – and many other models as well – is that we do not know what data they were trained on. If we knew this, then it would be easier to prompt for certain knowledge and/or behaviours. For example, say that we know the dataset contains examples of a doctor diagnosing a patient by asking questions. If we then get bad results prompting the bot to conduct such a conversation, we may continue adjusting our prompt until we find one that reliably returns interactions. But if such examples are not in the training data, we can pick another strategy for our prompting. Examples of such alternate strategies are providing examples in the prompt or creating a dataset and finetune the model.

Why is this important? Because prompting is very hard.

The black magic of prompting LLMs

Now this is the fun part. I say fun because I’m a masochist. Okay, that’s overly dramatic. Some responses from Emely have made me laugh out loud. But then my colleagues ask if I’m all right. They can hear the sudden release of frustration in those laughs.

To understand this, we need a brief primer (for a great overview with more details, check out Karpathys LLM intro). LLMs generate text – output – that is statistically likely based on what input you provide. If your input is “What is the capital of Sweden?” then it is highly likely that the output of the LLM will contain “Stockholm”. Of course, that assumes that the training data contained such example text. If the training data instead contained many statements that Malmö is the capital of Sweden, then Malmö would be more likely to be part of the output. Either way, the input to the system is commonly referred to as the prompt.

However, when we have conversations with Emely, ChatGPT or other LLMs, we also have multiple messages. In these cases, we have a general prompt (often referred to as the system prompt) with instructions on how the system should behave. The system prompt, together with the list of messages, create what is called “context”. So, for each message in a conversation, our context grows but the system message – how the bot should behave – stays the same. Technically, it is possible to change the system prompt whenever we want but for our discussion here let’s keep it simple. What is important to understand is that the next reply from the bot will be based on what is statistically likely to follow based on the entire context.

As an example, when Emely interviews someone for a job, our system prompt is something like “Your name is Emely and you are interviewing the user for the job as lorry driver”. To get conversations going, and avoid users being anxious or hesitant about what to say, we always have Emely start. So the first message would be “Hello my name is Emely, welcome to this interview for the position as lorry driver. Are you ready to begin?” The user then replies and maybe they say “Thank you, yes I’m ready!”. We then send the system prompt together with Emely’s first message, and the user’s message, this is the context the GPT model receives. Based on this context, we get a reply.

Finally, we also have a few parameters we can adjust to tune the output. Temperature controls randomness of the output. At its minimum output is almost deterministic but also boring with basically no variation, and conversations become less dynamic. Towards the maximum the output is more varied and creative but there is also a high risk of getting gibberish output. Another important parameter is the frequency penalty. Increasing this value will penalise new words if they already exist in the context, thus increasing variety of output.

Taken together, the above means that when we ask GPT for a reply in an ongoing conversation, the answer we get will depend on multiple factors: the system prompt, the previous messages, and our parameter settings. We have to combine these factors, try to predict how conversations will go, and craft a system prompt that will find a spot in a highly dimensional space from which to start the generation of words. This is why prompt engineering is hard. To explain this process in a more tangible way, let us imagine that prompt engineering is like fishing.

I strongly suspect that the fish are hiding in the eastern part of the lake. So I throw my line and bait there. I don’t know exactly what I will get when I make a throw, I will have to wait until I see what comes up. Maybe I get a fish, but is it the right kind of fish? Maybe I get a shoe. I’m not an expert fisher - I don’t even eat fish - but I suspect there are many additional variables here like time of day, wind direction, if I’m singing songs to the fish, and so on. Prompting LLMs is similar. We can’t say for sure what we will get, because it depends on not only our system prompt but also the rest of the context. The same prompt and different greeting phrases can produce very different responses. Most of the time we get fish, but with a specific combination of variables, we will get a shoe instead.

This is why Prompt Engineering is Black Magic. One of our main challenges is to make Emely stay in character. (Chat)GPT is heavily conditioned to answer questions, but most of the time we want Emely to ask questions. This is important not only for playing the role as a job interviewer but also for driving regular conversation. As you may know, there’s a back and forth between conversational partners. Staying in character extends to this back-and-forth, where certain combinations of message exchanges can cause Emely to switch roles. Sometimes it's obvious, and sometimes less so.

For example, one of our personas is meant for preschool teachers to practice speaking with parents who come to pick up their kid. The following is a translation of an actual conversation, where Emely plays the parent and the user is a preschool teacher:

Emely: Hey Angel! You all right? How's your day been?
User: Good, I think.
Emely: That sounds good! Can you tell me how Aida has been today? Has she been happy and active, or maybe a bit quieter today?
User: Aida has been very active. She has been painting elephants in different colours and using various paints.
Emely: Oh, that sounds lovely! Aida really loves to paint. Has she played with the other children today?
User: Yes indeed. Aida plays a lot with the other children. I was wondering; could you continue talking about colours with Aida and support her in practicing using them?
Emely: Absolutely! That sounds like a fun idea. We can definitely continue to encourage Aida to explore colours and express herself through art. We usually have different activities where the children can learn about colours and use them in various projects. I'm sure Aida will find it really enjoyable!

Did you spot the role switch? Emely has been instructed – prompted – to roleplay a parent picking up their kid from preschool. Yet in the last message, we get a reply that sounds like it’s the preschool teacher talking! But it’s subtle. Reading the conversation casually, one might miss it.

So we have to make sure entire conversations are coherent. On top of that, very small changes like punctuation in a prompt can dramatically alter behaviour. We had one instance where the difference between “, word” and “. Word” caused the output to be what we wanted 100% of the time in one case, and only 50% of the time in the other. With 50-100% we mean running the same dialogue to generate a reply 20-100 times in order to find potential outliers in replies. We have found that running tests at least that many times is needed to be reasonably sure we do not run into edge cases.

We also want our replies to be short and use simple Swedish. A simple way to keep answers short is to adjust the parameter for output length. But this often cuts sentences short and is thus unreliable. So the prompt has to instruct GPT to “keep things short” and find the variant of that instruction that produces the output we want. We have used various forms of “You are to reply with a maximum of two sentences” or one that works fairly reliable (probably due to what is contained in the training data) is “your reply may not be longer than a tweet”. A humorous side effect of the latter is that in rare cases this may cause the output to contain emojis.

Getting the model to use simple Swedish is even more tricky. We have used various forms of “use simple language”, “use language appropriate for that of a childrens book for five year olds” or “you are a beginner in the Swedish language and thus use very simple words”. Again, which one will work is highly dependent on the context, persona and scenario. We believed the childrens book worked okay, but later discovered that in some cases it will cause the model to reply as if it was a five year old. Like calling the user mommy, or asking for lollipops. Yeah, that happened. More commonly, it's more subtle like our pharmacy character asking the user if they had tried cough medicine for children before, even though nobody ever mentioned being a child or the medicine being for children. We have not yet found a prompt that reliably produces output at a certain language level suitable for (complete) novices.

Whats the takeaway here? Roughly, it's to avoid mentioning things in the prompt that you do not want. Anything mentioned in a prompt may cause the LLM to fetch words or concepts related to those words in the vector space. In our example above, it fetched things related to children.

Another quirk is that there is no situational awareness in roleplay. For example, the bot may suggest to go look at something, like showing the customer where all the red sweaters are in the clothing store. There’s a very high probability that we then get stuck in a loop of “yeah lets go!” – the conversation continues without any “action” being taken so to speak. The user would have to understand themselves that it is not possible for our bot to actually show them a red sweater, and redirect the conversation themselves. Which can be hard to grasp even if you know the language well.

All these issues cause our prompt engineering work to be difficult and time consuming. We cannot simply do tests with single replies, we have to test entire conversations. Our main way of doing so is to manually conduct conversations, just like our users do. And because the context – the conversation itself – impacts the replies, the possible combinations for how a slight change in our prompt text may impact results are huge.

We do also generate entire conversations. This uses a function built into Emely's UI where the user can get a suggestion for what to say. Apart from having to tune this prompt by itself, we also use it to have Emely and the “user” generate a full conversation. However, this brings in additional complexity since we now have two system prompts that each will impact the result of the other.

If you happen to know a few things about machine learning and LLMs already, you may wonder if we have tried fine tuning. If you do not know what fine tuning is, then briefly, it is using an existing model like OpenAIs GPT3.5 and train it on data specific to one’s use case. The answer to whether we have tried fine tuning is that we have indeed thought of it, but not had the time or resources to put together a dataset and do said fine tuning. It is on our list, so watch this space for a future blog post reporting our results!

What we have done however, is we have created and are actively working on a toolkit to help our prompt engineering.

Our tools for dialogue prompting

Most frameworks we have found for testing LLM output do not consider coherency across an entire conversation. Mostly they are focussed on single examples of input -> output. What such frameworks often do is use another LLM to evaluate answers, but unfortunately, that approach does not work well for us. Even using GPT4, it will fail to spot the subtle role switch in our example conversation from earlier, and claim the conversation has great flow and substance. It is possible we may find a prompt that can spot such subtle errors, but as we have seen, finding the right prompt is a time-consuming process so we have not gone far down that route.

Instead, we have created a Prompt Toolkit for developing Emely personas and scenarios. Currently it is a collection of three different tools, all built using Streamlit. The first one you may be familiar with as it is almost a 1:1 copy of OpenAI’s Playground. We have an editable prompt, together with the current message history. Click the button to send a request and we get back a reply.

A screenshot of a computer

Description automatically generated

Screenshot of our Prompt Playground tool. We can load existing conversation from our database, or manually add messages to a conversation. Getting multiple replies back with one click makes it easy to get a feel - vibe - for how prompt changes impact variety of the output.

The main difference from OpenAI's playground is that we can get multiple responses in one go, as seen at the bottom of the screenshot. This makes it much easier to quickly get a general sense of the variability of your changes. We use this tool to troubleshoot specific conversations where things have gone wrong X messages into the conversation. For example, in the conversation example we showed above with the role switch, we could load that conversation into our playground tool and start experimenting with the prompt. We can also easily change the greeting phrase, the last message or any message in the chat to see how it impacts results. In this instance, we had to change not only the prompt, but also change the greeting message to always contain the name of Emely’s daughter. In other words, it’s often – but not always – necessary to really reinforce the role meant to be played. And the initial greeting phase is also very important as it sets the tone for the entire conversation. A good idea then, when you need to control conversations as much as we do, is to have a hardcoded greeting. To have some variation, come up with a few different hardcoded greetings. But not too many because then you will have a hard time testing them all.

The next tool in our toolkit is a conversation generator. As mentioned above, we use our suggestion functionality to have GPT essentially speak to itself but using two different prompts for the different roles talking. This allows us to get a feel – vibe – for how both of our prompts for the particular scenario behaves. In the example screenshot we have in the left menu bar our selection for number of conversations to generate, and the specific combination of persona and scenario/job. After generation is done we can see each of these conversations to the right, and switch between tabs for each conversation in the generated dataset. In this example we can see (if you do not know Swedish, then Adjö means goodbye) that Emely says goodbye to the user (parent) after only a few exchanges. This indicates we need to change our system prompt with instructions either to encourage continued conversation or disallow saying goodbye. Again, if encouragement or punishment like language in the prompt will work is highly dependent on all the variables involved. It may also be the case that this is a niche case and only happens 1/1000 times, but we would only know from generating 1000 conversations and read through them all. Although this particular scenario where we get “early goodbye” could potentially be a valid use case for an LLM based test evaluator.

Screenshot of our conversation generator tool. This gives us an overview of how our main prompt for Emely interacts with our suggestion prompt for users.

Finally, the third tool in our toolkit is a regular chat. It works the same way as the chat our users use on Emelys main website, but can be run locally on the developer’s computer and will thus pick up changes made to the prompt in our backend code. In the conversation I have screenshotted you can see that despite all the difficulties with prompting we have mentioned in this article, we have managed to prompt Emely to always reply in Swedish, as well as driving the conversation by asking questions. Success!

Screenshot of our manual debug tool. Here we can chat with Emely as if we were a regular user, ideally by coming up with potential niche things to say.

Conclusions/Learnings

As we have seen, prompt engineering is an art. Magic if you so will. It may sound frustrating and difficult, and it sure is at times. But the upside is that when you make things work you will feel like a wizard. Or maybe a warlock. I guess it depends on your preference for calling this regular magic or black magic. So far, at NordAxon, we have managed to avoid the bloodletting required for blood magic.

Our approach for taming the black magic of LLMs is to develop tools that make our job easier.

If you are curious about our toolkit, we have discussed releasing it as an open-source project. We’d love to do that, but it requires some work disconnecting it from our internal systems and cleaning up the modules so you can hook it up to your own LLMs. We’re quite busy developing Emely and working on multiple other projects with our AI-needy clients so do follow us on Github and/or Linkedin and you’ll be the first to know when we get around to releasing it.

So, over to you dear reader: Are you more or less excited about prompt engineering after having read this article? Let us know what you think by email (emely@nordaxon.com). We hope this article has given you insights on the daily life of a prompt engineer and hopefully also amused you.