Friday, June 18Digital Marketing Journals

Chatbot best practices – KPIs, validation and more

I’m stating the obvious here, but it’s really important to know what you want to achieve, and how this can be measured. Many of your KPIs will be sector or domain specific, but I will give you some chatbot specific KPIs to think about. Listed in order of importance:

  • Satisfaction rate — the percentage of those users who were satisfied. I generally recommend offering a binary choice “satisfied” vs “not satisfied” instead of a rating from 1 to 5, but the choice is yours.

Like most technology, a bot is designed to automate tasks that would otherwise be done by a human operator. Before embarking on a chatbot it’s essentials that you know exactly what you are trying to automate. The best way of doing this is to first employ human agents to respond to your users’ messages. Why do this?

1. Chatbot Trends Report 2021

2. 4 DO’s and 3 DON’Ts for Training a Chatbot NLP Model

3. Concierge Bot: Handle Multiple Chatbots from One Chat Screen

4. An expert system: Conversational AI Vs Chatbots

Ok, you need to be mindful of GDPR, so you can’t log everything. For your purposes we don’t actually need personally identifiable information. What you’re after is the phrases users use. In particular, you’re interested in:

  1. entities — yes, a jacket. Entities (relevant nouns) form the basis of named entity recognition
  2. parts of speech —I want a black or white dress”. The adjectives, prepositions and conjunctions. You will use these to train your part of speech tagging models.
  3. sentiment — I’m not happy” or “thanks for your help”. You can use text classification and sentimental analysis to detect when users are satisfied or dissatisfied.

If you’ve followed our first piece of advice, you should have some decent training data. Now it’s time to put it to use.

  1. body — e.g. “last Monday”. These messages may contain additional entities
  2. sign-off — last couple of messages e.g. “thanks for your help”. Useful for sentimental analysis

Experienced IT professionals think carefully about validation and error handling when building apps or websites. You can usually rely on the UI to help enforce constraints. For example, by using a dropdown select box with the valid options. The challenge arises when trying to enforce the same constraints in a chatbot.

Quick replies

Some channels offer quick replies— prefilled responses which can act as a replacement for select dropdowns, radio buttons and checkboxes. Quick replies can be used as a means of constraining user behaviour, but should be used with care. Unlike dropdown boxes, the options are typically displayed horizontally or vertically and take up valuable screen real estate, especially on mobile devices. This makes them suitable for responses with only a few options.

Message validation

Free text entry is at the heart of a chatbot. It’s unconstrained, so good validation and error handling is especially important. Remember — whilst your NLU model may correctly identify an entity, this doesn’t mean your downstream systems can handle it. 100 pounds or last monday are examples of entities that an NER model will probably recognise, but need transforming for downstream consumption.

Here’s the typical chatbot flow:

  • Process reply
Agent: hello how can i help?
User: hi
User: i want to check my order status
User: order A123
Bot:  hello how can i help?
User: hi
Bot: Sorry i dont understand
User: i want to check my order status
User: order A123

Support chit chat

Basically you train the chatbot to recognise “chit chat” type messages, which it can either reply to or simply ignore. Taking the example above, the bot would either ignore the “hi” or reply with “hello”. Either way, it wouldn’t generate an error.

Buffer incoming messages

Buffer all incoming messages. Wait until N seconds have elapsed since the last message. At this point concatenate all the buffered messages together into a single message and process it. Taking the above example it would look like:

Bot:  hello how can i help?
User: hi
User: i want to check my order status
User: order A123
(wait N seconds)
Bot: Ok …

Buffer but short circuit

The same approach as described above, but instead of always waiting N seconds, you try to process the message buffer every time a message is received. If you can process it, you do so immediately, avoiding delay. Going back to the contrived dialog it would look something like:

Bot:  hello how can i help?
User: hi
(can’t process — wait)
User: i want to check my order status
Bot: Ok what is your order number?

As well as validating each user response, you will want to set up various “checkpoints”. This means telling the user what the bot has understood and asking them to confirm this. For example saying something like:

Drill down

We call the first strategy the “drill down” approach. Start out by asking users open questions e.g. “how can I help?” or “what are you looking for?”. Run the responses through the NLU models and algorithms and checkpoint the conversation.


We call the second approach the “bailout”. Put simply if you can’t understand the user’s needs you fall back to human intervention. See below for more details.

What can you do with the outliers? Firstly it’s important the system recognises when it’s failing to meet the user’s expectations. For your users, there’s nothing worse than talking to brick wall. One way of detecting this is to count the number of “sorry I don’t understand” type responses generated for each dialog. As mentioned above, checkpointing is also very important.

The reason you’re logging the conversations is to build up training data, allowing you to build accurate models. To borrow a cliché — this is a process not an event. Whilst the data captured during the initial “human” stage gets you started, you need to retrain the models as you collect more data.

Chatbots are freeform, users can say whatever they like. This presents challenges but also opportunities. Chatbots are great for market research.

Implementing an enterprise grade chatbot requires careful planning. It’s important to understand the KPIs and business drivers before embarking on the project. Having a means of measuring success is also really important.

Leave a Reply