Automating customer email processing

Context

One of our clients had a large volume of customer support messages, that were handled by a support team. Since customer requests could be on a number of a different topics, typically first they needed to be categorized by a first person, so it can be routed to the correct recipient. The problem is then this means that each email is processed twice, and each manual step introduces more delay in the response to the customer request.

Another question is whether some customer requests could be fully processed by an AI.

Solution

The first requirements for such a project is to have sufficiently of labeled data to train a machine learning (ML) model. While it's possible to use a large language model (LLM) to categorize messages in a non supervised way, it would not necessarily reflect best business practices. A second step is to design a clear classification plan. It's important that different categories don't overlap otherwise the model will have a hard time distinguishing between them.

Then a ML classification model can be trained and evaluated on a test set. For evaluation, aside from standard metrics like accuracy, precision and recall, it's important to include business metrics, such as the time saved by the support teams vs the cost of miss classification. For this use case, you would typically want to run the model locally, as it deals with personal data of your clients.

Once we have the classification scores for each category, it's then possible to decide,

• which messages can be routed to the correct recipient without human intervention, if we are confident about the model's prediction and the cost of a mistake are not high
• which messages could potentially be answered by AI, in the case of general questions
• which messages need to be handled by a human

Overall this project allowed our client to significantly reduce the time it takes to process customer requests, and to reduce the cost of the processing.