A Robot Instead of First Line Support

The system partially performs the functions of the first line support, it analyzes the incoming requests of users who complain about some problems. The query is generated in natural language, but using a problem classifier consisting of more than 450 categories. If the robot believes that it has understood the essence of the problem (in about 40% of cases), it assigns a task to the employee of the second line or sends a request to the agent with a description of its version of what happened. The agent who has received the recommendations agrees or disagrees with the robot’s opinion, and then sends the request to the second line.

On average, two hundred restaurants connected to the system generate more than 7 thousand requests per month. The robot does «its» part of the job in 22 seconds instead of few minutes that are usually required by a «live» first line of support. Initially, the robot was used only for making recommendations, but in the last three months it has been also independently appointing employees responsible for processing the request. Of the 40% of requests, for which the robot has made a decision in the last month, 16% ended with an independent appointment, 22% resulted in a correct recommendation, and there were only 2% errors.

Computerworld Russia spoke with Mr. Kashtanov about the project, its current capabilities and prospects.

— Seven thousand requests a month that you talked about, do they come through all the channels or just through your system?

This is the total number, but the system processes about 6.5 thousand, that is, the vast majority of requests.

— What is the peak load?

About 500 requests per day.

— The robot makes a decision in 22 seconds. Is it «pondering» the request during this time?

This is the average time between the moment when the request was registered in the system and the moment when the robot made a decision, that is, passed the request to a particular department of the support service. Part of this time is spent by the robot to «notice» the request; the robot tries to extract new requests from the system approximately every 15 seconds.

— Does the system take into account the position of the user?

That’s part of it. The position is determined on the basis of dictionaries, and the system determines which requests for services or problems are allowed for such a position on the basis of the data analyzed during the learning. For example, the system distinguishes between back office and front office employees. For example, an accountant will not ask questions about the cash registers, and the work of the cashier is not related to the financial management system.

— Is the robot able to respond to a burst of similar requests — for example, if one restaurant sends many requests at the same time, or different restaurants send many requests on the same topic?

At the moment, such mechanisms are not used, the neural network analyzes each request separately.

— That is, are such capabilities enough?

Absolutely. To be honest, complex models are more suitable for winning contests; as far as I remember, once a contest was won by a team that created a solution using 113 different algorithms. But if you solve a practical problem, it makes no sense to chase the increase in the percentage of correct decisions at any cost. It is necessary to understand what accuracy is enough for your solution to be most effective in operation and inexpensive to support, and whether it is advisable to further complicate it, whether it will pay off, or increase in the cost of operation will completely neutralize the increase in accuracy.

— At least approximately, what can be the criterion that further improvement is ineffective?

Any system must pay off. Let’s say we want to return the investment in a year or so, rather than in 10 years. Based on this, we determine whether it makes sense to continue to improve the system.

— Is it difficult to retrain the system to work in other areas? And how scalable is it?

The methodology itself is universal and can be used in many areas. In some areas, it may work better than in IT, and in others its performance will be worse.

Now we distribute requests to 450+ categories. If there are, say, a thousand of them, the robot will be even more effective, because a human simply will not remember so many categories and will not be able to distinguish many situations, especially if they lie on the border between the categories.

As for the number of requests per day, the available features of the system are much larger. Categories can also be much more numerous, but in this case it will be necessary to train longer to achieve an acceptable result, otherwise it will be difficult to identify possible nuances. In addition, we will need more data, and the customer cannot always provide them in the required quantity.

In this project, we quickly got the initial dataset, but it was not enough. We had to perform a deeper analysis to be able to, for example, use the context of the customer’s request.

When the data is ready, learning of the model itself takes a few minutes because text records are used. If we used images, the learning process would be a hundred times longer.

But image analysis is one of the possible directions of the system development. Requests for technical support often contain screenshots, and we have had cases where support staff expressed dissatisfaction about the fact that the request was sent to their group, although «the screenshot shows that the problem is quite different!».

We are now investigating the possible effect of image analysis. If improvements are not sufficient, we will focus on improving the text. But if we find out that the analysis of screenshots significantly increases the share of correct decisions, we will continue this work.