TOC-automation — a new tool for IT support at the highest level

A TOC is a technician on call. Essentially, it’s a highly qualified engineer who supports a customer’s IT infrastructure outside the standard business hours, in the evenings, during holidays or at weekends. A technician on call doesn’t have to come into the office but they have to be on call at their business phone number at all times. If there is a problem with the IT infrastructure of a customer, the technician on call gets a call from the on-call engineer of the monitoring service, which operates in 24/7 mode. After the technician on call gets the call, they connect to the customer’s system remotely and fix the issue.

TOC-automation is a system that is directly integrated into the customer’s IT service management and monitoring systems and, whenever there is an incident, the system automatically notifies the technician on call. The kind of incidents we’re talking about here are IT infrastructure issues that can affect the customer’s business processes and cause serious damage to their business.

Essentially, a TOC automation system allows us to eliminate the need for a monitoring service operator to manually call the technician on call because the notification will be automatically sent by the TOC-automation system. The system is programmed in such a way that if the call to the technician on call fails to go through, it automatically escalates it to the project manager. All relevant information about the incident is simultaneously sent to the technician on call by text.

The history of TOC-automation began with the establishment of an initiative research group. The group started its work in July 2019 and had completed it by September of that year as it had achieved all its goals.

Over a period of two months, the team studied and analysed foreign notification solutions. The SaaS solutions currently available in the market were scrutinised, such as PagerDuty and Voximplant. These solutions have very good tools for notifying staff of IT incidents but they are extremely expensive. The group decided to develop an in-house system and soon had a prototype up and running. Then a proof-of-concept system was implemented for one of the accounts, which demonstrated the viability of the solution.

Once the project management approved the project and it was confirmed that a minimum viable product was ready for deployment, the business case for the project was successfully pitched for deployment on 4 accounts. The results of the deployment of the minimum viable product once again demonstrated that the solutions selected were correct and that there was a demand for this kind of system and it could potentially save the company a lot of money. Since 2019, a lot of new functions have been added to the system. Today, in addition to calling and texting technicians on call, the TOC automation system can also email them and send messages to a pre-selected Telegram channel. Furthermore, the security policy has been improved, a reporting module has been developed and, in order to abide by the Service Level Agreement, functionality has been added to notify the technician on call only after the expiration of a pre-set waiting period. A special algorithm has also been developed to prevent technicians on call and management from getting spammed with messages in the event that a customer’s IT infrastructure experiences a series of recurring incidents of the same type.

One tangible advantage of TOC-automation lies in how flexible and customisable the system is so that it can be easily adapted to the needs of specific customers. Thus, the original solution was used to develop an automatic notification system for field engineers for the 1C Snaryad system (by phone and text). In the context of the deployment of specific solutions, one important benefit is the close integration of TOC-Automation with the popular 1C platform, allowing the system to be set up to make use of the on-duty schedules of the technicians on-call, the service time of engineering teams and make calls, if need be, to personal phone numbers of the engineers retrieved from their employee records in 1C.

We’ve already deployed TOC-automation for 17 customers and for 2 in-house projects at ICL Services. The main effect of the deployment of TOC-automation has been in improved efficiency of our 24/7 monitoring engineers and dispatchers as a result of the automation of phone notifications. Current estimates suggest that, at the moment, TOC-automation is saving us 6.81 full-time equivalents, allowing us to significantly boost the efficiency of our monitoring stations. Prior to the deployment of the system, every monitoring station was handling 4 projects. Now, each is handling 6–7 projects. In addition, the system drastically reduces human error and thereby improves the quality of service.