Toshiba and Toshiba Digital Solutions Develop AI for Road Surface Anomaly Detection that Contributes to Maintenance and Long-term Operation of Expressways and Demonstrates
Real-time Detection of Potholes that Pose a Risk of Serious Accidents

Currently, more than half of expressways in Japan came into service 30 years ago or earlier. Expressways are subjected to continuous wear due to high-speed, high-volume interactions between the road surface and vehicles, and the frequency of road surface anomaly due to aging has increased markedly in recent years. Approximately 3,200 potholes large enough to cause loss of vehicular control and thus to pose a risk of serious accident were detected on expressways managed by NEXCO Central in 2019. Pothole generation is characterized by negligible signs on the road surface in the early stages, and relatively rapid development of damage once anomalies become detectable in the road surface. Therefore, early detection and repair are essential.
Although highly efficient and accurate routine inspection is crucial for preventing accidents due to road surface anomalies, the road surface is currently inspected together with other inspection targets (e.g., road signs) visually by inspectors who ride in an inspection vehicle. When the inspector finds a pothole requiring urgent repair during a routine inspection, the vehicle is driven to a safe place to park: the inspector then gets out of the vehicle to walk to the pothole, takes pictures of it for reporting to the road control center, and arranges for urgent measures. When parking the vehicle near the pothole is difficult due to its location, the inspector must go to the next interchange, exit the expressway, and circle back to return to the location of the pothole. This results in considerable time loss. Also, visual inspection leads to variation in inspection quality.
If AI can be used for automated real-time detection and reporting of potholes on images acquired by a camera installed on a vehicle while driving, it will even out the inspection quality and also considerably reduce the time for urgent road repairs while ensuring the safety of personnel. However, such AI conventionally requires a large number of pixel-wise annotated images. Thus, considerable time and high cost are required to prepare training data. Also, application of conventionally trained AI to different types of anomalies and conditions is difficult, as there is the problem of many false negatives and false positives.

To address the issues above, Toshiba developed weakly supervised AI technology that predicts the presence/absence of anomalies and their positions based on the pixel-level anomaly score, and this is done by using training data labeled only with the presence or absence of an anomaly. This AI uses a deep learning model that outputs an anomaly score map of the input image. It learns to match the maximum value of the anomaly score map with the presence of an anomaly on the input image. Thus, an anomaly score map is output in which there is a high score at the position of an anomaly (i.e., the position where there is a difference in characteristics between a normal image and an anomal image) (Figure 2). This AI can reduce the time required to prepare the training data per image to approximately 1/100th of that using conventional technology (approximately 1 sec using this technology vs approximately 1 min 40 sec using the conventional technology.) (Figure 3)(*2).
Using this AI, Toshiba Digital Solutions developed a system that analyzes images captured by a camera installed on a vehicle and shares the information with the road control center in real time. In a verification experiment for improving routine inspection of expressways, which was conducted with NEXCO Central starting in September 2022 (Figure 4), Toshiba and Toshiba Digital Solutions collected road surface images while a NEXCO Central vehicle was driven on the Shin-Tomei Expressway and the Tomei Expressway within the coverage area of the Isehara Maintenance and Service Center, and prepared training data labeled only with the presence or absence of a pothole using knowledge held by NEXCO Central. Use of this world-first (*3) pothole detection model additionally trained with the above data improved detection accuracy to a ROCAUC (*4) of 84.22%, compared to 61.25% when the conventional model trained with images of ordinary roads was used (Figure 5). It was demonstrated that real-time detection of the position of a possible pothole can be achieved while driving at 80-100 km/h with a lower false positive rate of 4% compared to 77% with the conventional technology (Figure 6). This verification experiment demonstrated that the road surface anomaly detection AI developed by Toshiba and Toshiba Digital Solutions achieves highly accurate real-time detection of potholes, and that this accuracy improvement can be achieved with a markedly reduced workload for the preparation of training data, showing promise for real-world application in FY2024.

Figure 5: Speed and performance of pothole detection in the verification experiment

Toshiba and Toshiba Digital Solutions will continue conducting verification experiments with NEXCO Central to improve the accuracy of detecting potholes that require urgent repair, so that real-world application of the pothole detection system can be achieved in FY2024.

*1: Development of the world’s first AI model for pothole detection using weakly supervised learning with public data sets of ordinary roads was presented at Vision Engineering Workshop 2022 on 8 December 2022 (according to a survey by Toshiba as of September 2022).
Satoshi Ito, "Detection of various types of road surface anomalies using multiple instance learning”, Vision Engineering Workshop, OS1-O3, Dec. 2022. https://view.tc-iaip.org/view/2022/ (in Japanese)
https://www.c-nexco.co.jp/corporate/csr/csr_download/documents/2023/nexcocsr23_2_all.pdf (in Japanese)(20.14MB)
*2: Calculated based on average working time by researchers
*3: Development of an additionally trained model using weakly supervised learning with expressway data, verification of the use of the AI in an expressway setting, and demonstration of the effectiveness of the technology were achieved in a world first (according to a survey by Toshiba as of August 2023).
*4: An index used for evaluation of anomaly detection algorithm. The false positive rate (x-axis) was plotted against the true positive rate (y-axis) at various threshold settings, and the percentage of the area under the curve was calculated. A higher percentage indicates higher accuracy.