Toshiba Develops Infrastructure Inspection AI for Detecting Anomalies from Only a Few Reference Images with Unprecedented Accuracy

TOKYO—Toshiba Corporation (TOKYO: 6502) has developed artificial intelligence (AI) for detecting anomalies during infrastructure inspections. With only a few reference images of inspection targets and no real-world training, the technology can accurately detect cracks and rust as well as leaks, adhesion of foreign material, detachment of parts, and other infrequent, untrained anomalies.
This new AI not only reduces inspection workloads—particularly those that imperil workers or require travel, such as atop steel towers in the mountains, under elevated bridges, on slopes, and under solar panels—it also performs well in real-world situations where the scarcity of images from the field has hampered past efforts to use AI.
The technology uses features from pre-trained deep learning models to compare inspection photographs against reference images, and thus does not require real-world training as in the case of conventional AI. Toshiba’s proprietary correction technology enables the new AI to detect anomalies with high accuracy even when inspection photographs are taken from different angles than in the reference images, and also limits false positives in the case of unique patterns that are actually normal. In an assessment based on a public dataset, the AI was 91.7% accurate, the highest figure ever recorded.(*1) The new technology automates inspections, reduces workloads, and expedites the discovery of anomalies, helping ensure stable long-term operation of infrastructure.
Toshiba is scheduled to give a presentation on the details of this new AI on May 25, 2022 at ICIAP 2021, the 21st International Conference on Image Analysis and Processing, in Italy.

Infrastructure maintenance is becoming more important as society aims to position infrastructure for stable operation over the long term. Japan is facing a combination of infrastructure-related problems. Roads, bridges, tunnels, and other infrastructure built 40 to 50 years ago during the country’s period of economic growth are aging rapidly. Inspectors tasked with maintaining the infrastructure are also aging, and replacements are not forthcoming amid a shortage of labor. There is also a need to reduce the workloads of inspectors in dangerous places. Thus, AI is needed to maintain infrastructure more safely and efficiently. Early detection of the many unspecified anomalies is essential for streamlining infrastructure maintenance. If anomalies can be detected automatically from inspection photographs taken by drones or robots, inspection will not require as much work and anomalies will be detected sooner.
For instance, cracks, rust, and other specified anomalies can be detected by creating a model based on a large number of images of various types of cracks and rust as well as reference images, and then training AI technology in crack and rust detection. However, there are many more types of infrastructure anomalies and defects that lead to the anomalies, including water and oil leaks, detached members and parts, and adhesion of foreign material. There are many conventional ways to detect these unspecified anomalies, including preparing large amounts of training data on each of the many anomalies and training the AI, comparing differences in the brightness of precisely oriented reference images and inspection photographs, and using an auto encoder to reconstruct images of inspection targets in their normal state based on a large number of reference images (Figure 1). However, there are many real-world situations in which it is too difficult to take precisely oriented photographs to match reference images or gather the massive amounts of data needed to train AI. Inspectors cannot reach some areas because it is too dangerous or the infrastructure requires too much trouble or work to access. Examples include steel towers in the mountains, under elevated bridges, and other high places or locations with precipitous drop-offs, and under offshore wind turbines and solar panels. Despite all intentions to reduce workloads in dangerous places, the difficulty of introducing AI is a barrier against automation and digitization of inspections.

To address these problems, Toshiba developed AI technology for accurately detecting anomalies and defects that lead to anomalies from only a few reference images and inspection photographs.
The AI uses deep learning features from pre-trained models to compare a large number of images. The first step in detecting real-world anomalies is to identify the features of reference images and inspection photographs. Next, the AI automatically selects the most similar of the pre-trained deep learning features, and compares them to the images to identify differences, creating a score map of detected anomalies (defects). Because the AI is based on pre-trained deep learning models, there is no need to gather images from each site for training—the technology is ready to use at any inspection site (Figure 2). Existing anomaly score map-based technology tends to produce false positives when it identifies differences from the deep learning features because the edges or structures in inspection photographs differ from those in the reference images, regardless of whether anything is actually wrong.(*2) Because of this, inspectors must revisit sites for confirmation (Figure 3).
In response, Toshiba has used proprietary technology to correct these anomaly score maps by removing recurrent features from several reference images, successfully limiting the number of false positives (Figures 4 and 5). In a recent experiment simulating an inspection of the undersides of solar panels, the company verified that the new AI is capable of accurately detecting anomalies (Figure 6).
Toshiba also achieved the world’s highest accuracy with the AI in tests using public data, scoring a 91.7% Pixel-AUROC (an estimated per-pixel accuracy rating) compared with 89.9% by existing technology that also does not require training (Figure 7).
Toshiba’s new infrastructure inspection AI can be applied in high places, locations with precipitous drop-offs, and other dangerous areas where AI has so far been difficult to introduce. The technology is also capable of accurately detecting anomalies that occur less frequently and are more difficult to detect with existing image recognition technology than cracks and rust. The new AI automates inspections, reduces workloads, and expedites the detection of anomalies, helping ensure stable long-term operation of infrastructure.

Figure 7: Comparison of Accuracy Using a Public Dataset (Toshiba AI vs. Existing Technology)

In pursuit of more accurate and precise infrastructure inspections, Toshiba is developing AI to achieve higher accuracy in detecting cracks, rust, and other specified anomalies and higher precision in measuring the size of the anomalies when identifying their locations in photographs (Figure 8). The technology developed in this work is an AI for detecting anomalies without pre-training. Toshiba aims to continue research and development to further improve accuracy and precision and develop a system with a view toward releasing a commercial product to be used in infrastructure inspection systems in FY2023.

*1: Toshiba research as of May 23, 2022. The highest figure recorded using a public dataset.
Public dataset: Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection– a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6536–6545 (2018)
*2: Sub-Image Anomaly Detection with Deep Pyramid Correspondences arXiv:2005.02357v3