The rapid aging of societal infrastructure is an issue of great concern. Much of Japan’s infrastructure, such as its roads, bridges, and tunnels, were built during the country’s period of rapid economic growth. Over the next two decades, the percentage of this infrastructure that was built 50 or more years ago is expected to rise at an accelerating pace. To maintain the safety and security of our society, this infrastructure must be updated, and strategy maintenance and management needs to be carried out based on day-to-day inspection operations. However, at the same time, the working population is shrinking due to the declining birth rate and the graying of society. There is an increasingly severe shortage of personnel who support this infrastructure. That is why hopes are high for the use of AI-based image recognition technologies to appropriately and efficiently carry out infrastructure inspections and maintenance and management operations with limited human resources.
In this issue, we will look at initiatives being carried out to elevate the level of expressway inspection operations by leveraging Toshiba’s long history of AI technology research and development.

“Societal infrastructure” includes a wide range of infrastructure that supports our lives, such as roads, expressways, tunnels, bridges, dams, sewer systems, waterworks, power distribution networks, and airports. In the past, inspection has been performed primarily by experienced inspection personnel who visually check equipment and facilities for cracking, peeling, or damage. They would report on their findings and perform repairs, preventing major accidents from occurring. However, in recent years, while infrastructure is aging at an accelerating pace, it has become difficult to secure personnel to perform the inspections that are essential for their maintenance and management. Facility maintenance worksites have been stretched to their limits.

For example, vehicles drive faster on expressways than on ordinary roads. For motorcycles, in particular, even tiny potholes or level differences can cause drivers to lose control, resulting in major accidents. It is important to identify even minor changes in road surface conditions -- road surface anomalies -- and to make repairs so that roads remain safe.

At present, traffic control teams have the greatest number of opportunities to discover road surface anomalies requiring urgent response and to make repairs. These traffic control teams drive along expressways every day in their patrol cars to identify traffic situations and deal with fallen objects and accidents. There are also inspection specialists who regularly drive on expressways in their inspection vehicles, inspecting roads and structures. Both visually check for issues from their vehicles. When they discover anomalies, they return to the area where the issue was observed, check for a safe place to exit their vehicle, and go to the site to check the road status and take pictures. They notify the Road Control Center of any anomalies requiring urgent response and they perform emergency repairs.

With this approach, it takes time to confirm anomalies and inspection results are highly dependent on the experience and expertise of individual inspection personnel. Inspection personnel are stretched to the limit, using their ingenuity to allocate their limited human resources and perform inspections. They are little able to allocate personnel to more forward-looking initiatives such as developing successors or proposing and implementing new services.

Like other societal infrastructure, the amount of work involved in equipment maintenance, management, and upgrading is also rising for expressway worksites, and the problem of personnel shortages is growing more severe. There is a strong need for solutions to these problems.

The Central Nippon Expressway Company (NEXCO CENTRAL) is responsible for managing a combined total of roughly 2,200 kilometers of expressways including the Tomei Expressway, the Shin-Tomei Expressway, the Meishin Expressway, and the Shin-Meishin Expressway, which connect the three major urban areas of the Kanto, Tokai, and Kansai regions. Their management scope encompasses roughly 6,000 bridges, 440 tunnels, 250,000 lights, and 3,800 road information signs -- a prodigious range and number of facilities. Approximately 60% of the expressways they manage have been in operation for over 30 years. Some were even opened more than 50 years ago. These roads are deteriorating with age.

Every day, almost 20 million vehicles travel these expressways, and in fiscal year 2019, about 3,200 potholes were discovered.

To keep these expressways safe, secure, and convenient 24 hours a day, 365 days a year, NEXCO CENTRAL is implementing i-MOVEMENT, an innovative expressway maintenance management system utilizing next-generation technologies. Digital and other state-of-the-art technologies are leveraged to address various societal issues and environmental changes while evolving expressway mobility. NEXCO CENTRAL is actively engaging in diverse technical verification projects and deploying this system in their operations.

As part of these efforts, NEXCO CENTRAL and Toshiba are working together to apply AI to achieve greater sophistication in day-to-day inspections. Toshiba’s image recognition technologies are being used to raise the level of accuracy when detecting anomalies in expressways with the aim of making inspection work more efficient and making repairs more quickly.

Specifically, NEXCO CENTRAL has attached cameras to the front, back, left, and right sides of the vehicles they use for day-to-day traffic control, and they are driving these vehicles along their expressways, capturing images of the road surface. These images are automatically sent to the on-board AI analysis system to detect road surface anomalies on the spot. If the system detects any potholes or other issues requiring urgent repair, it sends the Road Control Center its road surface anomaly detection images along with location information so that traffic restrictions can be quickly put into place and repairs can be speedily conducted (Fig. 1).

The accumulated image and location data can be used in Maintenance and Service Centers to evaluate the results of AI state change analysis (evaluate the AI model). It can also be used as training data for retraining the AI model to further improve its accuracy.

This system makes it possible to detect road surface anomalies simply by installing cameras in vehicles, without relying on the experience or expertise of inspectors. It can also detect road surface anomalies in locations that can be easy to overlook for human eyes and it eliminates the need to stop on the expressway and photograph the problem site in person. This makes inspection safer for traffic control personnel and inspectors. In addition, inspectors have periodically performed inspections that are difficult to determine without a high level of personnel expertise, such as whether or not they are road surface anomalies and whether or not they are urgent. Creating a system in which inspections including these advanced inspections can be performed by vehicles would increase the efficiency with which inspection work can be conducted with limited human resources. This could allow inspections to be carried out more frequently than is possible with conventional inspections.

There are high expectations that this system will free up personnel to perform high level inspection work that requires human judgment or to engage in operations or provide services other than inspections.

Let’s look at the features of our unique AI-based image recognition technologies.

Our image recognition, which detects road surface anomalies, is performed by our proprietary road surface anomaly detection AI, which uses weakly supervised learning. This system reduces the effort involved in creating training data. Normally, when creating training data for AI, colors must be applied at the pixel level to indicate to the system which parts of the road are damaged. This takes roughly one minute and forty seconds per image. To raise the accuracy of the AI, it must be trained using a large amount of training data, and preparing this training data takes a prodigious amount of time.

The learning method used by Toshiba’s road surface anomaly detection AI, however, requires only that images showing road damage be labeled at the image level. This cuts the amount of time involved in preparing each training data image to roughly one second. This method is called “weakly supervised learning,” and was developed based on a machine learning method that can be used even when there is little training data or insufficient labeled training data. Creating the training data requires no specialized AI knowledge, so, for example, inspectors with operational knowledge can create training data themselves using images of anomalies they discover while working. This has the potential to improve the efficiency of AI development (Fig. 2).

NEXCO CENTRAL chose this technology for their system because of the ability to prepare training data easily and quickly to improve the AI’s accuracy and the system’s potential for improving day-to-day inspection efficiency and quality.

The activities for raising the level of day-to-day inspections have now reached the deployment verification testing phase. In preparation for the deployment of the AI system in expressway maintenance and management operations, we have created a verification testing system that not only brings together the AI technologies and expertise we have developed through the years but also leverages our services, with their extensive track record, to the fullest.

We developed a verification system in multiple stages, from Step 1 to Step 3, as shown below (Fig. 3).

In Step 1, we prepared an inference and learning environment. Images of road surfaces photographed by vehicles were analyzed by AI and inspectors with knowledge and expertise checked if the AI’s results were correct, evaluating the AI model. A tool was used to efficiently sort the images in which AI analysis had detected anomalies. These images were stored as training data and used to retrain the AI model. The retrained AI model then analyzed newly-taken road surface images. The results were verified, stored, and used to retrain the AI model. This cycle was repeated to improve the accuracy of the AI. Services provided by Toshiba’s SATLYS analytics AI were used so that the extraction of training data and its use in training could be performed even by people who were unfamiliar with AI.
In Step 2, AI analysis was performed on-board the vehicle taking images of the road surface. We created a system in which the AI model that had been developed and trained in Step 1 detected road surface anomalies and immediately classified which were the highest priority for performing repairs.
Lastly, in Step 3, we created a mechanism for notifying the Road Control Center by immediately relaying the information obtained in Step 2. We used the tried-and-tested RECAIUS Field Voice Intercom for these notifications. This is an application that allows a smartphone to be used like an intercom or walkie-talkie. Its speech recognition and speech synthesis functions enable it to send text at the same time as speech and to relay information from sensors through synthesized speech. In this system, the road surface anomaly detection AI is responsible for relaying detected anomalies to the Road Control Center. The information received in the notification is stored in the form of speech and text, so they can be reviewed later if they were missed or need to be rechecked. This helps reduce the likelihood of communication gaps or overlooked issues.

High quality training data is important for improving the accuracy of AI. In this project, based on records from past NEXCO CENTRAL inspections, images and location information from actual sites were compared against those obtained from the vehicle-mounted system to identify anomalies in images and extract them for use as training data. There are many different patterns of road surface anomalies, involving different sizes, shapes, and depths. The urgency with which repairs are required also varies, and the skills of inspectors are essential for making these decisions. Inspectors have therefore checked images to identify anomalies and create high quality training data. The training data that is being generated in this way is used as training data to update the AI model, improving the accuracy with which it detects anomalies.

The most up-to-date, trained AI model is being utilized within the road surface anomaly detection AI to improve the accuracy of vehicle-based state change detection (Fig. 4). The results of vehicle inspections are centrally managed using SATLYS, so they can also be checked by inspectors and used as training data.

The verification testing that is currently underway is focused on pothole detection. In addition to that, in the future, we will refine the AI’s detection accuracy, investigate methods for deploying the system in a manner optimized for a vehicle-mounted system, design an AI system that contributes to accurately responding to the anomalies that are detected, and investigate technologies and functions for further improving the efficiency and level of inspection operations.

Of course, we also wish to expand the detection scope of the AI to encompass not only holes and cracks in road surfaces, but also cracks in slopes, damage to signs, fallen rocks, toppled trees, and the like.

We will continue to use Toshiba’s technical strengths to improve the quality of the societal infrastructure that is so vital to having a safe, secure society and to solve operational problems.