Toshiba's Inspection Information Management AI Automatically Recognizes and Manages Imaging Locations and Subject Sizes from a Single Photo
-Contributing to automation of patrol and maintenance inspection tasks in power plant facilities and work-style reforms for the “new normal”-

Overview

TOKYO―Toshiba Corporation (TOKYO: 6502) has developed inspection information management AI for equipment inspections that can recognize the location as well as any signs of deterioration in equipment, including the size of any crack, simply by analyzing a single photo taken with a general-purpose camera ^{(Note 1)}. The system combines two Toshiba technologies: position-recognition AI ^{(Note 2)} that identifies locations from images, and 3D-recognition AI ^{(Note 3)} that accurately estimates sizes. The new AI will contribute to the efficiency of maintenance inspections at power plants and other facilities with weak or no GPS signals.

While GPS data can be used to record equipment locations in facilities such as power plants, when the signal is weak or non-existent, inspectors generally photograph signs of deterioration and cracks, and manually record the location and measure the size. This then has to be organized, and the photographs associated with floor plans. It is a time-consuming process, and error can creep in. Toshiba’s inspection information management AI automates the task, and enables information sharing in remote work. Photographs taken by inspectors, or even robots or drones, are uploaded to a server, and managed as a batch. The AI automatically looks for and recognizes image locations, the sizes of cracks, and any other signs of deterioration.

Toshiba plans to release the AI via Toshiba Energy Systems & Solutions Corporation’s IoT platform for energy systems ^{(Note 4)}.

Development background

The global infrastructure maintenance market continues to grow. The Japanese market is estimated at some 5-trillion yen a year (approx. US$47 billion), with maintenance of aging infrastructure a major concern, and the worldwide market is valued at about 200-trillion yen ($1.9 trillion), driven by the twin demands of aging and rising demand ^{(Note 5)}.

Japan faces the particular problem of labor shortages and the aging of workers involved in maintenance, and must perform inspection tasks more efficiently with fewer personnel. Furthermore, with the increase of telecommuting as part of the “new normal,” there is a growing need for mechanisms for remote confirmation of on-site conditions.

Issues with conventional technologies

Inspections and maintenance of infrastructure commonly start by photographing and recording the condition of the site and equipment. The photographs are then used to pinpoint locations on online floor plans ^{(Note 6)}. However, at indoor plant facilities or other locations with poor or no GPS reception, identifying the locations takes time.

Methods for identifying indoor positioning include installing wireless base stations and the use of markers, but the former solution is expensive, and when markers are used both the area of concern and the marker must appear in the same photograph, which requires a large number of markers.

Measuring degradation in images is another issue. On-site, measurements are generally made with a tape measure, but this is impossible for unreachable locations, and if an inspector forgets to take a measurement, it is necessary to revisit the site. There are contactless measurement methods, including the use of laser scanners and depth sensors, but then recording areas of concern requires bringing not only a camera but also other equipment to the site.

The developed technology

Toshiba has combined its position-recognition AI for determining location information from photographs and its 3D-recognition AI that recognizes sizes, in inspection information management AI for maintenance inspection tasks at power plant facilities.

First, photographs taken in the infrastructure facility are used to create a positional database that the location-recognition AI uses to link photographs with locations on a floor plan. Toshiba's AI uses a deep neural network model to automatically recognizes imaging directions and spatial positions on the floor plan from camera images, and is highly accurate, even when using low-resolution images.

Using deep learning to analyze image blurring that occurs due to distance from the subject allows the 3D recognition AI to measure distances regardless of information depicted in the background. This secures distance measurements as accurate as those from a stereo camera, even when using a commercial monocular camera.

The inspection information management AI simultaneously recognizes the location and size of subjects from a single uploaded photograph acquired during an inspection. This allows inspection information to be stored with online floor plans with no need for additional equipment. Accumulated inspection information can then be obtained by accessing the relevant floor plan.

The goal of the technology is to collect real-world physical data to construct information and knowledge that is easily used in cyberspace, thereby contributing to the realization of the cyber-physical systems (CPS) technologies that Toshiba Group is aiming for.

Future development

Toshiba will release the inspection information management AI via Toshiba Energy Systems & Solutions Corporation’s IoT platform for energy systems, with an aim toward practical application in FY22. In the future, the company aims to expand the scope and utilization of the inspection information management AI by adding anomaly-detection AI and meter-reading AI, as well as AI that detects and predicts aging by examining accumulated data. In addition to power plant facilities, Toshiba aims to expand application to other structures where GPS is unavailable or unreliable, such as warehouse or bridge inspections.