Toshiba Develops a Dialog Agent Using Generative AI that Can Appropriately Answer Even Vague Questions

Toshiba has developed dialog agent technology that can appropriately present operating procedures for maintenance, inspection, and troubleshooting. Recently, generative AI has appeared that utilizes high-performance large language models (LLMs), typified by ChatGPT (OpenAI), and chat bots (response generation AIs) equipped with these AIs have become widespread. With the existing technology, however, for users to receive an appropriate answer and obtain the information they want to know, such as operating procedures, some skill is required when formulating the question, such as ensuring that the question is well specified. The technology Toshiba has developed in the present work is able to respond appropriately to vague questions by adding a newly developed “response evaluation AI” to conventional “response generation AI” technology and having the two AIs work together in different roles. When a question is input by the user, the “response generation AI” generates multiple response candidates, and the “response evaluation AI” evaluates the fitness of these candidates and responds to the user by selecting the most appropriate one. If the “response evaluation AI” determines that the question is vague and difficult to answer, it does not directly answer the question, but automatically guides the user to making the question more specific by asking questions to the user. This makes it possible to respond appropriately to vague questions.
Toshiba has adapted this technology to the use case of presenting operating procedures for unknown points in operations, envisioning maintenance work at an actual plant. The proportion of interactions that result in the correct operating procedure being provided in response to vague questions from workers with little experience was 30% with conventional “response generation AI” technology alone but the rate increased to 73.3% with the developed technology. This technology is expected to promote the introduction of generative AI to worksites for tasks such as maintenance and inspection, and to improve productivity by presenting the appropriate operating procedures regardless of the skill level of the worker.
Toshiba will present details of this technology at The Institute of Electronics, Information and Communication Engineers Society Conference held at Nippon Institute of Technology on 10–13 September. In addition, Toshiba has been working on the application of this technology to maintenance work at Toshiba Group plants in order to confirm the utility of the technology.

Infrastructure such as bridges and tunnels and equipment such as that in factories and plants are aging, and the issue of efficient upkeep management has become a problem. Furthermore, the labor force is shrinking due to the aging population and low birth rate, highly capable skilled workers are retiring, and the shortage of maintenance and inspection workers has become a reality. Against this backdrop, the size of the market for maintenance and inspection services linked to labor-saving is forecast to be 2.935 trillion yen in 2025 (*1), and there is an increasing need for more efficient services.
For workers with little experience to work efficiently, it is important to quickly resolve unclear points and questions that arise during both preparation and actual work in order to prevent work stoppages as much as possible. Because of this, Toshiba has been working on introducing a chat bot that implements generative AI and can respond to questions asked using natural language as if asking a person, such as questions about knowledge of phenomena and past case studies at maintenance sites.
However, with conventional chat bots, the appropriate response for the desired information can be obtained if the question text is specific, but if the question is vague, the response can contain information unrelated to the question. Because of this, the user themselves needs to search for the required information from within the response, interrupting work. A chat bot is desired by which even users with little experience who tend to ask vague questions can appropriately obtain the information they need.

Toshiba has developed dialog agent technology that can appropriately respond even to vague questions by adding a newly developed “response evaluation AI” to a conventional technology “response generation AI” and having the two AIs work together in different roles (Fig. 1). The “response generation AI” generates multiple candidate responses to the question text from the user (Fig. 1(1)), and the “response evaluation AI” evaluates the fitness of each of the response candidates generated by the response generation AI (Fig. 1(2)). Through cooperation between these two AIs, multiple response candidates and fitness scores for them are elucidated for the user question text. The best answer is selected based on this score and output as a reply to the user.

Specifically, the “response generation AI” generates multiple patterns of response candidates for the user question, such as “response candidates for the case of answering” and “response candidates for the case of asking questions back” for getting the user to input additional information. The “response evaluation AI” rates the fitness of each response candidate from two perspectives: (a) containing only correct information in response to the question text and (b) having no redundancy. Next, the response candidate with the highest score is replied to the user. If the question text from the user was specific enough, the dialog agent answers the question by selecting from among the “case of answering” response candidates, whereas if the question text was vague and needs to be made more specific, the user is answered with a question selected from among the “case of asking questions back” response candidates. If they are asked a question back, the user inputs additional information by following the prompts. By using this kind of back-and-forth exchange, the chat bot helps workers to obtain the desired information appropriately and accurately even for vague questions.
Toshiba investigated the utility of this technology in the use case of proposing answers to questions about unknown points arising during work, envisioning maintenance work at an actual plant. This investigation compared the performance of a chat bot using only the conventional “response generation AI” technology and a chat bot using the dialog agent developed by Toshiba. When the problems were input, the pattern of entering specific information (“Error code 1234 is displayed on the screen of device A”, etc.) and the pattern of entering vague information (“Does not work”, “Job error”, etc.) were each investigated (*2). With the conventional technology, the success rate with specific questions was 76.7% but dropped to 30.0% for vague questions. With the proposed technology, a success rate of 73.3% was achieved for vague questions, which was virtually equal to the rate achieved for specific questions (Fig. 2). When answering vague question text, whereas the conventional technology answered everything using the first turn, the proposed technology performed an average of 2.3 turns of communication, and this result demonstrates that the proposed technology can answer appropriately by making the question more specific through asking questions back to the user.

Toshiba is planning to start verification experiments on on-site question work for maintenance work at Toshiba group plants sometime this year. Furthermore, Toshiba is pursuing research and development with the aim of providing a service outside the company.

• 2021 Edition: Current State and Future Prospects of Plan Maintenance and Business-Related Market | Survey Report | Fuji Keizai Group (fuji-keizai.co.jp)
• Investigation using 30 patterns each of vague questions and specific questions

Note about trademarks: Company names and product names in this document are trademarks or registered trademarks of their respective companies.