For the manufacturing industry, improving productivity is an important task. This is just as true during the prototyping stage. Rebuilding tools each time there are design changes and performing work while looking at diagrams and instructions often leads to the accidental omission of work steps. Furthermore, design and prototyping departments cannot easily share information with each other, which sometimes leads to misunderstandings. Attention is being turned to the use of Mixed Reality (MR) technology to solve these problems.

However, implementing MR involves its own challenges, requiring the preparation of specialized devices and environments. To resolve this, Toshiba Digital Solutions has developed MR Package with the aim of making MR easy to deploy quickly and inexpensively. Let's look at MR Package, together with automaker usage cases. 

The recently coined term "X Reality," or "XR," is a collective term for the technologies used to create new experiences by overlapping the virtual and real worlds. XR is now used not only in entertainment fields such as music and games, but also in a growing range of other fields, such as the automotive, manufacturing, and medical fields. In 2025, the XR market is expected to reach 1.2 trillion yen in Japan alone.*

* Source: Yano Research Institute "Domestic XR (VR/AR/MR) 360° Video Market Size Forecast" (issued January 6, 2020)

XR technologies include VR, AR, and MR. Their differences lie in the role played by the main subjects that are visually depicted (Fig. 1).

Manufacturing, especially the pre-mass production prototyping stage, involves the verification of prototypes. This verification work encompasses preparation for verification, actual verification work, result summarization, and the sharing of verification results (feedback into prototype design). The work performed before and after the verification work itself is referred to as incidental work. It includes preliminary work, such as arranging for the diagrams used in verification and preparing tools, summarizing the results of the verification and creating a verification report, sharing information, and providing feedback. All of these are performed by hand, an expensive and time-consuming process. Verification itself is also primarily done by hand, so it is prone to mistakes, accidental work process omissions, and reworking.

MR technologies and design data, which already exists in the form of 3D CAD data, can be effectively used to tackle these problems, improving work efficiency and quality.

For the automobile industry, as well, MR technology is increasingly being adopted not only for simulations in upstream processes such as design and planning, but also in processes in the field, such as prototyping and mass production (production lines).

Let's look at an example of how MR technology is already being used in vehicle prototyping processes, where full-fledged utilization of MR is expected to become more common in the future.

The MR technology-based solutions developed by Toshiba Digital Solutions have already been used in the automobile industry since 2019.

In vehicle prototyping, one customer is using them to verify if prototype vehicle weld points are correct. Previously, when inspecting weld points, the customer would create a life-sized model (a template gauge) with holes at the points on the design diagram where welds were to be made. This was then placed against the prototype vehicle to confirm if welding had been performed correctly. However, creating template gauges for the entire prototype vehicle was a highly time intensive process, and the template gauges needed to be remade every time there were design changes. This meant that incidental work performed before inspection work even began placed a significant burden on personnel. It also made inspections by inexperienced personnel very slow.

By using our solution to perform this inspection work, the customer was able to significantly improve the efficiency of both incidental work and inspection work. Instead of creating template gauges, which was difficult incidental work, design data for prototype vehicles (3D CAD data) is simply converted into 3D models. This is all the pre-inspection preparation that is necessary. The customer was also very happy to see that any design changes could be immediately reflected in the 3D models, and that weld points could be visually checked during inspection by displaying the prototype vehicle and the 3D model overlapped on a tablet device's screen.

In addition to reducing the workload involved in incidental work, the solution provided benefits in every area of inspection. It made it possible for even inexperienced personnel to quickly perform inspections, without making mistakes, and it reduced the number of missed work steps. (Fig. 2)

There are also high hopes for the system's effectiveness in applying coatings to prototype vehicles.

Unlike mass production by production line robots, much of the coating application performed in prototyping is done by hand. This coating work is done by directly applying a jig indicating the coating points to the prototype vehicle. Creating jigs is costly, and applying them is labor intensive. Furthermore, the actual coating process and the quality inspection work that is carried out after coating is completed are done while checking diagrams, making it difficult to achieve consistent quality. With this solution, it will be possible to visually confirm coating types and areas from the tablet device's screen. Personnel will be able to simply check the screen to perform coating and checking work correctly. This is expected to significantly contribute to the achievement of more consistent work quality and reduce the amount of time taken by these processes.

We released MR Package in October 2021 to provide even more customers with access to solutions such as these, which utilize MR technologies that have evolved through their use in actual worksites.

MR Package makes it possible to leverage MR technology using an ordinary tablet device, without the need for specialized devices or environments. Provided that there is 3D CAD data, the 3D models used by MR Package can be created simply by clicking on a computer screen. It is available for a low monthly subscription fee, making it easy for customers to deploy. Trial deployment is also supported

Its functions are based on the technologies used in the Toshiba Group's plant business operations. We have used the expertise we have developed through our deployment track record to tailor MR Package for the manufacturing industry. Within the Toshiba Group, there are many departments involved in design and development work. The diverse 3D CAD data that they are creating is being used to produce 3D models that are contributing to improvements in manufacturing sites. The functions needed by the manufacturing industry, in particular, are included standard in MR Package, so customers can begin enjoying their benefits immediately.

MR Package is composed of a data conversion tool, which turns 3D CAD data into 3D models that can be displayed as MR, and a tablet application, which displays 3D models as MR. (Fig. 3)

With the data conversion tool, which can be run using an ordinary computer, customers can easily turn as much 3D CAD data as they want into as many 3D models as they want, as many times as they want, any time they want.

The tablet application operates on tablet devices. It is used simply by importing data that has been converted into 3D models, so it can be used even in sites with poor internet connectivity. When a tablet device is pointed at an actual object, markers are recognized automatically and the selected 3D model is positioned. Furthermore, other functions often needed at worksites have also been included, such as the ability to check past records while performing work and the ability to perform evidence management using virtual notes. These improve MR Package's work utility and makes it easier to share all kinds of information with other parties.

Virtual notes are a function for attaching text, photographs, and other data to coordinates in 3D space by tapping the tablet's screen. For example, if an operator performing equipment maintenance in a plant discovers an area that requires repair, he or she can tap the screen and attach a virtual note, right there on the spot.

Repair technicians can check the comments and photographs in the note to immediately determine what needs to be repaired and what kinds of repairs need to be made. This shortens repair times, and the information can be used in evidence management. Furthermore, the perspectives and advice of experienced personnel can be recorded in virtual notes. This assists with the transfer of skills and the training of new personnel.

We are currently developing functions that use AI image processing technology to automatically determine the accuracy of weld point positioning during prototype vehicle inspection. Automating the quality evaluation work that is now being performed by people will assist with achieving a greater level of quality consistency, as it will no longer be dependent on the decisions made by individuals. This has the potential to provide even greater future benefits.

Expanding the usage scope of 3D models from completed product manufacturers to suppliers could also enable suppliers to use MR Package to efficiently perform checking and result recording for work items before sending components to suppliers, while at the same time maintaining consistent quality.

In order to further broaden the applications of 3D models and MR technologies, we aim to provide, in service form, a platform consisting of the environment required for sharing 3D models, work contents, and work records in real time; AI tools for making inspection pass/fail determinations; and IoT integration functions. This will not only improve work efficiency and quality, but will also be effective in tackling various operational issues such as the sharing of information between departments.

Toshiba Digital Solutions will continue to explore the frontiers of XR technologies in its quest to help solve the problems that customers face.

• TOSHIBA Cyber Physical Systems