Linux has rapidly come to be used in a wide range of applications in the industrial field, from domestic appliances to devices used in social infrastructure. However, the road leading here has not always been a smooth one. For many years, Toshiba has taken on difficult challenges related to power-saving, reducing startup times, real-time control, and more. We have contributed to the evolution of Linux by achieving these goals while creating new value. New challenges have arisen, such as long-term maintenance, improved reliability, and cyber-resilience, and our engineers are constantly pursuing further innovation. In this four-part running feature, we will look at the numerous difficulties Toshiba has faced and the measures we have used to overcome them.
In Part 1, we explored the challenges involved in using Linux in various devices and the history of Toshiba's efforts to tackle these challenges. In Part 2, we explain real-time control, one of the important requirements of industrial systems.

A control process that guarantees the completion of a task within a specified time frame is called real-time control, and the capability to achieve this is referred to as real-time performance. Let's look at the example of an automobile airbag. Figure 1 shows the sequence of events from the occurrence of a “collision” to the completion of the process of “deploying the airbag.” In general, airbags finish inflating 0.03 seconds* after an automobile collision so that they can catch the vehicle's occupants.

* Reference:
https://www.nissan.co.jp/COMPASS/ISF/3rd/PAPER/
https://global.honda/jp/safetyinfo/think_safety/vol16/airbag/

In the example on the left side of Figure 1, the latency—the time from the occurrence of the event to the start of processing—is short, and the process is completed within the specified deadline of 0.03 seconds. In contrast, in the example on the right side of Figure 1, the latency is too long, resulting in a failure to meet the deadline. In such cases, there is a high risk that airbag may not fully inflate in time, increasing the likelihood of serious injury to the occupant. To prevent situations such as this, meeting deadlines is a critical requirement in the real-time control of industrial equipment. If deadlines are not met, it could lead to serious consequences in society—for example, automatic ticket gates at train stations that fail to allow smooth passage for users, or power plants that experience frequent outages.

Real-time control is classified into two types: hard real-time and soft real-time. Hard real-time refers to systems where failing to complete processing within the deadline can lead to serious consequences—such as with automotive airbags. Soft real-time, on the other hand, refers to systems where slight delays beyond the deadline are acceptable.

Traditionally, dedicated real-time operating systems (RTOS) have been used to implement real-time control. However, with the increasing performance and functionality of modern hardware, the range of devices requiring real-time control has diversified, leading to a growing need for operating systems that offer greater flexibility and adaptability across various applications. Against this backdrop, Linux—available as open-source software (OSS)—has emerged as a promising option due to its wide hardware compatibility and rich development environment, offering the potential to significantly enhance system productivity. Furthermore, because Linux is continuously improved by a global community of developers, it offers strong prospects for long-term support and the integration of cutting-edge technologies. As a result, the adoption of Linux for real-time control applications has been steadily expanding.

Standard Linux is not an OS originally intended for hard real-time control. Therefore, when used in real-time systems, it required functional extensions. There are several ways to extend its functionality. One of these is the PREEMPT_RT patch, which has been under continuous development for over 20 years to improve Linux's real-time performance. After years of evolution, PREEMPT_RT was finally integrated as a standard feature in Linux with the release of Linux 6.12 in November 2024.

PREEMPT_RT's functions enhance Linux's real-time performance. However, simply using PREEMPT_RT does not guarantee achieving the desired level of real-time control. To ensure reliable real-time control, it is essential to identify processing characteristics and limits to avoid missing deadlines and to design systems appropriately. Now let's look at the challenges of performing real-time control using Linux and the measures Toshiba has used to tackle these challenges.

Linux is a general-purpose OS that is built for use in a wide range of use cases. Because of that, it can be used as across various fields and applications. However, when Linux is used as an embedded system for real-time control, this general-purpose nature can sometimes become a disadvantage. For example, the Linux used in servers or computers allocates memory when it is needed for program execution, switches between multiple programs while making it look like they are running simultaneously, and distributes loads between the cores of multicore CPUs, all with a reasonable frequency.

Let's think about how this "reasonable frequency" of operations affects real-time control, using that airbag example. In this simplified scenario, the process consists of two steps: (1) receive the impact signal from the crash, and (2) inflate the airbag. In the processing in step (1), what would happen if when attempting to run the program for receiving the impact signal, the system had to wait 0.01 seconds for another program to finish executing and when starting execution of the process for receiving the impact signal, there was nowhere to store the signal, so it took 0.05 seconds to allocate memory? If that happened, the time limit of 0.03 seconds for fully inflating the airbag (the deadline) would be passed before even processing step (2) was even reached. To prevent this, during real-time control, Linux's functions must be able to do the following:

• Prioritize the execution of programs based on their urgency
• Allocate memory to be used in advance

Not only must the above be performed, but further items must be assessed, namely the amount of latency that is feasible for Linux, the amount of time taken for processing, and the amount of time that can be allocated to the CPU.

■ Determining Linux's latency

As mentioned earlier, the time between the occurrence of an event and the start of processing is referred to as "latency." With real-time control, it is not as simple as "the shorter the latency the better." Instead, what matters is that the amount of latency can be kept within a certain range. In the case of periodic processing, when events happen in fixed intervals, as long as the related processing is completed by the deadline whenever an event occurs, there is no problem. The amount of latency can vary without creating any problems, provided that it remains within that span of time (Fig. 2).

We have conducted various evaluations of periodic processing. First, we evaluated latency when the CPU was under load conditions (Fig. 3). This measures latency by executing a predetermined periodic process 100,000 times. With the original version of Linux, for periods of both 500 and 300 microseconds, the average latency was roughly 3 microseconds, but the deadline was exceeded over 100 times, with the worst result (the highest latency) being over 28,000 microseconds. This indicates that while performance is often good, there is also an extremely large amount of variability when it comes to latency, so using this version of Linux would cause problems in practical applications.

Now let's look at PREEMPT_RT. With a 500 microsecond processing period, the average latency was roughly 12 microseconds, which is greater than it was for the original version of Linux, However, the deadline was never overrun, and even at its maximum, the latency only reached around 200 microseconds. If the real-time control processing time is 300 microseconds or less, then mathematically the response time of 500 microseconds could always be achieved. This ability to always achieve the necessary response time is vital for real-time control.

What about the 300 microsecond period evaluation for PREEMPT_RT? In this evaluation, the average latency was roughly 11 microseconds, but there were 93 deadline overruns, with the longest being an extremely high 27,000 microseconds. This evaluation was performed to confirm if high priority processing could be reliably performed when the CPU was under heavy loads, so this evaluation result is not enough on its own to conclude that a 300 microsecond period would not be achievable. However, it is clear that the shorter the period in which you try to achieve stable operation, the greater the constraints placed by processing load (processing time). These results also show that this also increases the level of precision required in system design.

■ Determining the amount of time that can be allocated to CPUs

The amount of time that a CPU is exclusively used when running a program is called "CPU time." In addition to determining latency, we also evaluated and improved the amount of CPU time that could be allocated to real-time processing. When we began our evaluation, our test system operated correctly when using Linux with standard PREEMPT_RT and allocating 1 millisecond of CPU time for 2 milliseconds of processing. However, when setting the CPU time to less than 1 millisecond, it became impossible to properly allocate CPU time (Fig. 4 - Before improvement).

Our investigation found that Linux had not been designed to accommodate times of less than 1 millisecond. We used a high precision timer to make improvements and were successful in accurately allocating CPU times with a granularity of less than 1 millisecond. After this improvement, we were able to allocate CPU times in intervals of 0.5 milliseconds for 1 millisecond processing periods (Fig. 4 - After improvement).

We proposed these improvements at a community meeting at which the creator of the function happened to be in attendance. I remember quite clearly what he said: "Huh? There's almost never any need for a granularity of less than 1 millisecond." But there are actually a lot of people who are struggling because of this limitation. After our presentation, another person gave a similar proposal, and their method is now another available option. Their implementation method differs from our own, and this was an excellent example of using this open forum to present and discuss problems in greater depth.

The number of cores in CPUs is rising, and CPUs can execute multiple processes at the same time. CPU performance improvements and the corresponding increases in the amount of processing that can be performed in parallel has made it possible for single computers to provide a multitude of functions. For example, they can perform both real-time control processing and general purpose processing. Devices can be periodically controlled and the results can be aggregated and displayed graphically. Device control must be reliably performed in periodic intervals, but the aggregated information can be viewed whenever desired.

However, when mixing this real-time processing and non-real-time processing in a single CPU, it is impossible to know exactly when non-real-time processing will be performed, and it could have an unforeseen impact on real-time processing. To avoid this, the system can be structured such that one CPU core is isolated and used exclusively for real-time processing. This ensures that it will not be affected by non-real-time processing (Fig. 5).

We tested out this architecture and found that when there are multiple periodic processes and CPU load is high, there are occasional instances of high latency (Fig. 6 - Before improvement). Looking more closely at the reason for this, we found that it was the result of Linux's internal processing, which is shared by both real-time and non-real-time processing. When there are multiple periodic processes being performed, it is sometimes necessary to adjust the timers that control when programs are started up. We found that this issue occurred because the processes were not sufficiently isolated. By implementing improvement measures, we succeeded at fully isolating the real-time control processing (Fig. 6 - After improvement).

In this way, Toshiba has carried out multiple rounds of evaluations and improvements to enable Linux to be used for real-time control, and it has applied this real-time control-capable Linux to industrial equipment.

We have looked at the challenges of performing real-time control using Linux and the measures Toshiba has used to tackle these challenges. As a result of these long years of effort, Linux has come to be selectively used in various products, based on those products' specific characteristics.

One example is TOSMAP-DS/LX, a controller used in power generation. With TOSMAP-DS/LX, we optimized Linux to achieve the real-time performance needed to control power generation.^*1 Another example is Vm Series typeS, a unified controller with a fast, fixed processing interval of 0.5 milliseconds (1/6 of the amount of time it takes to fully inflate an airbag). This typeS controller, which supports the IoT, assigns the functions needed for a controller capable of real-time control to a separate CPU core than the functions used by the computer for general-purpose processing, so the two can be performed simultaneously.^*2

*1: Ref. Toshiba Review "TOSMAP-DSTM/LX Next-Generation Controller for Thermal Power Plants to Achieve Efficient Operation and Low Environmental Burden" (https://www.global.toshiba/content/dam/toshiba/migration/corp/techReviewAssets/tech/review/2013/11/68_11pdf/a08.pdf)

*2 Ref. Toshiba Review "Industrial IoT Controller for Realization of Edge-Rich CPS" (https://www.global.toshiba/content/dam/toshiba/jp/technology/corporate/review/2021/01/f04.pdf)

In this issue, we have learned about Toshiba's efforts related to real-time control, an important element of embedded Linux. We have repeatedly developed and evaluated various technologies and coordinated with the community to make it possible to use Linux, which was developed for general purpose applications, in real-time control applications. As a result of these efforts, we have achieved the quality and performance needed for industrial applications.

In the next issue, we will turn the spotlight on the community and delve into the creation of OSS communities dedicated to addressing issues faced by the entire industrial field and the technological development work performed by those communities. Don't miss it.