Posted by Achim Nohl on March 22, 2012
In the last month, I had the opportunity to get some hands-on experience with hardware virtualization and hypervisors. My knowledge so far on this has been mainly limited to what I could read about it and what other people are saying about it. However, the PowerPoint slides I’ve seen leave a lot of white fog between the bullet items. This didn’t make me feel very comfortable talking about this topic myself; but, there was no escape. Hypervisors play an increasingly important role for system designers in context of supporting multiple guest operating systems on the same device, or taking advantage of ARM®’s new big.LITTLETM processing. The fog is not all gone, but let me provide you some insight on what I found out. As a disclaimer, I’m not going to (and I cannot) write an expert almanac about all the aspects of virtualization covering Xen, VMWare, etc. Instead, I’m going to focus on my personal experience that I believe will be relevant to you as well. This post is the starting point for a series on this topic in this blog.
Clearing up terminology confusion
The fact that this whole concept is (also) called “virtualization” is high on the list of the top challenges I was facing when explaining it to my colleagues, or management. As a background, I am working in technical marketing in the area of “virtual prototyping,” and our product is called “Virtualizer.” In order to avoid the same confusion here, let’s separate those terms in context of embedded systems:
I used a VP on my desktop PC to bring up ARM Linux along with an embedded hypervisor. Using a VP, I’ve been able to integrate, run and debug a software stack including Android, Linux and a hypervisor on an ARM big.LITTLE processing-based system. The VP integrates Fast Models from ARM to simulate the processors.
The first example of this processing concept combines a high performance ARM Cortex™-A15 MPCore™ processor along with an energy efficient Cortex-A7 processor. The two main use-models which have been introduced by ARM are called “big.LITTLE Task Migration” and “big.LITTLE MP.” Those use models are nicely described by the Linaro organization.
The so called big.LITTLE Task Migration use model is that the software can seamlessly migrate from one processor to the other, depending on the use context and resulting performance requirements. It sounds like this requires a heavy re-write, or modification of the software stacks (e.g., Linux/Android) to support this; in fact, it doesn’t. This task migration is achieved by having the software run not on the hardware, but on-top of a new layer. This layer operates in the hypervisor mode and performs the task-migration without Linux/Android even knowing about it — although, the Linux/Android power management will initiate it as illustrated in the next figure.
Hypervisors and Interrupts
The task of the hypervisor software layer is to shield the hardware and payload (e.g., Linux/Android) software from directly communicating with each other. We call Linux/Android payload, as it is the software the user is intending to run. Why is a hypervisor needed? Interrupts are a good example here. Let’s assume we run multiple guest OSs on a system. Interrupts coming from the hardware could be for either one of the OSs. Therefore, a hypervisor needs to first intercept the interrupt from the system, and then decide to which guest OS it was addressed. For big.LITTLE processing it’s the other way around; here, we have multiple processor clusters sharing the same interrupt controller. The hypervisor (open source example available here) ensures the transparency of the clusters for the OS and does the task migration. But, the hypervisor needs its own interrupts and should not interfere with the OS.
Hardware Supported Interrupts Virtualization
In order to enable interrupt trapping in the hypervisor, ARM provides specific hardware support in its processors. In the processor’s hypervisor mode, a higher privileged exception vector allows trapping interrupts even before the OS can react on it. Thus, when an interrupt come in, it is the hypervisor that first handles it. If it is for the OS, then it will configure a virtualized interrupt controller, which is a replicate of the real interrupt controller seen by the OS. The OS will then handle the interrupt as if there was no hypervisor in between. Here, the MMU (Memory Management Unit) virtualization plays an important role as well. More about that in my next blog.
Interestingly, an interrupt has quite a way to go before it arrives at the user’s application, all the way from the hardware, through the hypervisor, into the OS. Virtual prototypes are extremely useful for debugging this chain, as all aspects of the system can be observed and traced as shown in the figure below.
Keeping the overview on what is going on and having a bird’s eye view to assess where things go wrong are really helpful here. Where does the interrupt stop? Stuck in the hypervisor? Not arriving at the interrupt controller? The VP ensures that at any time the different layers of the SW and even HW can be traced and debugged to spot and identify software integration bugs. First of all you have clarity on which CPU is currently active. Second, you can spot the time where the actual interrupt arrives from the hardware. Next, tracing of the system is possible across the layers of the software, from the hypervisor up to Android. The VP tracing and debugging is aware of the hypervisor layer by tracking the mode that is exposed from the underlying CPU models (ARM Fast Models). Furthermore, debugging services are always available, because debugging does not depend on any embedded software daemons to function. This is very useful for debugging the interaction between the hypervisor and the Linux kernel. You can simply have one debugger attached to the hypervisor, and another one attached to the Linux kernel. This becomes really compelling when debugging the task migration. Even during phases where one CPU is powering down and the other is powering up, debugging and tracing is possible without limitation. This is a delicate phase as the entire context of one CPU is saved and subsequently restored by the other CPU. This can result is obscure and hard to find defects, for example, if the context saving is incomplete because it forgets about saving the secure mode registers. I will write more about the task migration, in context of energy/performance optimizations in a follow up post.
If you’d like to see a live demonstration and explanation of all this, we’re presenting at the DesignWest Conference (formerly called the Embedded Systems Conference) in San Jose in the ARM Partner Pavilion and also in a joint technical session with ARM as part of the Android Summit. You can also see a demonstration from ARM and Synopsys at SNUG Silicon Valley this coming Monday, March 26th at the Designer Community Expo or Tuesday, March27th in the afternoon tutorial of the Systems track.
Patrick Sheridan is responsible for Synopsys' system-level solution for virtual prototyping. In addition to his responsibilities at Synopsys, from 2005 through 2011 he served as the Executive Director of the Open SystemC Initiative (now part of the Accellera Systems Initiative). Mr. Sheridan has 30 years of experience in the marketing and business development of high technology hardware and software products for Silicon Valley companies.
Malte Doerper is responsible for driving the software oriented virtual prototyping business at Synopsys. Today he is based in Mountain View, California. Malte also spent over 7 years in Tokyo, Japan, where he led the customer facing program management practice for the Synopsys system-level products. Malte has over 12 years’ experiences in all aspects of system-level design ranging from research, engineering, product management and business development. Malte joined Synopsys through the CoWare acquisition, before CoWare he worked as researcher at the Institute for Integrated Signal Processing Systems at the Aachen University of Technology, Germany.
Tom De Schutter
Tom De Schutter is responsible for driving the physical prototyping business at Synopsys. He joined Synopsys through the acquisition of CoWare where he was the product marketing manager for transaction-level models. Tom has over 10 years of experience in system-level design through different marketing and engineering roles. Before joining the marketing team he led the transaction-level modeling team at CoWare.