Archive for 2012

How to win over the embedded software developer, their customer and their boss.

By Nithya Ruff
Achim Nohl was taking a well-deserved vacation last week and asked me to be his guest blogger. To many of you who are regular readers of Achim’s blog, I am new to Synopsys and joined only a few weeks ago to manage the Virtual Prototyping product. I came from Wind River where I managed Embedded Linux product marketing. Having come from 20+ years of managing software of all kinds, I saw the guest blog as a perfect way to talk about a software marketeer’s view of virtual prototypes and embedded software development.

I still vividly remember the manual and tedious process of debugging the operating system I had written in my computer science OS 401 class. Not only was the data not all in one place, it involved reams of paper and tracing the calls manually. Frankly, it does not feel any different these days in the world of embedded software. Every element of the device stack is put together separately, and often there is no holistic view of the entire device. Debugging and testing is more of an art than a repeatable and systematic science. Let’s look at why this is so.

Embedded systems cover everything from very small systems inside watches, to sensors in a building, to complex multicore-based core-of-the-network systems. They share the characteristic of being dedicated to a specific function and not being general-purpose. These systems often require real-time performance and a smaller footprint. Everything is customized to fit the task—the Operating System (OS), middleware and applications. Given the wide range of applications, the choice of the hardware is specific to the device being created. The software stack created is influenced by the hardware characteristics and behaviors. Hence, there are all kinds of hardware architectures, and even many variations of custom hardware used in the embedded device space. In fact, hardware is the first and a key component of the decision making process for a project. The rest is built around the hardware selection.

In the late 1990s and early 2000s the success of Linux in the server space began to bleed over into the embedded space, especially in networking. Linux was inexpensive to acquire and provided almost everything needed, such as tools, protocols and trained people. Most of all, it provided access to source code and could be modified as needed. Linux also was available on other architectures such as ARM, MIPS, and PPC and X86. This was important to embedded developers so they could select the right hardware for the task at hand. Chip vendors had started using Linux in their boot-up software and in SDKs due to its openness and broad appeal. They contributed changes upstream for their architecture and made sure architecture and chip-specific drivers and code were in the mainline kernel branches. They made it easier for the average embedded developer to use their chip sets. This adoption of Linux was just the beginning.

The next major development in the embedded space was the availability of software stacks such as Android. In 2007, when Android was released, the mobile handset software industry was fragmented with dozens of flavors of OSes and stacks to create mobile phones. Around that time, the launch of Apple’s iPhone had set the bar high for the smart phone market. Apple had managed the selection and integration process tightly from hardware, OS, apps, user interface (UI), manufacturing and even the service provider to deliver the phone. When introduced, the Android stack was another disruptive force in the smart phone market, making it possible to use a fully integrated stack aimed at mobile devices. It contained all the layers that phone OEMs in the past had to painfully put together. This enabled OEMS to focus on their differentiation at the UI and application layer to get to market faster. Seeing the Android example, other more complete stacks were created in the consumer space such as the set-top box stack and the gateway stack.

The embedded software development process historically has been a waterfall process. The chip is designed first with the OS bring-up and software development happening when the first silicon is available. Next, the semi enablement teams spend time to ensure that the tools, OSes and sample code actually work and create an SDK to ship to their ecosystems and early customers. By the time the average systems company can start developing devices it is a good six to nine months after the silicon has been released. There is also a good deal of duplication in effort on the software side between what the chip vendor does, the ecosystems and what tool companies do, as well as customers. There is delay and duplication in the value chain and no common tools, methodologies or processes used across the embedded commercial and open source ecosystems.

The increase in the complexity of SOCs and software involved in embedded designs, as well as the increased pressure to get differentiated products to market faster, makes it uncompetitive to do things serially, without some common practices, methodologies and an integrated hardware and software approach. More and more hardware companies are thinking of themselves as systems companies that need to take a user/application driven approach to designing their SOCs, and not the other way around. These innovative companies are creating integrated hardware-software teams that work together right from the start on a common design. They create a virtual prototype of the SOC as a model, which they share throughout the design process to collaborate and to ensure that not just the OS but any reference and sample applications and application software are also tested for application scenarios. Virtual Prototypes or software models of the hardware have become the common language that both teams can speak to debug and develop the platform.

Using such a common reference point avoids misunderstandings and creates a cohesive plan for the SOC. Being in software, these virtual platforms are instrumented for optimal debugging and visibility into the execution flow, providing a bird’s-eye view of a command and sequence the code is executed in. A more final prototype can be shared with lead customers months before the hardware is available, thereby winning sockets that would otherwise have been lost. The prototype also can be provided to the software enablement team and key field application engineers to get them ready for the chip launch. Often, the silicon enablement elements, such as tools and software are key criteria for OEMs in their selection of silicon. By the time the first silicon is available, the field is ready, lead customers have had a chance to evaluate the SOC and OS vendors are ready with their tools and support for the product. The impact on socket wins and cost of enablement is significant.

A common myth is that once hardware is available, the virtual prototype is no longer needed. Because the software-based prototype is easy to share, and provides more controlled debugging and wider visibility to trends, there are numerous other ways to leverage these prototypes. OEMs or systems companies can use this to extend and deepen their development, debug and test process. Just as the semi tools did, OEMs/Systems companies can use these prototypes as a common medium for debugging and communicating across distributed development teams.

Often hardware is hard to ship and can cause delay or be damaged in the delivery. The hardware complimentary virtual prototypes extend and enhance the software development process significantly. There is an increase in software integration complexity for OEMs because of the different sources of software. This makes regression and system testing very challenging. Virtual prototypes and the analysis tools that come with them provides the ability to test the whole stack with a holistic view of the flow, from the user interface down to the hardware. Imagine being able to see what the impact on power utilization is when a browser on a phone is opened—or the performance hit as a result of uploading a video to your Facebook account. All of these can help optimize applications and devices for power and performance. I wish I had these tools when I was debugging my OS in class.

So for a software geek like me, having the target hardware in a software layer is very attractive and makes it a valuable key element of the development process. It reduces the wait, provides more controls and empowers me to create a more integrated and complete device. I can see the possibilities and the transformation of embedded software development from looking at the problems with new eyes. I am won over by the benefits of virtual prototyping for embedded software development and see more of us using this to build smarter and cooler devices. I look forward to sharing more of my embedded software perspectives in the coming months as Achim’s guest blogger.

—Nithya Ruff is director of product marketing for Virtualizer Solutions at Synopsys.

 

In the last month, I had the opportunity to get some hands-on experience with hardware virtualization and hypervisors. My knowledge so far on this has been mainly limited to what I could read about it and what other people are saying about it. However, the PowerPoint slides I’ve seen leave a lot of white fog between the bullet items. This didn’t make me feel very comfortable talking about this topic myself; but, there was no escape. Hypervisors play an increasingly important role for system designers in context of supporting multiple guest operating systems on the same device, or taking advantage of ARM®’s new big.LITTLE^TM processing. The fog is not all gone, but let me provide you some insight on what I found out. As a disclaimer, I’m not going to (and I cannot) write an expert almanac about all the aspects of virtualization covering Xen, VMWare, etc. Instead, I’m going to focus on my personal experience that I believe will be relevant to you as well. This post is the starting point for a series on this topic in this blog.

Clearing up terminology confusion

The fact that this whole concept is (also) called “virtualization” is high on the list of the top challenges I was facing when explaining it to my colleagues, or management. As a background, I am working in technical marketing in the area of “virtual prototyping,” and our product is called “Virtualizer.” In order to avoid the same confusion here, let’s separate those terms in context of embedded systems:

1. Virtual prototypes/Virtual platforms (VP): A virtualized embedded system in the form of a simulation on a host PC for the purpose of developing the embedded system (HW and/or SW).
2. Virtualization using hypervisors: An embedded software (ESW) layer between the operating system (OS) and the embedded system hardware. The hypervisor can serve multiple guest operating systems and bridges differences of the real underlying hardware and the target hardware for which the OS has been built for.

I used a VP on my desktop PC to bring up ARM Linux along with an embedded hypervisor. Using a VP, I’ve been able to integrate, run and debug a software stack including Android, Linux and a hypervisor on an ARM big.LITTLE processing-based system. The VP integrates Fast Models from ARM to simulate the processors.

big.LITTLE processing?
The first example of this processing concept combines a high performance ARM Cortex™-A15 MPCore™ processor along with an energy efficient Cortex-A7 processor. The two main use-models which have been introduced by ARM are called “big.LITTLE Task Migration” and “big.LITTLE MP.” Those use models are nicely described by the Linaro organization.

The so called big.LITTLE Task Migration use model is that the software can seamlessly migrate from one processor to the other, depending on the use context and resulting performance requirements. It sounds like this requires a heavy re-write, or modification of the software stacks (e.g., Linux/Android) to support this; in fact, it doesn’t. This task migration is achieved by having the software run not on the hardware, but on-top of a new layer. This layer operates in the hypervisor mode and performs the task-migration without Linux/Android even knowing about it — although, the Linux/Android power management will initiate it as illustrated in the next figure.

Hypervisors and Interrupts
The task of the hypervisor software layer is to shield the hardware and payload (e.g., Linux/Android) software from directly communicating with each other. We call Linux/Android payload, as it is the software the user is intending to run. Why is a hypervisor needed? Interrupts are a good example here. Let’s assume we run multiple guest OSs on a system. Interrupts coming from the hardware could be for either one of the OSs. Therefore, a hypervisor needs to first intercept the interrupt from the system, and then decide to which guest OS it was addressed. For big.LITTLE processing it’s the other way around; here, we have multiple processor clusters sharing the same interrupt controller. The hypervisor (open source example available here) ensures the transparency of the clusters for the OS and does the task migration. But, the hypervisor needs its own interrupts and should not interfere with the OS.

Hardware Supported Interrupts Virtualization
In order to enable interrupt trapping in the hypervisor, ARM provides specific hardware support in its processors. In the processor’s hypervisor mode, a higher privileged exception vector allows trapping interrupts even before the OS can react on it. Thus, when an interrupt come in, it is the hypervisor that first handles it. If it is for the OS, then it will configure a virtualized interrupt controller, which is a replicate of the real interrupt controller seen by the OS. The OS will then handle the interrupt as if there was no hypervisor in between. Here, the MMU (Memory Management Unit) virtualization plays an important role as well. More about that in my next blog.

Interestingly, an interrupt has quite a way to go before it arrives at the user’s application, all the way from the hardware, through the hypervisor, into the OS. Virtual prototypes are extremely useful for debugging this chain, as all aspects of the system can be observed and traced as shown in the figure below. 

Keeping the overview on what is going on and having a bird’s eye view to assess where things go wrong are really helpful here. Where does the interrupt stop? Stuck in the hypervisor? Not arriving at the interrupt controller? The VP ensures that at any time the different layers of the SW and even HW can be traced and debugged to spot and identify software integration bugs. First of all you have clarity on which CPU is currently active. Second, you can spot the time where the actual interrupt arrives from the hardware. Next, tracing of the system is possible across the layers of the software, from the hypervisor up to Android. The VP tracing and debugging is aware of the hypervisor layer by tracking the mode that is exposed from the underlying CPU models (ARM Fast Models). Furthermore, debugging services are always available, because debugging does not depend on any embedded software daemons to function. This is very useful for debugging the interaction between the hypervisor and the Linux kernel. You can simply have one debugger attached to the hypervisor, and another one attached to the Linux kernel. This becomes really compelling when debugging the task migration. Even during phases where one CPU is powering down and the other is powering up, debugging and tracing is possible without limitation. This is a delicate phase as the entire context of one CPU is saved and subsequently restored by the other CPU. This can result is obscure and hard to find defects, for example,  if the context saving is incomplete because it forgets about saving the secure mode registers. I will write more about the task migration, in context of energy/performance optimizations in a follow up post.

If you’d like to see a live demonstration and explanation of all this, we’re presenting at the DesignWest Conference (formerly called the Embedded Systems Conference) in San Jose in the ARM Partner Pavilion and also in a joint technical session with ARM as part of the Android Summit. You can also see a demonstration from ARM and Synopsys at SNUG Silicon Valley this coming Monday, March 26^th at the Designer Community Expo or Tuesday, March27^th in the afternoon tutorial of the Systems track.

Transaction-level models are the main building blocks of virtual prototypes, which are used for early software development. In my last blog post, I briefly introduced the different kinds of software tasks and the implications for models. Today, I want to talk about the modeling requirements for early SoC bring up. As I mentioned, understanding the software requirements correctly provides two clear benefits: 1) it makes modeling easier through a more focused application and 2) it increases the value for the software developer through more tailored modeling capabilities such as debug features.

Within the context of pre-silicon software development for SoCs, the dominate use model for virtual prototypes is the bring-up of both the operating system and the chip validation suite. To bring up an operating system such as Linux, you must consider multiple aspects: primarily the CPU architecture specific code, the peripheral device specific drivers and the SoC/platform (aka machine) specific configuration.  Once a new CPU architecture is supported, or a device driver is available, they can be leveraged by the different platforms. Still, it is surprising how much code you have to write (and validate) to support a new platform. Of course, this platform support code will need to register all the different devices and set up the memory map and interrupts of the platform, but interestingly a lot of the platform specific code just deals with clock and voltage control (e.g. TI OMAP). Power domains are defined; voltage levels and frequencies are configured all over the place. Linux provides specific clock and regulator frameworks as well as the so called CPUFREQ framework for DVFS which are also used by the various power management frameworks.  Most of the settings are managed by the board’s power management IC (PMIC), or the on chip power-reset-clock-management-unit (PRCM).

This is not just happening for advanced features such as DVFS.  These techniques are also employed to get the simplest peripherals to work such as a UART. Before the UART driver can access the UART, it has to be clocked and powered. If this is not the case, the register interface of the UART will be in an undefined state and many TLMs do not take care of this situation today. In order to have a model support these aspects of the software, the model should perform a simple check to verify any abnormal operating conditions and inform about them. This simple addition to the model will greatly improve the value of the model and make software development and debugging more efficient.

Transaction Level Model - Clock/Voltage/Reset Sensitivity

Of course, it is necessary that these models are also equipped with ports that provide information about voltage and frequency, so that the model can check whether the levels are within acceptable limits. This does not mean that we need cycle accuracy with sensitivity to the clock, but rather we are interested in supplying the model with numeric information about the frequency and voltage.

The software is driving the clock tree and power network by configuring the power reset clock management (PRCM) unit or the power management IC. Thus, our virtual prototype should have an abstract model of this component that provides an accurate register interface. The internals of this model are not relevant in this context unless we also want to develop the firmware here. All that is required is that the clock and voltage lines are driven correctly by this model and the topologies of these networks are accurate.

Power Reset Clock Management Unit

So, in summary, with three relatively simple additions to an abstract, loosely timed virtual prototype we can greatly increase its value:

• TLM should check voltage and frequency values are in acceptable ranges
• A register accurate model of the power management IC and/or power management IC
• Accurate topology of the clock, reset and voltage networks.

By the way, did you know that wrong constraints and settings of the software-driven voltage regulators can result in hardware damage? I bet you’d love to be able to just restart your simulation rather than explaining to your management why the board can now only be used for decoration in the office lobby.

In this post, I would like to share some perspectives on the transaction-level models needed for the creation of virtual prototypes. Just recently, TLMCentral kicked off a contest seeking the “best” model for a mobile phone sensor device. What actually makes a “good” model?  The most ad-hoc answer to this question has historically been that the best model is the fastest and most accurate. However, if you ask about the reason for this, you probably won’t get a very precise answer. What does “accurate” mean? What is the definition of “fast”?  Will an accurate and fast model be available early enough to start software development before hardware is available and therefore meet time-to-market demands?

 

Simulation performance and accuracy (timing and function) are for sure valid aspects.  But, there is much more to consider.  The opportunity for the “best” model arises when user requirements are understood correctly.  When this happens, model creation can be greatly simplified and sped up. The model will have more value for the end-user at an even lower cost.

 

A good model, in my eyes, is one that serves the purpose of the user at the right point, on time at a proper cost. Here, the user task has to be carefully reviewed before selecting or creating a TLM model. What is the user intending to accomplish with the model? Simply identifying that a model should be used for software development is not the right level of diversification we are looking for. It does not constrain the requirements! Thus, we would again need a model that has to be able to do everything. While this is a challenge that often has to be tackled by TLM model providers, it is not something that you want to deal with when enabling a certain class of the end-users in your project.

 

In the remainder of the post I want to provide some software perspectives on models. In context of those perspectives I would like to introduce the aspects that make a model valuable.

 

The mobile phone application developer uses models in context of emulators being a backend to SDKs. This class of users demands that the models executes at speeds closer to the real device. For interface IP models, it is necessary for the model to be connected to the real world. E.g. the develop can use his/her real “iPhone” to drive the sensor model in the virtual prototype. S/He values the fact that he is able to record and repeat the stimuli for testing. It sounds complex, but doesn’t need to be. An application developer will not look underneath the APIs he is using which leaves great room for abstraction/simplification. In the context of Android, a sensor model may be a combination of a simple middleware along with a generic interface model (e.g. a UART) that is used to send and receive sensor data. This is close to the way it is realized in the Qemu Android emulator.

 

 

 

However, if the focus is middleware-, and not application-development, then this level of abstraction does not hold true anymore. Still, we typically have fewer requirements on simulation performance. The scenarios in which we are using the model are mainly in context of short embedded directed tests. E.g. For compliance testing, Android provides small sensor test applications for developers that are creating their own sensor middleware. The middleware is interfacing with the driver of the OS. Thus, our model should be accurate up to the level of the driver’s user interfaces.  This implies that the coarse grain states of the IP are modeled correctly as those are typically reflected in how the driver is operated. E.g. the sensor is activated and configured to provide updated gyroscope values every 500ms via the file “/sys/class/sensors/gyroscope”.  

 

To enable this, it is not required to realize such a model register accurate or register complete in the first place. Only a subset of the driver needs to be realized and functions such as clock, reset, voltage control can be neglected to simplify the modeling. The middleware developer will get more value through the additional debug and tracing capabilities we can put into the model. A good model will assist the developer and inform him/her about the coarse grain operation. Why are things happening in the model, or why are things not happening? This level of visibility can be of great help for understanding the IP and debugging the embedded software.

 

Someone who develops drivers or brings up a device in context of a board configuration will need more functional completeness and accuracy. Here, an accurate representation of the register map is almost mandatory. This also concerns functions for the power management integration such as clock, reset, and voltage sensitivity to bring-up and test for the Linux clock/voltage regulator frameworks. The model has to provide the proper response when configuring it through the register interface and also trigger interrupts. Therefore, a certain level of timing accuracy is required. However, this does not refer to having the data-path of the model being cycle- accurate/-approximate. The correct operation of the IP in the system context requires responses of the model at proper times. E.g. a timer will fire interrupts in periodic intervals. But again, model abstraction can be done to simplify the task. For the purpose of driver development or board bring up, the most important aspect is the control plane. Here, e.g. the data plane of a video accelerator IP can be abstracted to just show dummy data.

 

In summary, creating a model efficiently requires careful review of the end user design task. The model development can be staged and aligned with the software development plan, to incrementally enable new software development tasks. Understanding the requirements correctly allows creating models at lower cost and with higher end-user value.