How to win over the embedded software developer, their customer and their boss.

By Nithya Ruff
Achim Nohl was taking a well-deserved vacation last week and asked me to be his guest blogger. To many of you who are regular readers of Achim’s blog, I am new to Synopsys and joined only a few weeks ago to manage the Virtual Prototyping product. I came from Wind River where I managed Embedded Linux product marketing. Having come from 20+ years of managing software of all kinds, I saw the guest blog as a perfect way to talk about a software marketeer’s view of virtual prototypes and embedded software development.

I still vividly remember the manual and tedious process of debugging the operating system I had written in my computer science OS 401 class. Not only was the data not all in one place, it involved reams of paper and tracing the calls manually. Frankly, it does not feel any different these days in the world of embedded software. Every element of the device stack is put together separately, and often there is no holistic view of the entire device. Debugging and testing is more of an art than a repeatable and systematic science. Let’s look at why this is so.

Embedded systems cover everything from very small systems inside watches, to sensors in a building, to complex multicore-based core-of-the-network systems. They share the characteristic of being dedicated to a specific function and not being general-purpose. These systems often require real-time performance and a smaller footprint. Everything is customized to fit the task—the Operating System (OS), middleware and applications. Given the wide range of applications, the choice of the hardware is specific to the device being created. The software stack created is influenced by the hardware characteristics and behaviors. Hence, there are all kinds of hardware architectures, and even many variations of custom hardware used in the embedded device space. In fact, hardware is the first and a key component of the decision making process for a project. The rest is built around the hardware selection.

In the late 1990s and early 2000s the success of Linux in the server space began to bleed over into the embedded space, especially in networking. Linux was inexpensive to acquire and provided almost everything needed, such as tools, protocols and trained people. Most of all, it provided access to source code and could be modified as needed. Linux also was available on other architectures such as ARM, MIPS, and PPC and X86. This was important to embedded developers so they could select the right hardware for the task at hand. Chip vendors had started using Linux in their boot-up software and in SDKs due to its openness and broad appeal. They contributed changes upstream for their architecture and made sure architecture and chip-specific drivers and code were in the mainline kernel branches. They made it easier for the average embedded developer to use their chip sets. This adoption of Linux was just the beginning.

The next major development in the embedded space was the availability of software stacks such as Android. In 2007, when Android was released, the mobile handset software industry was fragmented with dozens of flavors of OSes and stacks to create mobile phones. Around that time, the launch of Apple’s iPhone had set the bar high for the smart phone market. Apple had managed the selection and integration process tightly from hardware, OS, apps, user interface (UI), manufacturing and even the service provider to deliver the phone. When introduced, the Android stack was another disruptive force in the smart phone market, making it possible to use a fully integrated stack aimed at mobile devices. It contained all the layers that phone OEMs in the past had to painfully put together. This enabled OEMS to focus on their differentiation at the UI and application layer to get to market faster. Seeing the Android example, other more complete stacks were created in the consumer space such as the set-top box stack and the gateway stack.

The embedded software development process historically has been a waterfall process. The chip is designed first with the OS bring-up and software development happening when the first silicon is available. Next, the semi enablement teams spend time to ensure that the tools, OSes and sample code actually work and create an SDK to ship to their ecosystems and early customers. By the time the average systems company can start developing devices it is a good six to nine months after the silicon has been released. There is also a good deal of duplication in effort on the software side between what the chip vendor does, the ecosystems and what tool companies do, as well as customers. There is delay and duplication in the value chain and no common tools, methodologies or processes used across the embedded commercial and open source ecosystems.

The increase in the complexity of SOCs and software involved in embedded designs, as well as the increased pressure to get differentiated products to market faster, makes it uncompetitive to do things serially, without some common practices, methodologies and an integrated hardware and software approach. More and more hardware companies are thinking of themselves as systems companies that need to take a user/application driven approach to designing their SOCs, and not the other way around. These innovative companies are creating integrated hardware-software teams that work together right from the start on a common design. They create a virtual prototype of the SOC as a model, which they share throughout the design process to collaborate and to ensure that not just the OS but any reference and sample applications and application software are also tested for application scenarios. Virtual Prototypes or software models of the hardware have become the common language that both teams can speak to debug and develop the platform.

Using such a common reference point avoids misunderstandings and creates a cohesive plan for the SOC. Being in software, these virtual platforms are instrumented for optimal debugging and visibility into the execution flow, providing a bird’s-eye view of a command and sequence the code is executed in. A more final prototype can be shared with lead customers months before the hardware is available, thereby winning sockets that would otherwise have been lost. The prototype also can be provided to the software enablement team and key field application engineers to get them ready for the chip launch. Often, the silicon enablement elements, such as tools and software are key criteria for OEMs in their selection of silicon. By the time the first silicon is available, the field is ready, lead customers have had a chance to evaluate the SOC and OS vendors are ready with their tools and support for the product. The impact on socket wins and cost of enablement is significant.

A common myth is that once hardware is available, the virtual prototype is no longer needed. Because the software-based prototype is easy to share, and provides more controlled debugging and wider visibility to trends, there are numerous other ways to leverage these prototypes. OEMs or systems companies can use this to extend and deepen their development, debug and test process. Just as the semi tools did, OEMs/Systems companies can use these prototypes as a common medium for debugging and communicating across distributed development teams.

Often hardware is hard to ship and can cause delay or be damaged in the delivery. The hardware complimentary virtual prototypes extend and enhance the software development process significantly. There is an increase in software integration complexity for OEMs because of the different sources of software. This makes regression and system testing very challenging. Virtual prototypes and the analysis tools that come with them provides the ability to test the whole stack with a holistic view of the flow, from the user interface down to the hardware. Imagine being able to see what the impact on power utilization is when a browser on a phone is opened—or the performance hit as a result of uploading a video to your Facebook account. All of these can help optimize applications and devices for power and performance. I wish I had these tools when I was debugging my OS in class.

So for a software geek like me, having the target hardware in a software layer is very attractive and makes it a valuable key element of the development process. It reduces the wait, provides more controls and empowers me to create a more integrated and complete device. I can see the possibilities and the transformation of embedded software development from looking at the problems with new eyes. I am won over by the benefits of virtual prototyping for embedded software development and see more of us using this to build smarter and cooler devices. I look forward to sharing more of my embedded software perspectives in the coming months as Achim’s guest blogger.

—Nithya Ruff is director of product marketing for Virtualizer Solutions at Synopsys.

 

In the last month, I had the opportunity to get some hands-on experience with hardware virtualization and hypervisors. My knowledge so far on this has been mainly limited to what I could read about it and what other people are saying about it. However, the PowerPoint slides I’ve seen leave a lot of white fog between the bullet items. This didn’t make me feel very comfortable talking about this topic myself; but, there was no escape. Hypervisors play an increasingly important role for system designers in context of supporting multiple guest operating systems on the same device, or taking advantage of ARM®’s new big.LITTLE^TM processing. The fog is not all gone, but let me provide you some insight on what I found out. As a disclaimer, I’m not going to (and I cannot) write an expert almanac about all the aspects of virtualization covering Xen, VMWare, etc. Instead, I’m going to focus on my personal experience that I believe will be relevant to you as well. This post is the starting point for a series on this topic in this blog.

Clearing up terminology confusion

The fact that this whole concept is (also) called “virtualization” is high on the list of the top challenges I was facing when explaining it to my colleagues, or management. As a background, I am working in technical marketing in the area of “virtual prototyping,” and our product is called “Virtualizer.” In order to avoid the same confusion here, let’s separate those terms in context of embedded systems:

1. Virtual prototypes/Virtual platforms (VP): A virtualized embedded system in the form of a simulation on a host PC for the purpose of developing the embedded system (HW and/or SW).
2. Virtualization using hypervisors: An embedded software (ESW) layer between the operating system (OS) and the embedded system hardware. The hypervisor can serve multiple guest operating systems and bridges differences of the real underlying hardware and the target hardware for which the OS has been built for.

I used a VP on my desktop PC to bring up ARM Linux along with an embedded hypervisor. Using a VP, I’ve been able to integrate, run and debug a software stack including Android, Linux and a hypervisor on an ARM big.LITTLE processing-based system. The VP integrates Fast Models from ARM to simulate the processors.

big.LITTLE processing?
The first example of this processing concept combines a high performance ARM Cortex™-A15 MPCore™ processor along with an energy efficient Cortex-A7 processor. The two main use-models which have been introduced by ARM are called “big.LITTLE Task Migration” and “big.LITTLE MP.” Those use models are nicely described by the Linaro organization.

The so called big.LITTLE Task Migration use model is that the software can seamlessly migrate from one processor to the other, depending on the use context and resulting performance requirements. It sounds like this requires a heavy re-write, or modification of the software stacks (e.g., Linux/Android) to support this; in fact, it doesn’t. This task migration is achieved by having the software run not on the hardware, but on-top of a new layer. This layer operates in the hypervisor mode and performs the task-migration without Linux/Android even knowing about it — although, the Linux/Android power management will initiate it as illustrated in the next figure.

Hypervisors and Interrupts
The task of the hypervisor software layer is to shield the hardware and payload (e.g., Linux/Android) software from directly communicating with each other. We call Linux/Android payload, as it is the software the user is intending to run. Why is a hypervisor needed? Interrupts are a good example here. Let’s assume we run multiple guest OSs on a system. Interrupts coming from the hardware could be for either one of the OSs. Therefore, a hypervisor needs to first intercept the interrupt from the system, and then decide to which guest OS it was addressed. For big.LITTLE processing it’s the other way around; here, we have multiple processor clusters sharing the same interrupt controller. The hypervisor (open source example available here) ensures the transparency of the clusters for the OS and does the task migration. But, the hypervisor needs its own interrupts and should not interfere with the OS.

Hardware Supported Interrupts Virtualization
In order to enable interrupt trapping in the hypervisor, ARM provides specific hardware support in its processors. In the processor’s hypervisor mode, a higher privileged exception vector allows trapping interrupts even before the OS can react on it. Thus, when an interrupt come in, it is the hypervisor that first handles it. If it is for the OS, then it will configure a virtualized interrupt controller, which is a replicate of the real interrupt controller seen by the OS. The OS will then handle the interrupt as if there was no hypervisor in between. Here, the MMU (Memory Management Unit) virtualization plays an important role as well. More about that in my next blog.

Interestingly, an interrupt has quite a way to go before it arrives at the user’s application, all the way from the hardware, through the hypervisor, into the OS. Virtual prototypes are extremely useful for debugging this chain, as all aspects of the system can be observed and traced as shown in the figure below. 

Keeping the overview on what is going on and having a bird’s eye view to assess where things go wrong are really helpful here. Where does the interrupt stop? Stuck in the hypervisor? Not arriving at the interrupt controller? The VP ensures that at any time the different layers of the SW and even HW can be traced and debugged to spot and identify software integration bugs. First of all you have clarity on which CPU is currently active. Second, you can spot the time where the actual interrupt arrives from the hardware. Next, tracing of the system is possible across the layers of the software, from the hypervisor up to Android. The VP tracing and debugging is aware of the hypervisor layer by tracking the mode that is exposed from the underlying CPU models (ARM Fast Models). Furthermore, debugging services are always available, because debugging does not depend on any embedded software daemons to function. This is very useful for debugging the interaction between the hypervisor and the Linux kernel. You can simply have one debugger attached to the hypervisor, and another one attached to the Linux kernel. This becomes really compelling when debugging the task migration. Even during phases where one CPU is powering down and the other is powering up, debugging and tracing is possible without limitation. This is a delicate phase as the entire context of one CPU is saved and subsequently restored by the other CPU. This can result is obscure and hard to find defects, for example,  if the context saving is incomplete because it forgets about saving the secure mode registers. I will write more about the task migration, in context of energy/performance optimizations in a follow up post.

If you’d like to see a live demonstration and explanation of all this, we’re presenting at the DesignWest Conference (formerly called the Embedded Systems Conference) in San Jose in the ARM Partner Pavilion and also in a joint technical session with ARM as part of the Android Summit. You can also see a demonstration from ARM and Synopsys at SNUG Silicon Valley this coming Monday, March 26^th at the Designer Community Expo or Tuesday, March27^th in the afternoon tutorial of the Systems track.

Transaction-level models are the main building blocks of virtual prototypes, which are used for early software development. In my last blog post, I briefly introduced the different kinds of software tasks and the implications for models. Today, I want to talk about the modeling requirements for early SoC bring up. As I mentioned, understanding the software requirements correctly provides two clear benefits: 1) it makes modeling easier through a more focused application and 2) it increases the value for the software developer through more tailored modeling capabilities such as debug features.

Within the context of pre-silicon software development for SoCs, the dominate use model for virtual prototypes is the bring-up of both the operating system and the chip validation suite. To bring up an operating system such as Linux, you must consider multiple aspects: primarily the CPU architecture specific code, the peripheral device specific drivers and the SoC/platform (aka machine) specific configuration.  Once a new CPU architecture is supported, or a device driver is available, they can be leveraged by the different platforms. Still, it is surprising how much code you have to write (and validate) to support a new platform. Of course, this platform support code will need to register all the different devices and set up the memory map and interrupts of the platform, but interestingly a lot of the platform specific code just deals with clock and voltage control (e.g. TI OMAP). Power domains are defined; voltage levels and frequencies are configured all over the place. Linux provides specific clock and regulator frameworks as well as the so called CPUFREQ framework for DVFS which are also used by the various power management frameworks.  Most of the settings are managed by the board’s power management IC (PMIC), or the on chip power-reset-clock-management-unit (PRCM).

This is not just happening for advanced features such as DVFS.  These techniques are also employed to get the simplest peripherals to work such as a UART. Before the UART driver can access the UART, it has to be clocked and powered. If this is not the case, the register interface of the UART will be in an undefined state and many TLMs do not take care of this situation today. In order to have a model support these aspects of the software, the model should perform a simple check to verify any abnormal operating conditions and inform about them. This simple addition to the model will greatly improve the value of the model and make software development and debugging more efficient.

Transaction Level Model - Clock/Voltage/Reset Sensitivity

Of course, it is necessary that these models are also equipped with ports that provide information about voltage and frequency, so that the model can check whether the levels are within acceptable limits. This does not mean that we need cycle accuracy with sensitivity to the clock, but rather we are interested in supplying the model with numeric information about the frequency and voltage.

The software is driving the clock tree and power network by configuring the power reset clock management (PRCM) unit or the power management IC. Thus, our virtual prototype should have an abstract model of this component that provides an accurate register interface. The internals of this model are not relevant in this context unless we also want to develop the firmware here. All that is required is that the clock and voltage lines are driven correctly by this model and the topologies of these networks are accurate.

Power Reset Clock Management Unit

So, in summary, with three relatively simple additions to an abstract, loosely timed virtual prototype we can greatly increase its value:

• TLM should check voltage and frequency values are in acceptable ranges
• A register accurate model of the power management IC and/or power management IC
• Accurate topology of the clock, reset and voltage networks.

By the way, did you know that wrong constraints and settings of the software-driven voltage regulators can result in hardware damage? I bet you’d love to be able to just restart your simulation rather than explaining to your management why the board can now only be used for decoration in the office lobby.

In this post, I would like to share some perspectives on the transaction-level models needed for the creation of virtual prototypes. Just recently, TLMCentral kicked off a contest seeking the “best” model for a mobile phone sensor device. What actually makes a “good” model?  The most ad-hoc answer to this question has historically been that the best model is the fastest and most accurate. However, if you ask about the reason for this, you probably won’t get a very precise answer. What does “accurate” mean? What is the definition of “fast”?  Will an accurate and fast model be available early enough to start software development before hardware is available and therefore meet time-to-market demands?

 

Simulation performance and accuracy (timing and function) are for sure valid aspects.  But, there is much more to consider.  The opportunity for the “best” model arises when user requirements are understood correctly.  When this happens, model creation can be greatly simplified and sped up. The model will have more value for the end-user at an even lower cost.

 

A good model, in my eyes, is one that serves the purpose of the user at the right point, on time at a proper cost. Here, the user task has to be carefully reviewed before selecting or creating a TLM model. What is the user intending to accomplish with the model? Simply identifying that a model should be used for software development is not the right level of diversification we are looking for. It does not constrain the requirements! Thus, we would again need a model that has to be able to do everything. While this is a challenge that often has to be tackled by TLM model providers, it is not something that you want to deal with when enabling a certain class of the end-users in your project.

 

In the remainder of the post I want to provide some software perspectives on models. In context of those perspectives I would like to introduce the aspects that make a model valuable.

 

The mobile phone application developer uses models in context of emulators being a backend to SDKs. This class of users demands that the models executes at speeds closer to the real device. For interface IP models, it is necessary for the model to be connected to the real world. E.g. the develop can use his/her real “iPhone” to drive the sensor model in the virtual prototype. S/He values the fact that he is able to record and repeat the stimuli for testing. It sounds complex, but doesn’t need to be. An application developer will not look underneath the APIs he is using which leaves great room for abstraction/simplification. In the context of Android, a sensor model may be a combination of a simple middleware along with a generic interface model (e.g. a UART) that is used to send and receive sensor data. This is close to the way it is realized in the Qemu Android emulator.

 

 

 

However, if the focus is middleware-, and not application-development, then this level of abstraction does not hold true anymore. Still, we typically have fewer requirements on simulation performance. The scenarios in which we are using the model are mainly in context of short embedded directed tests. E.g. For compliance testing, Android provides small sensor test applications for developers that are creating their own sensor middleware. The middleware is interfacing with the driver of the OS. Thus, our model should be accurate up to the level of the driver’s user interfaces.  This implies that the coarse grain states of the IP are modeled correctly as those are typically reflected in how the driver is operated. E.g. the sensor is activated and configured to provide updated gyroscope values every 500ms via the file “/sys/class/sensors/gyroscope”.  

 

To enable this, it is not required to realize such a model register accurate or register complete in the first place. Only a subset of the driver needs to be realized and functions such as clock, reset, voltage control can be neglected to simplify the modeling. The middleware developer will get more value through the additional debug and tracing capabilities we can put into the model. A good model will assist the developer and inform him/her about the coarse grain operation. Why are things happening in the model, or why are things not happening? This level of visibility can be of great help for understanding the IP and debugging the embedded software.

 

Someone who develops drivers or brings up a device in context of a board configuration will need more functional completeness and accuracy. Here, an accurate representation of the register map is almost mandatory. This also concerns functions for the power management integration such as clock, reset, and voltage sensitivity to bring-up and test for the Linux clock/voltage regulator frameworks. The model has to provide the proper response when configuring it through the register interface and also trigger interrupts. Therefore, a certain level of timing accuracy is required. However, this does not refer to having the data-path of the model being cycle- accurate/-approximate. The correct operation of the IP in the system context requires responses of the model at proper times. E.g. a timer will fire interrupts in periodic intervals. But again, model abstraction can be done to simplify the task. For the purpose of driver development or board bring up, the most important aspect is the control plane. Here, e.g. the data plane of a video accelerator IP can be abstracted to just show dummy data.

 

In summary, creating a model efficiently requires careful review of the end user design task. The model development can be staged and aligned with the software development plan, to incrementally enable new software development tasks. Understanding the requirements correctly allows creating models at lower cost and with higher end-user value.  

With this post, I would like to continue the topic of my earlier post “Can we stop power-hungry bugs from clawing their way through application software stacks?” In my previous post, I wrote about the difficulties software developers face with writing battery friendly software. I indicated that virtual prototypes (VPs) can address many of those challenges by providing visibility into the energy consumption of the software under development. Now, I would like to shed some light on how virtual prototypes can accomplish this.

In order to give software developers the ability to profile the impact of software changes on energy consumption, the underlying execution target needs to expose the energy consumed by the different components that make up the system – over time and ideally through the perspective of software activity. Therefore, the profiling environment needs to be able to quantify the energy contribution from the different platform components.  However, development boards with power measurement probes are only accessible for a handful of software engineers, and are typically made available too late in the design cycle to allow development and testing of the power management software functions.  Moreover, they are not available to the community of application software providers whose software has significant influence on a battery’s mileage.

The virtual prototypes that are typically used today by software developers have no notion of power. Their models are purely functional, abstract in their timing behavior and with no link to implementation or silicon process technology. If you claim that VPs can still be very helpful for software developers to get energy under control, it is likely that hardware engineers will view you as a charlatan since, from their perspective, gate-level simulation is a minimum requirement to gain a decent impression on the energy footprint of the system. This is somewhat true, but the hardware engineers have a different problem to solve. Software developers do not intend to optimize logic circuits. Instead, they need to make sure they utilize the system power management features in the best possible way. A debuggable, repeatable, deterministic, scenario-driven trend-based analysis of the system’s dynamic power consumption is more relevant for them, then the ability to derive the exact energy consumption of the system (uWh). This task is perfectly feasible with a VP.

Here, the characterization of the important power states and events in the system components is the most relevant task. For example, a CPU can switch to the lower power idle state if software synchronization is performed via a semaphore rather than a spin-lock style active wait. The LCD brightness can be reduced if no user interface interaction is required. A position update can be done via 3G or GPS with different power states and durations. Exposing the power states is of great benefit to the software developer, but even energy-trend estimation can be accomplished with coarse grain, state/event-based characterization from datasheets.

The question is will the incorporation of power data into system simulation simply increase the complexity of the modeling problem? In addition to making sure that we have all of the functional models, do we now also need all of the associated power models? The answer is no. Power and functionality can be orthogonalized. With VPs, power models can be provided as an overlay to the functionality of the VP.  Through the introspection capabilities of a VP, functional models can drive power model overlays through observable properties such as signals, registers or other instrumentation. This allows the user to create power models for only the aspects of the system s/he is interested in.

A single VP can be characterized to reflect the energy dynamics for very different system flavors. A functional Ethernet model can be characterized to mimic different hardware vendor parts and can be even characterized to mimic the dynamic power of a WiFi radio. A WiFi radio may transmit/receive in low power or high power state depending on the data rate requested by the software. Such a coarse grain WiFi power model can be driven by inspecting the data rate of the generic Ethernet model.  This is sufficient for many software design decisions and proven by the success of the Power Tutor Android Application. This Android application comes with power models for various hand-sets (e.g. HTC Dream/Magic), and implements such a WiFi model with surprising accuracy. In this case, the power models are driven by embedded software.

Here lies another big opportunity for VPs. We can use embedded software functions (e.g. driver suspend, resume, send) to realize power models for parts of the system that are not even functionally modeled in the VP. Now we are starting to use VP as an enabling tool for software developers, rather than a pure simulation model of a piece of hardware.

In the future, we expect that temperature will become part of the puzzle for the software developer. Power consumption is highly temperature dependent. Interestingly, this impacts the software developer in the choice of CPU idle management strategy (Run-Fast-Then-Stop or Dynamic-Voltage-Frequency-Scaling).

There is much more to say with regard to this topic, even going beyond the software perspective and how VPs can help to architect power friendly platforms. Also, there are a lot of imperfect aspects such as standardization of formats etc. However, already today, there is a great chance to get some aspects of the pest under control using virtual prototyping.

Have a relaxing christmas!

Identifying and describing power issues is tough, let alone trying to solve them. “Power” issues can be very diverse. It’s even more difficult to explain how virtual prototypes can help to analyze “power” consumption. We often approach it by introducing how power information can be reflected in virtual prototype models, but there are many different goals and conflicting views on accuracy, granularity, modeling approach, data sources, etc.

Some engineers will care about power, but most will care about energy efficiency. Some engineers will look at just a single block, while others care about processor subsystems and entire platforms. Some software engineers focus on a single piece of software, others care about the entire software stack.

So instead, we need to approach it from the software engineers’ intent. For example, software stacks using Android with multiple tens of millions of lines of code implement various power-saving strategies on almost every software layer, starting at the OS kernel and ending up in the application layer. A software power inefficiency or malfunction can quickly cause a 5x drop in device standby time. A Google example shows that a simple RSS feed application that wakes up the phone every 10 minutes for just 8 seconds at a time to do some updates, via the Internet, can cut the standby time of the phone in half. An almost endless list of battery drain problems reported by end-users can be found here. Of course, the most prominent example is the recent iOS bug. Why is so difficult to get this under control? Let’s examine some of the key issues:

 1. Profiling the impact of software changes on energy. A software developer needs to be able to determine the relative impact of his implementation choices on energy. For example, is it “better” to use a spin-lock or a semaphore in the Android sensor poll function when waiting for accelerometer data from the hardware? A spin-lock is faster, but does not allow the CPU to go idle while waiting. The developer needs to find the right balance between responsiveness and energy consumption. While this example only looks at the CPU, most cases are concerned with the efficient use of the other HW components, too.

Here’s another example: In an application that requires location information, is it “better” to use coarse-grained updates from the GPS, or fine-grained updates for 3G? The data needed to compare the two options are as follows: 1) the time it takes to complete the task, and 2) the power level to compute the energy (e.g. GPS: 25 seconds * 140mA at about 1mAh, Network: 2 seconds * 180mA  at about 0.1mAh). Again, the decision cannot be made only by the energy savings. It also requires an understanding of the performance demands of the particular service. Are we looking at a driving navigation or a walking navigation? Both would have different requirements on the precision.

 2. Interference and side effects of local software changes on other modules. An application or service that requires a periodic network update can trigger a cascade. Other less important services may just be waiting for a connection to be established. So instead of doing a 1 second update every 10 minutes we may easily trigger a 1 minute online activity caused by waking up background applications. Here, the software developer needs to understand those cascades through an energy profile that shows an unexpectedly high peak and an estimation of the resulting standby time. Handset makers face this interference challenge when integrating multiple software modules from various suppliers.

 3. Ensuring that multiple concurrent power management frameworks collaborate. System-wide and runtime power management are just two examples in the Linux kernel that need to collaborate. The former turns the whole phone into a suspend state and wakes it up without loss of data. The latter controls power states of hardware components while the phone is used. A typical issue is when a component that has been shut down by the runtime power-management  prevents the device from going into a system-wide suspension because it needs to be woken up first. Power management debugging can easily become a nightmare, because even debugging and tracing services are impacted and may shut down during suspend/resume phases. Moreover, wrong settings in the constraints for the voltage regulator drivers can even result in lasting hardware damage.

 4. Many inefficiencies and malfunctions only appear in usage context sensitive scenarios. Energy inefficiencies or power malfunctions are highly context-sensitive. The context is defined by how the device is interacting with the environment. Next to the user, the device and the software stack are linked to the environment through network 3G, WiFi, NFC/Bluetooth, cameras and many other sensors (acceleration, orientation, magnetic field, proximity, temperature, etc.). When optimizing software for energy, the usage scenario that drives the stimuli to the device needs to be repeatable in order to get deterministic and comparable results. Testing requires automation of those scenarios along with a certain level of randomization. During testing, developers need to check that all applications/services are implemented according to the coding guidelines (e.g. for application using the sensors, the programming reference says: “Always make sure to disable sensors you don’t need, especially when your activity is paused. Failing to do so can drain the battery in just a few hours. Note that the system will not disable sensors automatically when the screen turns off.”)

 Looking at the above challenges, it becomes clear that to effectively debug and optimize power using virtual prototypes we have to look beyond it as just a way to instrument the models with power information. In my next blogs I will explain how the four important requirements mentioned in this blog can be successfully addressed using virtual prototypes. Maybe we can tone down those software bugs’ appetites.

Virtual prototypes are essential to effectively debugging a system and still meeting market windows.

In my last blogs I have been focusing on introducing the technical advantages of virtual prototypes in the context of debugging embedded software. In this blog I would like to introduce how this can impact an entire supply chain.

The increasing complexity of software in terms of code-size, functions and layers, along with multicore SoCs, also demands more capable debug solutions. Those debug solutions have to go beyond a single CPU and provide a holistic view of the entire system with all its signals and registers for internal and external communication. A virtual prototype can deliver on those requirements independent of the hardware design schedule. Early, more productive and predictable debugging provides real advantages to VP adopters today. But it becomes even more compelling when you extend these advantages to your own customer, or even the customer’s customer. In the context of a supply chain, the impact of this time advantage can be multiplied with each stakeholder.

For example, Altera recently announced availability of the “Industry’s First Virtual Target for Software Development on SoC FPGAs.” “Virtual Target” is the term used by Altera to describe the virtual prototype (VP) for its SoC FPGA. Altera has identified VP as an essential tool that enables its customers to bring up their software earlier, more efficiently and with less risk. This again enables their customers to be ready for the market faster, with better quality software.

However, what’s also important for successful VP adoption by software developers is the support of a standard software development ecosystem. Before being able to educate software developers and convince them of the value of new complementary, VP-enabled debugging methods, it is important to make sure the software developers can use the software tools they are familiar with. I have seen that users quickly realize how much more they can suddenly do with a VP. Often this requires just a few hours of sitting together and doing some hands-on exercises with the user’s software.

When VPs are used to virtualize FPGAs and enable software development for custom hardware extensions, a few more aspects should be considered. In essence, the extension of the VP with hardware such as accelerators or peripherals is important. Otherwise, the VP cannot be used for custom device-specific driver development. In this case, the extension is achieved through an FPGA-in-the-loop extension where the real FPGA is connected to the VP using a virtualized PCIe interface. The advantage of this approach is that RTL can be re-used by the customer. Peripherals can be tested with the real hardware PHYs and real I/O. Custom accelerators, for example video and wireless, can be tested with the final targeted performance in context of the software. Altera offers this extension with its new SoC FPGA in its recently announced Virtual Target.

Of course, Altera is not the first to make this connection between early software development and the supply chain’s needs. The mobile industry has been working on this for some time. As software complexity grows at even the lowest layers of the SW stack, such as the OS kernel, drivers and middleware, the benefit of early customer enablement becomes more and more important for SoC vendors.

For sure, the general concept of using simulation for supply chain enablement is not new. In the context of enabling application developers for mobile phones, simulator-based SDKs have a long history. A big success story for simulation-based SDKs is, of course, the iPhone SDK and Google Android SDK. The first phone, the HTC Dream, was released in October 2008. Currently there are 167 apps available. To a large extend this has been enabled through the Android SDK, which Google released as an “early look SDK” version in 2007. The Android SDK uses the popular Qemu emulator for ARM as a back end that simulates an ARM-based hand-set. This is similar to a VP, but in general is more generic as some HW is even functionally abstracted. Google has achieved two important goals with this:

1. Google has educated a huge SW community on its product.
2. Google got a significant amount of feedback from developers that are already familiar with the iPhone to improve Android for its 1.0 release in September 2008

Today mobile device manufacturers are continuously extending their software-based emulators to support their custom hardware and win new software developers for it. For example, Kyocera is providing add-ons to simulate its “Dual Screen” devices, LG offers support for “Real 3D” and Samsung enables SW developers for their “Galaxy” Tablet.

Again, a significant additional benefit is that customers can provide feedback much earlier about concept flaws or unmet requirements. And when using a VP this can happen early enough so that problems can still be fixed without going through silicon re-spin. While the mobile device industry seems an obvious case, it’s interesting to note that the FPGA industry’s complexity has advanced enough that Altera has invested in virtual prototyping. So, who’s next? Xilinx?

In my last post I introduced the debugging challenges during porting, or developing native code, for Android.  This time I would like to outline how virtual prototypes can enable  software debugging and perform in an even better way. Before I describe what “better” refers to exactly, I want to shed some light on some prominent myths around the value of VPs for debugging.

Whenever VPs are mentioned in the context of debugging, it’s claimed that they offer many debugging advantages over real hardware. It is said that VPs have better controllability, visibility and provide determinism. But is this really true? And what does this really mean for a software developer?

For many people, a VP is nothing more than a piece of software simulating a HW/SW system. This software (the VP) can be debugged on the host PC of the user. Using the host’s software debugging tools, or VP vendor-specific tools, the user is able to inspect all aspects of the platform such as memories, registers and signals. Also, he/she can suspend the simulation in the host debugger, and above all has full control over the VP. The next time anyone executes the VP it is likely that it will perform the same functions in the same order as before, and that its behavior will be fully deterministic (as long as there is no interaction with non-deterministic external components such as the user). This all sounds great, but it does not consider the real needs of a software developer.

“Run-mode” software debugging using a VP

While a software developer is definitely thankful for inspecting the hardware beyond what is possible with a software debugger, in the first place he/she will still need to debug software.  This is of course also possible with a VP. Most VPs do contain virtualized interfaces such as USB, Ethernet or UART. Those interfaces allow a software debugger to connect to an embedded software agent. This agent establishes the connection between the debug target and the debugger. Now a VP can be easily combined with arbitrary software debug tools and even integrated into SDKs. As an example, the Google Android SDK connects to the debug target through Ethernet. This works perfectly with a VP.

Interestingly, this is not any different from the way it is done with real hardware. But unfortunately, also no real advantages exist. The praised controllability, visibility and determinism has to be re-evaluated. The control is now with the software debugger. When you suspend using the SW debugger, the VP will keep running. Timers, watchdogs and other SW processes continue to work (so called “run mode” debugging). In order to inspect the HW and correlate the platform state with the software state, you would need to suspend the VP. But, if you suspend the VP, then the debugger communication will likely time out. Remember, the embedded software agent is also halted.  Also, the agent itself influences the execution of the overall system and has side effects. Now, the determinism is also gone. But, there is a solution to combine the best of both worlds.

Virtual prototypes as debug servers

To marry the advantages of VPs and software debuggers, the VP simulation framework needs to take over the task of the software agent and act as a debug server. The debug server establishes the link between the software debugger and the VP simulation. Now we can take advantage of the VP’s “view from the top” and serve even multiple software debuggers with information from all over the platform. The debug server also orchestrates the control and makes sure all debug views are synchronized. The debug server itself does not interfere with the execution of the platform. Neither the hardware models, nor the software, can tell whether they are debugged or not. Now the VP can really play its hand and deliver all the advantages to the software engineers.  Summarizing, it is important to recognize that a VP is more than a model. A VP requires tooling to make it useful beyond simulation.

Coming back to Android and my previous post: The 21 steps to set up a debugger to debug native code can now be reduced to two. Using a VP with a debug server, you set a breakpoint at the address and process that you are interested in; afterward you simply connect the software debugger. No more worries about when and how to launch the gdb-server agent.  But VPs even help in the context of using run-mode debugging of Android native code libraries. The user of the VP can easily inject an endless loop within the start-up code of an embedded software process just by writing the instructions into the VP memories. No change and recompilation of the embedded software is needed, as required in the best practice.

Using the Google Search Debugger

I am spending a reasonable amount of my time helping our users to bring up Android on their chipsets or devices. I am not an Android expert, I gathered my limited knowledge about Android mainly during debugging. I must admit that the first tool I am using when a problem appears is a Google Search. This brings me to the respective expert forums where I hope somebody else has already done the job for me. With Android, this works reasonable well as Android produces an extensive debug log from which I can copy and paste messages into the search bar. With more than 7000 users and 16000 messages just in the Android porting group, chances are high to find something useful. Often the solution to the problem is already there for me.

Android forums did sometimes also the opposite and put me and a completely wrong track. The fact that a symptom is the same, it does not imply the root cause is also the same. Android is a huge SW stack sitting on-top of Linux, which runs on the HW. A bug is often only triggered because of a special combination of Android/Linux version & patch  combinations and the specialties of the underlying HW platform. How to get back from Googling into Thinking? Today, I try to force me limiting my Google Search debugging to 20min.  After this time I get my hands on into debugging the problem.

Hurdles of using a SW debugger for Android applications and the Android platform

Let’s start debugging the problem. Unfortunately, it is not just launching the software in a debugger and step through the code as we are used to for desktop applications. There are quite a number of hurdles to overcome before we can actually debug. When porting to new hardware, the root cause can be everywhere. Even proven applications, stable Android runtime modules or Linux drivers cannot be excluded.  I have often wasted a lot of time debugging when I have refrained myself from investigating SW modules that are said to be stable. Of course, the stability of a module has to be considered, but one should not hesitate to question and debug those as well. Just recently I have found a defect in the Android Apps installer, which has been triggered by a very specific kernel version and filesystem type (http://tinyurl.com/3b3szbj, could only reply my fix directly to the author). Debugging is not exactly easy with Android. Not because Android is so complicated, but just from a pure practical perspective it can be very hard to setup  a debugger.  For application developers, the Android SDK is providing very convenient debugging capabilities for the Java layer. The companion NDK (native development kit) allows developers to package native code libraries with their applications. But, only one year after its first release in June 2009 it actually provided support for native code debugging.  Beyond the NDK and SDK, when the whole Android platform is subject of debug, then things get much more complicated and solutions are sparse. A very useful guideline for debugging I have found in the web, it still requires 12 steps to set up the debug session and another 9 steps to connect the debugger.

Why is setting a debug session more difficult with Android?

What applies to Android may as well apply to many other integrated SW stacks. Each high level device function is delivered by the interaction of many distributed software entities, some of them may be written in Java, others in native code. Most of them executing in the user space, but a fair amount of functions of course also reside in the OS and its drivers.

Another challenge is imposed by the fact that processes and threads are launched and terminated by other processes automatically. First of all you do not know which processes are contributing to a high level device function, and even if you know, the process is often terminated already before you had a chance to connect a debugger. In order to solve that,  the best practice is to add an busy wait loop into the code and recompile the respective module.  This will give you the chance to connect the debugger at the right point in time and before the bug has eventually passed. The same technique is required when debugging on the Java/C boundary as there is no way to auto-step from Java code down to native code within the debugger. Once you can connect a debugger, it all works pretty well as long as you stay in one software module. But, it is a very tedious process as it requires to modify and rebuild software module, bundle new filesystem images, reproduce complex scenarios each time you want to debug another piece in the puzzle.

If you do not know which SW function to debug, you may add those halts to many places with dangerous side effects. Halted processes may cause time outs or simply impact the sequence of events in the system masking the bug. Many more challenges exist, but I want to stop here as I would like to switch over to talk about some solutions to these problems.

What’s next?

In my next post, I would like to introduce how Virtual Prototypes (VP) can help to overcome some of the debug challenges appearing with SW stacks such as Android. As a simple example, the controllability of a VP will help us to intercept the execution of a single SW process. Moreover, I will show you how to inject a halt without modifying the software using a VP. But beyond fixing deficiencies in debug flows, VPs enable much more if used in the right way. A VP can act as a debug server which significantly eases the debug setup and turns you much quicker from an experimenting/guessing mode into systematically analyzing the problem.

I am involved in discussions about adoption of system-level technologies a lot. System-level design in EDA and embedded software are always intertwined as the software is the main factor changing when going beyond RTL. Given that system-level design technologies expand beyond the traditional realm of hardware, their adoption is non-trivial for project teams. The overall situation reminds me more and more of Malcolm Gladwell’s “Outliers”: for success several factors have to fall in place together, not all of them in our control.

In “Outliers”, Malcolm Gladwell describes “The Story of Success” as it relates to people. He argues very convincingly how several things need to fall in place. Amongst them is being born at the right time of year to be more mature and therefore be chosen for special training in sports. Also, the need being there at exactly the right time for a technology like computers to be needed. In addition the right background is crucial to have the ability to develop a skill – parents supporting training, education capabilities or available of excess compute time on a mainframe.

The same is true in principle for technology adoption, and most certainly for system-level design technologies. The time for adoption needs to be right. Being too early means you simply may not have enough breathing room to get to the mainstream., which almost certainly means that you will fall into the chasm instead of crossing it. Having the right technology combined with the right environment for it to be adopted, is crucial. The actual user need has to be there too. If the users can get by with traditional techniques, then the reasons to change methodologies, i.e. to drop old habits, are not good enough. Finally, when the time is right, the technology exists, the need is there, users need to know about the solution too. That’s where the power of marketing comes in.

So where are we with system-level technologies? Let’s look back 15 years. As some of you know, I cam to the US in 1997 to run the technical marketing for the Felix Initiative at Cadence. The resulting product VCC has been described  – together with Synopsys’ Behavioral Compiler – as one of the two system-level trailblazer projects in “ESL Design and Verification: A Prescription for Electronic System Level Methodology (Systems on Silicon)” (Martin, Bailey, Piziali). Were we too early? Probably. Leading customer told us though that they would need function-architecture co-design as implemented in VCC at the time. They documented with their tool purchases that they had a need. Was the technology right? Well, it worked for the early adopter designs and most of its concepts – like transaction-based design – find itself now in technologies like SystemC TLM 2.0. Were user getting by with traditional techniques? Absolutely! It was getting harder, but why abandon a methodology that’s still works? Did users know about it? Well, we marketed it quite a bit, and knowledge about system-level design approaches was growing.

So what was missing? The technical barrier to adopt the technology was simply too high. In the case of VCC we were addressing hardware and software designers with a combined methodology. We had developed techniques to abstract hardware (processors, busses, peripherals etc.), to abstract software (RTOSs, drivers and the actual functionality) and we could even create the implementations from the complete executable system-level specification. Heck, at DAC in 2001 we were demoing a flow in which the customer told us during the demo whether a task should run in hardware or software and then we automatically re-built the design and did show the resulting layout of the hardware together with the implemented software image.

The results were awesome: We achieved much faster development times and more robust designs meeting their specifications. Too bad that software and hardware developers had to change the way they are doing things completely. Also the complete ecosystem of Processor, IP, RTOS and middleware providers had to provide models to make it all work.

In the spirit of fellow Blogger Steve Leibson’s law, that “it takes 10 years for any disruptive technology to become pervasive in the design community”, I am tempted to formulate my own “Schirrmeister’s Law” as “The ability to adopt a new design technology is inversely proportional to the number of changes it requires”. Sounds trivial. Still something I learned the hard way during the VCC days and something we often overlook today.

If I could just quantify and measure it. That is hard, but here is an example: Almost 15 years later Virtual Prototyping is going through its adoption. How many changes does it require? Well, compared to VCC it is much more adoptable given that the software developers do not need to abstract anything. The real software – namely .elf files – run on abstracted hardware. The number of changes – or additional steps – is limited to the hardware developer.

So what about the other adoption criteria? Is the time right? Well, software development has probably never gotten as much attention from the EDA side as it has these days. And our customers are expressing their concerns in every meeting I am at.. Do we have the right technology combined with the right environment for it to be adopted? The successes seem to speak for themselves – we have real users achieving real results. Is the user need there, do users get by with traditional techniques? We recently did a survey of about 800 embedded software developers. One of the questions we asked was “What target execution environment are you using for your embedded software development?”. The results are shown on the left. The top four answers are pointing to hardware based techniques, so they seem to work. Probing for limitations on those techniques users confirmed issues which I have written about in this Blog before – like the ability to control and debug the execution. Finally, whether software users are aware of virtual prototyping was a question intended to check whether target users know about the technology. There is definitely room for marketing effort here as the awareness was lower than we had hoped.

Bottom line, the adoption of system-level technologies like Virtual Prototyping remains a hot topic. Certainly the industry has lowered the barriers to adoption, but there is still some distance to go. The number of changes we ask users to adopt technologies is often still significant enough to shy away from adoption, even if it means mortgaging the future …

I will think about how to measure my postulated “Schirrmeister’s Law” above with quantifiable data. As always, I welcome your thoughts.