The other day I listened in while my friend Coverage told the neighborhood kids the Thanksgiving story.  It went something like this. Once upon a time the verification engineers in the Old World design houses were not free to practice verification the way they wanted. Instead, they had to follow a strict directed test methodology imposed to them by long standing traditions. One day a group of them gathered a few of their test benches, boarded a chartered bus they called “Randflower” and left to join a new company where they could practice their own verification methodology rooted in the principles of freedom.  From then on verification engineers would be free to write coverage targets based on verification plans written in a language that everybody could understand and even testbench variables would be free to take whatever values they wanted as long as they were following some constraints. However, the road to success was not to be an easy one. They had to work long hours, come up with new methodologies and tools to take advantage of them.  The long nights and weekends of hard work soon started taking their toll. Many of them quit and moved to the software industry and those who stayed turned weak with frustration and exhaustion. But then one day a miracle happened. The indigenous designers in the company who had been working there for many years gathered together and shared their knowledge about the design with the verification engineers! This gave them the strength required to close on the remaining coverage targets and finally the fruits of their hard work were ready to be harvested. The verification engineers had a huge Tape-Out Feast where all the designers were invited to jointly celebrate the Turnkey Verification Environment they developed and thank their design counterparts for their invaluable help in their hour of need. They vowed that from then on they will write testbenches that can be easily shared among many verification groups and that they will share methodologies and best-practices for the benefit of the entire verification community.

Over the years the story morphed slightly, turnkey turned into turkey and the sharing of testbench components evolved into a more generalized spirit of sharing. However, to this day people gather around the dinner table on Thanksgiving to celebrate the bravery of those verification pioneers who stood up for their own verification beliefs and the kindness of the designers who in spite of being suspicious at first, helped them out to meet the deadline and avoid disaster.

From my friend and I a warm Happy Thanksgiving!

Halloween and Elections just zoomed by so naturally they were the focal point of the conversation I had the other day with my friend Coverage. First, I asked him whether he applies his expertise to teach kids how to strategize the candy collection process. I figured trick-or-treating has enough similarities with hunting for bugs: the candies are the bugs, the kids are the verification engineers (pardon the analogy) and the kids want to collect as many candies as possible. Since not everybody in the neighborhood may be inclined to treat, the kids need to come up with some good coverage points to make sure they hit the houses that are most generous with their candy. Pretty straightforward I thought however, the frown on my friend’s face was a clear indication that I didn’t exactly hit the nail on the head. “Verification”, he said “is a marathon, not a sprint, it’s not about going out one night and hitting the bug jackpot”. He continued to explain that it is more like Halloween lasted for many months and although the goal was still to collect as many candies as you can before the time is up, scaling the process down to a couple of hours won’t paint the correct picture nor will it teach us what paint brushes to use. Ideally, he reasoned, one would design and apply coverage based candy collection in such a way that it not only maximizes the total quantity of candies but also provides a relatively constant stream of candies over the life of the collection process. Bug-overload and sugar-overload are nasty beasts to tame so there’s no need to rush the coverage driven attack plan. When to start looking at coverage is something that is sadly often overlooked but important nevertheless in the quest for an efficient verification strategy. After all, knocking on a few neighbor’s doors will provide enough candies to get your kids started and similarly,  finding the first bug load does not require collecting coverage numbers.

OK, let’s move on to Election Day, and no, my friend and I did not discuss Election Coverage, we talked about a different kind of coverage. Ever wondered why before elections when you open the TV, they show most of the candidates in diners? It is because diners are excellent coverage points for voters. Candidates want to collect as many votes as possible before tape-out, sorry election, and they want to hit the places where they maximize their chances for finding the voters that could be swayed to change their vote. Sorry folks but we’re going to pretend the voters are bugs, just for allegorical purposes, this blog is definitely not trying to make a political statement . The kind of bugs you’re looking for varies, for instance if you’re a Republican candidate, you want to find bugs that are on the fence, Independent bugs, mild Democratic bugs, etc. but the point is that if you want to efficiently scour for votes, you need to set up your cover targets at the key crossroads of the various neighborhoods in your district. Similarly, defining your coverage points on the key intersections of simulation paths and hitting them will have a good potential of bringing out some bugs from the shadows. This may sound easy but alas, diners are not the only cover points you need to worry about, there are ads, town hall meetings, mail-ins, etc. sadly, the cost of elections just as the cost of verification is only going up!  Voters and bugs get more sophisticated by the day, their number is on the rise so there are many voters to convince and many bugs to catch. My advice, get my friend involved in your campaign and come Election Day, you too could be a winner.

Metric Driven Verification, Coverage Driven Verification, in one form or another everybody is talking about the same thing so I thought about asking my friend how he likes to be in the driver seat. With a sheepish grin he replied that it depends whether he gets to drive a Trabant or a BMW. Both German cars mind you, but… So that got me thinking. Lately, we all have been talking up methodologies, verification plans, intelligent testbenches, guiding metrics, assertion densities, etc., etc. Now these are all great topics and we definitely need to keep nurturing them however, one rather important detail seems to have slipped out of the conversation and that is the good ole’ workhorse of every verification flow, the simulator. For no matter how careful we design our testing strategies, how cleverly we place our coverage targets and how many assertions we sprinkle around the design, we still rely on the simulator to get us to our destination. Whether we define our destination as squashing every bug in sight or meeting ever shrinking delivery timelines, it will be the reliability and performance of the simulator that will save the day. From his great driver vantage point, my friend confessed that while having a GPS to tell him where he is, an intelligent radio to alert him on traffic jams, and an on-board computer to report instant gas mileage are all great assets, once he gets on the Autobahn and the pedal hits the metal, there is nothing like driving the fastest car on the road. So much for car advice.

As I enviously looked at my friend speeding away in his 3 letter simulation-mobile I was once again reminded that in the end without unrelenting technology improvements and meticulous engineering of the engine, whatever we build on top of it will make little difference. No matter how skilled a driver he may be, it will take my friend much longer to reach his destination.

When my friend first muttered these words, I thought he decided to leave the rewarding world of verification and become a political pundit. I was dismayed because I don’t like the term pundit, it rhymes with bandit. Ever wondered why we have verification gurus and they have political pundits? In any case, it turned out it was about a far less controversial kind of stimulus package, the one you feed to your design to tickle its inputs and make its transistors go crazy. Phew!

However, once my mind wandered towards the halls of power, I discovered, strangely enough, that there are a lot of similarities between the government stimulus and the one that’s the bread and butter of verification engineers. To begin with, both of them are a big deal. If you live in the US and turn on the radio, chances are somebody is talking about how massive the economic stimulus is. Well, if you want to verify anything these days, you’re probably going to throw massive stimuli at it. To manage their stimulus package, the government has set up a large bureaucracy; you’re going to set up a large testbench for yours. Your testbench will take a lot of time and effort to develop and debug and once you’re done with it, you’ll want to reuse it over and over again. Similarly, it takes a while to get large bureaucracies off the ground but once they’re in business they tend to find a way to reuse themselves over and over again. Next, there’s the question of when you should stop stimulating and that’s when things get really interesting. See, the government hasn’t yet figured out how to design uncontroversial cover targets and measure how well their stimulus hits them and that’s why the benefits of the stimulus package continue to be widely debated. We did, my friend can vouch for that.

I could keep going but this is not a town hall meeting. Let me just say that in verification, we like our stimulus to have both an evenly distributed effect on the design to make sure we sensitize as many paths as possible, as well as a more focused effect on the areas that implement the main functionality of the DUT. As a result, there is a place for both randomly generated stimuli as well as more targeted ones. Although it may be a stretch to call the economic stimulus random (no, let’s not go there), its stated intention is to spread it in a reasonable manner to stimulate as many economic paths as possible. There is also the equivalent of a directed stimulus and is commonly referred to as pork-barrel but the main topic of this blog is to talk about the kind of stimulus that you want to spread around. And that is where stimulus coverage enters the scene. Often maligned, derided or simply ignored, stimulus coverage is the ugly duckling of the family, far less glamorous than its brothers who capture complex arbitration scenarios or exotic packet combinations. Today, my friend asked me to make a case for bringing stimulus coverage out of the shadows and into the limelight of our coverage strategy.

I will start by first pointing out that we need to make sure that our stimulus is legal (I’m not going to go into what that means for the government one), so we spend a great deal of time developing constraints to ensure that. These constraints become very complex very quickly partly because the environments in which the verified blocks operate are getting more and more complex and partly because we strive to capture only realistic scenarios on the inputs of our blocks because we don’t want to send the block into the twilight zone and we don’t want to waste valuable simulation cycles. So now that we’ve spent all this time developing these complex constraints, how do we know that the outcome is the kind of stimulus we were shooting for? Try functional cover groups that capture what we think is reasonable stimulus, aka stimulus coverage. You may object that this is double work because it boils down to expressing the same stimulus intent once implicitly through the constraint set and once explicitly through the stimulus coverage. I think it is well worth it. After all, the whole existential purpose of the testbench is to sensitize potential bugs and to make sure you observe them. Stimulus is the one that takes care of the first part so whatever we can do to make it more robust will pay back dividends many times over. I wish the same will be the case with the economic stimulus…
The second thing to keep in mind is that the constraint solvers are not perfect. There is no guarantee that they will always provide a solution if one exists and moreover, the solution may not be exactly what you expect. When a range of solutions can be generated, how do you guarantee that the solver distributes the values the way you imagined it or that it touches the “interesting” values in that range? Again, stimulus coverage can help. Once you explicitly documented what the key values or ranges are through the stimulus coverage targets, you need to make sure you cover them all. For that, you can manually play around with distributions, add constraints, etc. Not rocket science by any means but potentially very messy. Very well suited for an automated tool as my friends keeps pointing out

To conclude, stimulus is a crucial part of verification (and the economic recovery some say) and any effort we spend in baking it better will be well worth it. Stimulus coverage is a straightforward way to address that. Obviously it will not guarantee that you will sensitize every corner case bug in the design but it will at least make sure you have a reasonable starting point. Go back to the government’s stimulus. Ideally, it would help every business in the country in some way. Yes, we can? Neh, sorry Mr. President that’s too much to ask but one step in the right direction is for instance to send some money to every state. If you don’t send any money to say California, it’s close to certain you’re not going to help too many businesses there so the first thing to do is to make sure every state is covered. Kind of like our stimulus coverage. Clearly, there is much our governments can learn from verification engineers so maybe one day our gurus will become their pundits.

I hope I didn’t let my friend down and was persuasive enough to send you writing stimulus coverage right away and if the Echo of my blog will ever so feebly resonate in your verification approach, my friendship will be stronger than ever.

You may remember the suggestive metaphor of a butterfly flapping its wings in California and causing a tornado on the other side of the globe. No, I am not attempting to apply chaos theory to coverage (although… ), I am merely picking up where I left off last time and make a case that the various parts that collectively define the coverage problem are tightly connected and that the decisions we take for each part will inevitably have a profound effect on the others. Let’s take a look at some of these connections and the potential pitfalls if we ignore them.

The connection between the coverage model and the checkers is pretty obvious, what good is perfect coverage if you haven’t checked anything? And thoroughly checking expected outcomes without generating sufficient scenarios is like getting hooked up to the most sophisticated EKG machine while sleeping and concluding that your heart works perfectly. Either way, it just doesn’t work; my buddy Coverage and his pal Checker always work as a team.

Next, there’s coverage model and coverage results interpretation. Sure, you can easily write crosses on any signals you want but are you prepared to handle the deluge of data that’s going to come your way? When you’re looking for the needle in the haystack, adding more hay is in general not the shrewdest move.

Now let’s look at my favorite, the more obscure connection between coverage model and coverage convergence. Can you develop your coverage model so that it is easier or faster to converge on it without of course compromising the quality of the model? You sure can and you sure should. Let me give you an example. Say you’re in Phoenix, Arizona and you plan to visit the Grand Canyon. You can do that by either visiting the North or South Rim. Both will achieve the same goal of seeing the Grand Canyon but if you pick as your coverage target the North Rim your journey will take longer and will be more difficult.  Oh, and if you think it’s a stretch to use the Grand Canyon as a stand-in for a bug, there are some semiconductor companies out there whose functional bugs ended up in shipped products and I’m sure for them even the Grand Canyon seems tiny.
What I’m getting at is that everything else being equal, the fewer levels of logic between the signals you choose to sample in your coverage model and the signals you randomize, the better chance of convergence you have. As a corollary, the sooner you sample the signals, the sooner you will know whether you hit your coverage targets and therefore the faster you will converge. Sure, there will be some coverage targets that lie deep in your design and can only be sampled towards the end of simulation but let’s KISS whenever we can!

The connection between interpreting coverage results and coverage convergence seems to be pretty straightforward. You look at what targets have not been hit yet and if you can properly interpret what that actually means, you can decide if they need to be excluded or more effort needs to go into covering them. The trick though is the interpretation part. Think of the map from Phoenix to the Grand Canyon. You may know exactly where you are on the map but if you’re zoomed in too much, you won’t be able to tell how long you still have to drive or even what roads you will have to take next. Similarly, simply looking at the coverage results may provide you with precise data but only limited information. Knowing exactly how many times a coverage target was hit rarely translates into understanding exactly which feature has been verified. You will need to bring the coverage results to a higher level of abstraction if you want to use them as a map towards coverage convergence.

I deliberately left extracting the project risk factor from coverage results at the end. Although it is arguably the most challenging piece, more art than science as a wise friend of mine keeps reminding me, it should be crystal clear that it is strongly connected on all the other pieces of the coverage puzzle. The piece of mind that will hopefully engulf you once you push the tape-out button will depend on what kind of coverage model you designed, whether you properly matched coverage and checking, how well you interpreted the coverage results, and how far you pushed for coverage convergence. If you don’t believe that, there is a good chance that a butterfly flapping its wings in California will cause a chip failure somewhere in the world.

I once had lunch with the CEO of a start-up and asked him what skills he thought I needed to acquire if I wanted to run my own company. He quickly replied I needed to understand Marketing. Now if you come from the Engineering side of our world as I do, your reaction is probably similar to mine at the time: “What?” I thought, “is getting ready for DAC really that important??”. In case this is indeed what you’re thinking, here are 2 must-reads: “Marketing High Technology” by William Davidow and “Crossing the Chasm” by Geoffrey Moore. It turns out that there is a lot more to Marketing than meets the ignorant eye. Among the many interesting concepts, the one that stuck with me is called in one way or another, “The Whole Product”. Its beauty lies, as it often does, in its simplicity, and goes something like this: real success comes from delivering to the market a Whole Product and not just a piece of technology. It is pretty common sense, you don’t just put out a chip or a piece of software and hope somebody will figure out what to do with them, you either build (or make sure somebody else does) complete systems around the chip or integrate your software within existing flows, you strike alliances with other software and/or hardware providers, you build a support infrastructure, delivery channels, prepare documents, trainings, etc. etc. You get the idea and if you don’t, try typing iPhone in your search bar.

So what does all this marketing insight have to do with my friend, you may wonder. Well first, I’d like to convince you that he is not just some superficial dude and that there’s a lot more complexity to his character.  Understanding this complexity and its implications will be the key to your friendship and the first step is to realize that you cannot only look at the coverage hole problem, you need to tackle the whole coverage problem. To understand what makes the whole, let’s try to identify its parts.

First, there’s obviously the coverage model. We really want to cover every single path through the design but if we advocate that we risk referral for psychiatric treatment so we need to find some meaningful proxies instead. Various forms of code coverage along with thoughtfully considered functional coverage tailored to one’s specific requirements will have to do for now.

Second, there’s checking the correctness of the implementation. After all the main purpose of verification is to make sure that the implementation matches the specification so whether you use assertions, scoreboards, whatever, there must be a way to infer a pass/fail outcome. Many would argue that this has nothing to do with coverage but to me that’s like touring the country with a blindfold, what good is it that you’ve visited all the landmarks?

Third, there’s interpretation of the coverage results. This is about how to extract useful information out of an ocean of data and presenting it in a form conducive to analysis. Different ways of filtering, projecting, highlighting and customizing the coverage reports are a good start.

Fourth, there’s folding the insight you gained from the coverage results into the overall risk equation of your project (check out my previous blog). This will require you to figure out which uncovered parts of the design constitute a manageable risk and which need to be addressed. May the Force be with you!

Finally, there’s coverage convergence. Random constrained simulation will get you 80% of the way but chances are you won’t hit all your coverage targets and you’re going to have to do some heavy lifting for the ones you didn’t. Understanding how the random variables control the signals sampled in your functional coverage or the branch conditions in your code is of course the key but that may require a brain fusion procedure among various verification and design engineers.

So far I haven’t probably told you something you didn’t already know in some form or the other but what I hope to convince you is that all these elements are tightly connected and that you will have to carefully consider their dependencies in order to solve the whole coverage problem. In the next blog, I’ll take a crack at analyzing these connections and how they affect your coverage planning. For now, keep in mind that just as a whole product strategy is key to marketing success, a whole coverage strategy is key to verification success!

Guess I should start with a disclaimer: I don’t know much about risk management. If I did, I would start an investment outfit call it “Golden Socks”.  “Golden Socks” would capitalize on the universal dream of software ownership, help fuel a software bubble and get you to buy software you cannot afford.  At the same time “Golden Socks” would bet that you won’t be able to pay maintenance for your software and have to give it back. Or something like that.

My friend Coverage on the other hand, he is an expert. Many people whose responsibility is to decide when enough verification is, well, enough verification, call upon his expertise to help them out. You see, some people would have you believe that verification is all about finding as many bugs as possible. And of course they are right but with one rather important refinement. It’s about finding as many bugs as possible before the time is up. And therein lies the problem. “To tape out or not to tape out: that is the question”, has been driving princes and managers mad ever since Hamlet moved into verification.

If your business is to provide IP for a customer and you’re late, you may find that somebody else is eating at the customer’s table. If you’re in time but there’s smoke coming out of the smoke test you may find yourself in need of a smoke (wait, talk to the Surgeon General first). If you’re in the systems business and you control the IPs, the SoC, the board, the firmware, the apps, etc. it is the market window that controls your destiny and you can ill afford to miss it. If you make it but end up having to recall your chips because a functional bug slipped through, uhh, head hunters may become your new best friends.

Everybody knows there are still bugs, nobody knows how many, is there anybody who can help? Yup, you got it, it’s my friend Coverage. Now mind you, he’s not going to be able to tell you whether you’re ready to send your chip to the backend folks but he’s going to give you one quantifiable dimension into the complex risk management decision you’re facing. If IP is your cup of tea and you properly developed your coverage model and correctly mapped coverage targets to features, the kind of coverage targets you’re hitting should tell you whether the main functionality of your IP is taken care of. You send it to the customer and by the time they get to shine the light in one of its un-covered corners, you’ll have another rev ready with a hopefully decimated bug population in that corner. You made the deadline, the customer is happy. If systems is the name of your game, chances are you won’t get too many shots at getting it right. You need a much more comprehensive coverage model tailored for the block and system level verification and need to prioritize your coverage with a lot more care.  If your final coverage results looks like Swiss cheese but it mainly comes from IPs you’ve used to tape out before or otherwise trust, you may turn a blind eye and let it slide. However, if the smell of Swiss cheese comes from your own carefully crafted targets built around fresh code it may be worthwhile biting the bullet and delay tape out cause something may be rotten in Denmark. Bottom line, the risk equation will be different for different applications but coverage will still be one of the main factors you should take into account.

There’s obviously no free pass and there are various other factors to consider but if you thought your coverage model through and if you’ve interpreted your coverage data correctly, you have a powerful tool to gauge your risk. My friend will become yours too.

Coverage and I got acquainted fairly late in my life. This has nothing to do with me getting old and my dear wife outfitting me with Abercrombie gear to cover my age, it has to do with me joining a startup called Nusym back in 2008. That’s when Coverage and I became friends. Nusym pioneered the technology that lies at the root of what the good marketing folks call “Coverage Convergence Technology”, “Intelligent Testbench” and some other tempting names that make even the most cold blooded verification engineer purr with anticipation. Nusym’s exciting technology has in the meantime become part of the Synopsys family and I have followed it partly out of loyalty to my friend Coverage.

I have been bothered for a while now by the way people generally refer to my friend Coverage and today is the day I will gather my courage and take a stand. You see, Coverage is often called just a metric to measure things like verification quality or project completness. First of all just a metric is demeaning for all metrics in the world. Metrics are important, they allow us to quantify the various activities we engage in and they give substance to the otherwise elusive notion of progress. In a recent blog, Janick referred to constrained random verification as a journey. Now think of your verification manager as the kids in the back of the car on this journey, the manager keeps asking “are we there yet, are we there yet?” and all you can do is point to the latest coverage numbers and say “be patient, we’re getting close”. So yes, my friend Coverage is also a metric but not just a metric, he is a very useful metric, a good friend in time of need and aggravated management eager to tape out definitely qualifies as time of need.

However, my friend Coverage is a lot more than a metric. Think again of the journey, only this time picture yourself as a road inspector whose job is to make sure all roads are in good shape. Sure, you can aimlessly drive on whatever road you fancy but you see, some roads are more important than others. Take for instance the roads around popular landmarks, amusement parks, shopping malls, etc. A lot of people travel on them so they better be safe. Functional coverage targets are the landmarks that need to be visited, they give random constrained verification direction and prioritize the paths that need to be explored. Today, the steering towards the landmarks that have been missed is a job better suited for voodoo practitioners. The car has no GPS oh and heck, there is even no driver at the wheel! The turns have to be decided in advance with only a very imprecise map available. Tomorrow however, some clever technology (a bell should be ringing at this point) may help make the map more precise and even better, put you, the verification engineer, behind the wheel.
Today my friend Coverage just sits on those landmarks and shouts “come here, come here” and that’s already plenty useful, but tomorrow he may be able to throw some crumbs along the way a la Hansel and Gretel or talk you through the turns you can take to get to him. Your journey will then be a lot more efficient and relaxing.

But for that, please don’t call him just a metric anymore.