Synopsys Cloud
Cloud native EDA tools & pre-optimized hardware platforms
Request a Free Trial →
Table of Contents
Cameras are pervasive in everyday life, from medical diagnostics to automotive, smartphone, and video surveillance. Consequently, half of today’s data are images and videos, most of them being generated at the edge with portable systems. Compared to the cloud, the edge brings many challenges: compactness, low power consumption, and latency. How do you get a great image out of a smartphone without emptying the battery in 2 hours? How can an autonomous driving system analyze images from almost ten cameras simultaneously without requiring massive cloud data treatment capacity? Enabling the next generation of edge and pervasive cameras will need a renewed toolbox.
In a previous blog post, I reviewed how digital twins are a key enabling technology, not only for airplanes or complex power plants, but also for imaging systems. Digital twins will foster an optimized product lifecycle through simulations, enabling better specifications (avoiding over-specifications), process control anticipation, and earlier software development. Digital twins will also foster customized imaging systems, such as efficient video surveillance cameras that are not always on.
Pain points cited by teams developing cameras include changing specifications, discrepancies between design and manufacturing, and complexities related to the design and assembly of multiple components. In this blog post, I’ll review how digital twins, through image simulation, can enable better, more streamlined specifications and virtual testing of imaging systems.
A Camera Is a Complex System
An imaging system is a multiscale, multiphysics model with multiple interdependent components. A lens set captures photons from a scene; a CMOS image sensor converts these photons into electrons; an image signal processor chip converts the raw electronic signal into a readable, colored, and denoised digital image; a digital signal processor computes the digital image to get information; and an I/O chip interfaces the imaging system with some external component.
In the end, an imaging system must provide a good image. But what do we mean by that? Most of the time, a good image still refers to photographic-quality images taken with a DSLR. However, if this standard is applied to images for soil monitoring, medical diagnostics, or autonomous driving, it could lead to over- or mis-specifications of the desired results.
Let’s see how imaging system architects and teams can use the image simulation features of the Synopsys Optical Solutions portfolio to render and optimize the desired image from an imaging system.
Workflow and Results
As described in the following simplified example, simulating an image using CODE V, LightTools, and RSoft Photonic Device Tools is already possible. The example features an imaging system embedding a four-lens set and a CMOS sensor with 1.12 micron RGB pixels and a microlens array.
Example imaging system with a four-lens set and CMOS sensor
1. Simulate the image from the lens set optical properties using CODE V
The first step is to design the lens set and generate the corresponding PSF. We use the Image Simulation (IMS) feature in CODE V to simulate an image. Any 2D image can be imported into the tool. You then define the grid of PSFs to be computed over the object and the FFT grid size. You can add color, relative illumination parameters, and image sensor properties such as pixel size and the number of pixels. IMS will compute the convolution of the input image with the optical parameters defined and calculated by CODE V and will output the resulting image.
The output image on the right is the result of the convolution of the input image with the optical parameters of the lens set as computed by CODE V
Image quality can then be assessed on the simulated image. Further optimization of the lens set can provide the desired image quality for the targeted application.
2. Simulate parasitic light with LightTools and RSoft Photonic Device Tools
Most imaging systems must be optimized further to avoid parasitic light and compromise the desired image quality. Ghost and flare are typical parasitic effects of imaging systems. You can use RSoft Photonic Device Tools and LightTools to predict ghost and flare in our simplified example of an imaging system. We want to predict parasitic light in the system because of sensor diffraction and reflection properties.A CMOS image sensor is usually a collection of micrometer size pixels that create a strong diffraction pattern. We must use RSoft Photonic Device Tools to get the optical properties of this surface. More specifically, DiffractMOD RCWA is an efficient tool for rigorously calculating transversely periodic device diffraction properties. It will output the reflection/transmission power for each diffraction order, the total reflection/transmission, the amplitude/phase/angle for each diffraction order, and the field distribution in the simulation domain. All results are stored in a BSDF file that can be exported to LightTools for further simulations. The third step is to import the lens set geometry from CODE V, add the sensor properties, import the BSDF files from RSoft, add a source (sun source in the following image) and use the ray path tool from LightTools to run Monte Carlo simulations to analyze all possible paths for parasitic light. The simulated result will resemble the image shown on the right.
On the left: the LightTools camera model; on the right, the map of the parasitic light as seen on the sensor
3. Combine the simulated image with the parasitic light map using COM API
The final step is use COM API to combine the simulated image from CODE V with the parasitic light computed with LightTools and RSoft. Gamma and energy ratios can be adjusted in this convolution.
Convolution of the simulated image from CODE V with the parasitic light map from LightTools; the simulated image is displayed on the right
Assessing the image with image quality metrics on simulated images enables further optimization of the system to achieve specifications suited to the targeted application. Virtual testing of simulated images through CODE V, LightTools, and RSoft Photonic Device Tools is a powerful workflow that avoids over-specifying an imaging system and saves time by anticipating the assembly and testing phase through simulations.
Continue Reading
The Dream Job and Beyond: Daren Reid Reflects on a Career in Software Development
Apr 01, 2024 / 4 min read
Tags:
Optical,
Inside Synopsys