Today’s digital cameras match or slightly exceed the performance of silver halide film. Computer graphics has achieved the goal of photorealism. Now the goal is to go beyond simply matching paper and silver halide – to create display technologies which can present any visual stimuli our eyes are capable of seeing.
One area of rapid development is in dynamic range. A new crop of technologies using High Dynamic Range imaging (HDR or HDRI) aim to extend the dynamic range of digital imaging technologies way beyond traditional media.
In this article, I’ll look at recent advances in the field of high dynamic range imaging. I will cover the basic concepts of dynamic range, and talk about new HDR technologies.
The Problem
Here’s the problem in a nutshell:
* Real-world scenes contain light ranges that exceed a 50,000:1 dynamic range.
* For over a thousand years, media has been limited to around a 300:1 dynamic range.
* So you have a mapping issue: how do you represent light values in a scene using a much more limited set of light values for a particular media?
Old Solutions
Master painters were very clever about mapping scene intensities to canvas. They used a large number of tricks
Dynamic Range in Photography
Photography involves a capture device (the camera), a storage medium (e.g. film), and a display or output device (e.g. paper).
The dynamic range of each stage (capture, storage and output) plays a crucial role in the quality of the results. In general, technologies with greater dynamic range produce more realistic results. But photography is a compound process, and the dynamic range of each stage must be considered. When the dynamic range of the source scene is too great for any one stage of the process, something must be sacrified: you must either give up detail in the shadows or the highlights. Photographers have to know and work withing the limitations of their camera, storage and output devices.
The Next Horizon: Digital HDR
Over the next decade, the imaging industry will inevitably transition to high dynamic range (HDR) imaging, creating devices that provide a latitude range far greater than traditional silver halide film. This change will affect all aspects of image making. Each of the systems in the image workflow will be modified, including capture, storage, editing and output.
Capture
Today’s cameras have ample resolution. So the next area of product differentiation for camera manufacturers will be the quality of the pixels, rather than the number of pixels. This shift is starting to happen already.
For example, Fuji’s SuperCCD S3 Pro camera has a chip with high and low sensitivity sensors per pixel location to increase dynamic range (the same trick as used in color film to achieve broader latitude).
Although the resulting chip has lower overall resolution, it captures greater dynamic range. This tradeoff of resolution for dynamic range is the beginning of an important trend.
A second alternative is to merge multiple images to increase dynamic range. Paul Debevec at SIGGRAPH 97 showed how to take multiple photographs at different exposures and merge them together to create a single high-dynamic range image. This technique is now incorporated in products such as Photogenics and Photoshop. For now, the technique works best if the camera is mounted on a tripod. But researchers have already built HDR image “stitchers” which merge multiple images and automatically account for camera motion between snaps.
For most consumers, “HDR” will simply mean that the camera records more details in shadows and in highlights. Just as RAW images extended the detail held in digital images, HDR will further increase the available tonal range.
Consumers will benefit from the true point-and-shoot ability that broader latitude offers, because HDR cameras will produce usable images from a much wider range of lighting situations.
Eventually point-and-shoot cameras will lose their built-in flash. Anyone who uses a camera with a cheap flash soon learns that the pictures generally look better if you turn the flash off. Sensors are becoming more sensitive, cameras are getting smaller, and light metering is getting smarter. Add three or four stops of dynamic range, and the flash becomes a creative ad-on rather than a requirement.
Professional photographers will also benefit from HDR. With HDR technologies, photographers can really push the creative envelope, exploring the extremes of high-key and low-key effects.
Professional cameras will offer a multitude of HDR image-taking modes. For example, they will automatically blend multiple images taken with different exposures, with and without flash, possibly using multiple light sources, to produce a single and extremely maliable master image.
Storage
All image file formats have range limitations. Formats such as JPEG and GIF provide eight bits per color channel (often referred to as 24 bit color). Using 8 bits, you can represent 256 different intensities per channel. Most 8 bit formats use a “perceptual” mapping, meaning that they use a gamma (exponential) curve rather than a linear map of intensities. See Human vision and tonal levels or the venerable Gamma FAQ for an explanation of this. While JPEG and GIF are great for moderate dynamic range images, banding becomes apparent if you edit the image extensively, and the formats cannot store high-dynamic-range scenes.
Newer formats including JPEG2000, RAW and PNG offer up to 16 bits per color channel, which is plenty for most purposes. However, there is no support for “underage” or “overage” – these image formats state that “0″ should be mapped to the darkest black of the display, and “65536″ (or the equivalent) should be mapped to the whitest white. If you want to represent images that contain brightnesses beyond what your monitor can currently display (e.g. as produced by HDRShop), you need to look elsewhere.
The most exciting HDR image format today is OpenEXR, developed by Industrial Light & Magic.
I say this partly because their documentation includes photos from Star Wars (see above). But it also supports both 16-bit and 32-bit float representations, lossy and lossless codecs, and has a great definition for underage and overage.
Other examples of High Dynamic Range formats include SGI’s TIFF LogLuv format, Floating Point TIFF format, Radiance’s RGBE format, and Portable Float Maps (PFMs).
HDR image formats are especially significant for archival and stock uses, since they store data with enough precision to record what we can see, rather than what our displays can show.
There are a range of proprietary formats that offer medium or high dynamic range. The various RAW formats support whatever dynamic range the underlying device associated with the RAW file uses. Personally, I am not a fan of RAW formats for long term image stoage, since they are too device and vendor specific. However, that’s a whole other debate.
I don’t yet know if OpenEXR will become a consumer standard, or if it will remain a file format used only by Hollywood. Microsoft, Apple, Adobe, Canon and others will no doubt have a big hand in shaping that decision.
Editing
Of all imaging tasks, editing is the one that demands the highest dynamic range. Editing operations need high precision to avoid aliasing artifacts such as banding and jaggies.
Audio professionals know this. Editing tools like ProTools already use 48 bits per sample internally, even though the common CD output format only supports 16. Why should we image makers accept anything less?
Recently, Adobe announced supports for 32-bit-per-channel HDR images in Photoshop CS2, a great step forward.
Idruna Software is another company doing interesting HDR software. I played with their PhotogenicsHDR when it first came out, but I found it a little hard to use. Perhaps the newest PocketPC version is different…
Photoshop users are familiar with the issues of low dynamic range today. With 8 bit channels, if you brighten an image, information is lost irretrievably: darkening the image after brightening does not restore the original appearance. Instead, all of the highlights appear flat and washed out. To avoid this problem, you must work in a carefully planned workflow.
With a true HDR tool, if you brighten an image and then darken it, you should see something very close to the original image. True HDR editing tools will enable image workers to follow a much more flexible and simplified workflow, using fewer adjustment layers, with fewer aliasing artifacts. I expect HDR software will lead to increases in productivity and greater expressiveness.
It will take the imaging software industry some time to retool and retrain. There are plenty of unsolved issues.
With HDR, for example, you run into the issue of representing brightness values present in the image but beyond what your current monitor can show. Do you clamp to the monitor’s gamut, show zembra stripes, map the colors some other way?
Another issue is how to create graphical user interfaces for HDR editing. Many designers are familiar with the RGB 0-255 color values, and can type in RGB color numbers directly using this system. e.g. 128,128,128 is a mid gray. But what happens when intensities go from 0 to several million? Where is mid gray? And how do you represent that in a graphical interface? If the tonal range goes from 0 to something close to the brightness of the sun, where is “white” on that scale? Do you mean monitor white, paper white, 3200k white, 5600k white …?
A third unsolved issue is image size: If each channel of an image is 32 instead of 8 bits, the image becomes four times larger. Switching to HDR therefore makes a 100mb image take up 400mb. Not surprisingly, editing operations take about four times longer. Software will need to become smarter about scheduling work. Live Picture, an early image compositing tool, did a good job of this, but is no longer available. I expect to see a revival of these techniques as people grapple with 10GB images.
Tone Mapping
Most LCD/CRT displays (and of course printed paper) have low dynamic range.
So if you want to output an HDR image on paper or on a display, you must somehow convert the wide intensity range in the image to the lower range supported by the display. This process is called tone mapping.
One old tone-mapping method is the manual dodge-and-burn technique familiar to photographers – where you manually select different tonal ranges for different regions of the image, using a dodging or a burning tool. HDR software will of course support manual dodge and burn.
Another solution is to use an automated tone mapping filter to reduce the dynamic range of an HDR image. There are already several filters to choose from.
The left image above shows what you get if you display Paul Debevec’s HDR photo of Stanford Memorial Church using a very basic tone mapping technique (simply clamping to the nearest available color on the monitor). Some areas of the image are “blown out”, and the shadow areas are muddy and lack detail.
The right image show’s Fattal, Lischinkski, and Werman’s tone mapping algorithm, which uses a more sophisticated adaptive approach – you see more details in the shadows and the highlights (look at the stained glass windows), though the image also has a somewhat “flat” or “computerish” quality.
Tone mapping is a hot area of research in computer graphics. As with HDR file formats, there is currently no clear winner. Several techniques are listed in the resource section. I expect the major companies to each champion their own tone-mapping technologies in service bureaus and print finishing.
Output
Over the past decade, display companies have steadily improved the dynamic range of LCD and DLP displays. Today many digital displays have a 2000:1 dynamic range, unheard of ten years ago. This trend of increasing dynamic range will continue.
A few displays available today indicate where the market is going. The most astonishing is the BrightSide HDR display with a claimed contrast ratio of 60,000:1, good enough to reproduce the effect of a sunlit scene. They achieve this using high-power white LEDs.
The only bad thing about the BrightSide display is that once you look at it for a few minutes you just assume that this is how images are supposed to look – it is such a transparently great technology that until you see a normal image on a normal display you don’t really think of the HDR display as that exciting. The display is still in the very-expensive bracket, but this will change quickly.
Of course, HDR displays work best if you have lots of HDR images. I anticipate a huge market for stock HDR imagery. See the Flickr HDR group for starters.
Applications
Today, the main users of HDR imaging devices are specialized professionals working in the film, animation and VR industries. Some applications are listed below (see the Resources page for links).
Film – Tools such as HDRShop by Paul Debevec enable you to convert a series of photographs into a light probe – a special image that represents the lighting environment in a room. You can then use the light probe to light virtual objects, so that the virtual objects actually appear to be lit by the light from the room. This technique is especially useful for compositing computer graphic objects into images of real scenes. Hollywood films use light maps extensively to blend CGI into a scene.
Panoramas – Another use for HDR is in panoramic images. Panoramas often have a wide dynamic range, e.g. one part of the panorama may contain the sun, another part may be in deep shadow. Online web panoramas constructed from HDR images look much better than non-HDR equivalents.
Games – A third use for HDR is in computer games. Recent computer graphics cards support HDR texture maps. With HDR texture maps, you can render objects using light probes, in real time, yielding much more dynamic and interesting lighting effects. “High Dynamic Range Lighting Effects” are now all the range in high-end games.
As more consumer-oriented HDR products arrive, I believe the largest application of HDR will be in consumer photography, though the term HDR is unlikely to be seen – instead you will see branding terminology, e.g. companies will make up words like “DynaChrome”, “MaxBright”, “SuperColor” etc.
Do we really need HDR?
I recently read this comment from Sam Berry:
… the whole article has no mention of the fact that the reason most controlled lighting is almost always done to ratio of less than 8:1 even with neg film /modern digital capable of much more is because that’s what looks good. HDR technology now means you can reproduce your harsh midday sunlit scene perfectly, and it will look identically awful compared to the original.
The debate boils down to this: Does an image with a 300:1 dynamic range look good because it represents a physical sweetspot — something to do with our perceptual system that works well at that ratio? Or is it that all we’ve had access to for hundreds of years are reflective images with a roughly 300:1 dynamic range, so we are accustomed to that?
I had a similar question in my mind before seeing the BrightSide HDR display. Now, after looking at a HDR image on a 50,000:1 HDR display, I am no longer concerned about over-brightness, 50,000:1 is still way less than the brightness of looking directly at the sun. It wasn’t blinding. It isn’t a question of harsh. Images simply looks better when they look more real.
In the coming decade, HDR digital imaging technology will arrive, and change how we take, manipulate, store, use and display images forever.
No comments:
Post a Comment