Lighting VFX Shots the IBL Way

This is how we do it in 2009

About five years ago I wrote this tutorial about integrating a CG object into a real, photographed scene. I've also written tutorials on ambient occlusion and Image Based Lighting (IBL) in the past too. Today most visual effects studios use a combination of these techniques to light shots. The main change in the last few years has been the almost wholesale adoption of ambient occlusion (defined here) combined with some kind of measured on-set environment lighting to get most of the way to a final render. Traditional CG lights are still used but generally only to add extra kicks and highlights rather than to define the overall lighting scheme as used to be the case.

So what's the method in all this?

In order to produce shots in the same way as a VFX studio you'll need a good digital camera with manual controls, a tripod, a fish-eye lens or a reflective ball and access to the actual environment you plan on recreating. Let's look at a super simple example I've put together. I haven't spent as long as I would were I doing this for real so it's a little rough around the edges but it should help you to understand the processes involved.

 A real scene

This is a photographed scene. I am going to create a CG pepper mill to replace the real one seen here. My goal is to show you what you can achieve using only IBL and occlusion. The only ray-tracing that I shall do is the occlusion pass. It's worth bearing in mind that in VFX work we often do not use multi-bounce ray-tracing or "final gather" type effects because they are expensive to compute and even with a render-farm we still have to make serious efforts to keep our renderings as efficient as possible. Ambient occlusion is regarded as being a good compromise between old fashioned all CG spot-lighting techniques and "final gather" type effects because it only has to be computed once as it is lighting independent. By that I mean that the occlusion pass remains the same whatever lights are added to the scene. "Final Gather" is light dependent and has to be recomputed every time a light parameter is changed which is a big hit on render times.

So what is required to achieve this lighting technique? The first thing is a background photographed element into which the CG can be composited.

 The photographed plate

All I've done is to remove the pepper mill and took another photo. I am working on a still image here but generally back-plates will be a series of frames making up a moving shot. This plate has to be tracked to produce a CG camera that exactly matches the movement and field of view of the real camera. This ensures that the CG objects will appear to be in the same physical space as the real ones. There are a variety of tracking packages out there such as Boujou.

As I am working on a still, and do not have tracking software at home, I just made a careful note of the focal length of my camera lens and found out the size of its sensor, added those deails to my camera in Maya and then eyeballed the line up using some proxy geometry I made for the table that the objects rest upon. I took measurements of the real table with a tape measure to ensure that the CG proxy object had the same proportions as the real one. As well as our background we also need to take photographs to enable us to recreate the lighting.

 A series of light probe photographs

Here at the bottom of the image is a series of photographs I took of a reflective sphere (light probe) over several exposures from 1 second to 1/8000 second. It is very important that each of these photographs lines up perfectly with the others so it is vital to use a good tripod to ensure stability. I halved the exposure each time I released the shutter e.g. if my current exposure was 1/125 second, my next picture would be taken at 1/250 second. As can be seen from the thumbnails at the bottom, the full range of lighting from the deepest shadow colour to the brightest highlight is captured. The main image is just a blow up of the middle picture of the top row just so the layout of tripod and camera can be seen. It is best if you take your photographs horizontally level with the light-probe and that the probe is centred in frame. It is also best if you take these pictures at the maximum resolution your camera will allow.

 The completed light probe HDR image

The set of photographs from the previous image is loaded into Photoshop CS using the "Merge to HDR" option on the "File" menu. The resulting image is a single floating point High Dynamic Range (HDR) image that captures all the lighting range present in the set of multiple exposures. This image is cropped to fit the sphere to the edges and saved out as a floating point TIFF image. This image will be our sole light source for rendering our CG object. N.B. The image above is just a JPEG showing the cropping; as JPEGs are 8-bit only this still does not have all the dynamic range present in the original.

 A tracked CG scene with geometry placed.

Here is my tracked Maya scene showing a hastily modelled CG pepper mill. You can also see proxy geometry for the table, the wall at the back of the photographed plate and the camera.

 Ambient Occlusion pass

By making all the geometry apart from the pepper-pot visible to rays but invisible to the camera it is posssible to render this ambient occlusion image. The darkening contribution of the close wall and table are visible on the pepper mill geometry. Almost all renderers now have an ambient occlusion shader available either out of the box or online.

 Reflection occlusion pass

As the pepper mill is quite shiny and has a chromed metal cap it is also necessary to render some reflection contribution. This pass is is very similar to the ambient occlusion except that instead of the rays being scattered over a hemi-sphere around the shaded point's surface normal the rays stay closer to the normal making the occlusion sharper. If the mill were made of chrome I would make the reflection occlusion as sharp as possible, i.e. the rays would only travel down the surface normal, this is what I ahve done for the metal cap. The majority of the mill we are trying to copy has slightly dulled varnish so I picked a reflection occlusion blur value somewhere between ambient occlusion and perfect reflection as you can see in the above render.

 Diffuse IBL pass

Here is the meat of this tutorial. The above image is the result of using the IBL HDR map we made earlier in an environment light. In order to create diffuse illumination I have blurred the map a great deal. I added the map to an ambient light as an "Angular Map". Strictly speaking the map I made isn't quite an angular map but for the sake of this tutorial it's close enough. If you have access to something like Paul Debevec's HDRShop you could convert your map into a lat-long formatted image which is the most popular layout for HDR environment maps and is most widely supported by renderers.

I attached a standard, pure white, lambertian diffuse surface to the wooden part of the pepper mill and rendered with the single ambient light with the blurred HDR map plugged into it. There's no shadowing or ray-tracing going on here, just a simple map look-up. This render is fast and if your map doesn't immediately line up orientation-wise it is quite quick to try different angles of rotation on the light until it does the right thing. The metal cap is left black because it has almost no diffuse contribution, it is mostly reflection.

 Diffuse texture pass

Here is a scrappily painted diffuse texture map rendered on the wooden part of the mill. This is a flat texture with no lighting information within it. The darkening you see around the base and at the top are due to years of grime build up. Yuck.

 Reflection IBL pass

Here we use the HDR map with almost no blur on it to create this image. The metal cap is included in this render as reflected light forms the majority of its look. We also can see the reflection of the window in the top of the wooden mill lid very clearly. Once again this is just an environment light and a map-based reflection call in the shader.

 Reflectivity pass

As the wooden parts of the mill are pitted and have areas where the varnish is more worn I've created this very crude map to represent these variations. Where the mill is white there's still plenty of varnish, the closer to black we get the less reflectivity we will see. This map is another flat texture render. The metal cap is left white as it is pretty much perfectly reflective.

 The recipe for combining the CG passes

This diagram shows how to combine your passes to create a complete CG object. As you can see the three elements that make up the diffuse contribution: ambient occlusion, diffuse texture and diffuse IBL are multiplied together. The result of this operation is then added to the product of the reflectivity contribution: reflection occlusion, reflectivity map and reflection IBL pass. You could perform all these operations within a single shader and render out a single complete CG pass. I would counsel against this. I think it is better to render each of the passes out to disk and then do the layering in your compositing software as it will give you the maximum control of the final image and it is quicker to tweak a parameter in the comp than it is to have to re-render the whole pass again.

 Layered up CG elements

Here's the result of the layering process described above. It is now ready to be composited into the photo. There's just one thing left to do first...

 Contact shadow pass

In order to get te CG to feel connected to the photographed plate it is necessary tor render a contact shadow. Here I've used the proxy table geometry and made the pepper mill visible to rays, but not the camera. As the shadows that the camera casts on the table are very soft I have rendered a semi-soft reflection occusion pass here to act as the shadow. If the shadows were harder I would have used a regular CG spot light and matched the angle of the real lighting in the scene to cast a believable shadow. Now we can composite this into the plate.

 Contact shadow composited into photographed plate

Here's the contact shadow in the plate. As you can see I've tinted it to be the same brownish shade as the camera's real shadow.

 CG object composited into photographed plate

And here's the finished comp. It's a long way from perfect, there's no attempt to blur the CG to match it into the plate, nor have I balanced the black levels, added grain or any of the other techniques that a proper compositor would use to seamlessly blend CG into photography. This is just the layered CG over the background. It's not as good as it could be given more time and better maps but it shows what can be achieved in less than an afternoon with the right preparation and photographic elements. I hope you found this useful.

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 Unported License.