When a background is not of a consistent and distinctive color, you can’t remove the background with keying effects. Under these conditions, you may need to use rotoscoping—the manual drawing or painting on individual frames to isolate a foreground object from its background. (See Rotoscoping introduction and resources.)
Keying is defining transparency by a particular color value or luminance value in an image. When you key out a value, all pixels that have colors or luminance values similar to that value become transparent.
Keying makes it easy to replace a background, which is especially useful when you work with objects too complex to mask easily. When you place a keyed layer over another layer, the result forms a composite, in which the background is visible wherever the keyed layer is transparent.
You often see composites made with keying techniques in movies, for example, when an actor appears to dangle from a helicopter or float in outer space. To create this effect, the actor is filmed in an appropriate position against a solid-color background screen. The background color is then keyed out and the scene with the actor is composited over a new background.
The technique of keying out a background of a consistent color is often called bluescreening or greenscreening, although you don’t have to use a blue or green screen; you can use any solid color for a background. Red screens are often used for shooting non-human objects, such as miniature models of cars and space ships. Magenta screens have been used for keying work in some feature films renowned for their visual effects. Other common terms for this kind of keying are color keying and chroma keying.
Difference keying works differently from color keying. Difference keying defines transparency with respect to a particular baseline background image. Instead of keying out a single-color screen, you can key out an arbitrary background. To use difference keying, you must have at least one frame that contains only the background; other frames are compared to this frame, and the background pixels are made transparent, leaving the foreground objects. Noise, grain, and other subtle variations can make difference keying very difficult to use in practice.
After Effects includes several built-in keying effects, as well as the Academy Award-winning Keylight effect, which excels at professional-quality color keying. (See Keying effects and Matte effects.)
For information on the Keylight effect, see its documentation in the folder in which the Keylight plug-in is installed, or on the Foundry website.
Though the color keying effects built into After Effects can be useful for some purposes, you should try keying with Keylight before attempting to use these built-in keying effects. Some keying effects—such as the Color Key effect and the Luma Key effect—have been superseded by more modern effects like Keylight.
The Key Cleaner and Advanced Spill Suppressor effects are most effective when applied together, in that order, after a keying effect like Keylight.
Use the Keylight + Key Cleaner + Advanced Spill Suppressor animation preset (located in the Image-Utilities presets folder) to apply all the three effects. The Advanced Spill Suppressor effect is turned off by default to allow you to sample the key color in the Keylight effect or if the footage does not have any color spill to be removed. For more information, see Key Cleaner and Advanced Spill Suppressor effect.
Mark Christiansen provides tips and techniques for using Keylight in an excerpt from his book After Effects Studio Techniques: Visual Effects and Compositing on the Peachpit Press website. In an excerpt from the “Color Keying in After Effects” chapter of After Effects Studio Techniques, Mark Christiansen provides detailed tips and techniques for color keying, including advice on which keying effects to avoid and how to overcome common keying challenges.
For a step-by-step tutorial demonstrating the use of the Color Difference Key effect, the Matte Choker effect, the Spill Suppressor effect, and garbage masks, see the “Keying in After Effects” chapter of the After Effects Classroom in a Book on the Peachpit Press website.
Jeff Foster provides free sample chapters from his book The Green Screen Handbook: Real World Production Techniques. The sample chapters cover basic compositing, color keying, garbage mattes, hold-out mattes, and how to avoid common problems with greenscreen shots. For more information, see the Adobe website.
Rich Young collects more tips and resources for keying on his After Effects Portal website.
Tips on color keying and compositing from experienced compositor, Chris Zwar.
Keep in mind that generating a high-quality key can require the application of multiple keying effects in sequence and careful modification of their properties, especially if the footage was shot without considering the requirements of the compositor.
For tips on shooting footage so that color keying is easier and more successful, see Jonas Hummelstrand’s General Specialist website.
Light your color screen uniformly, and keep it free of wrinkles.
Start with the highest-quality materials you can gather, such as film that you scan and digitize.
Use uncompressed footage (or, at least, files with the least possible amount of compression). Many compression algorithms, especially the algorithms used in DV, HDV, and Motion JPEG, discard subtle variations in blue—which may be necessary to create a good key from a bluescreen. Use footage with the least color subsampling possible—for example, 4:2:2 rather than 4:2:0 or 4:1:1. (For information about color subsampling, see the Wikipedia website and the Adobe website.)
Robbie Carman and Richard Harrington provide an excerpt on the Peachpit website from their book Video Made On A Mac that demonstrates how to plan, shoot, key, and composite a greenscreen shot.
A garbage matte (or junk matte) removes unneeded portions of the scene, resulting in a rough area that contains only the subject that you want to keep. When you are working with a poorly lit or uneven color screen (for example, a bluescreen or greenscreen), sketching a garbage matte around the subject can greatly reduce the amount of work that you have to do in keying out the background. However, if you spend a lot of time making a perfect garbage matte that exactly outlines the subject—essentially rotoscoping—you lose the time-saving advantage of keying.
Aharon Rabinowitz provides a video tutorial on the Creative COW website that shows how to create a super-tight garbage matte using Auto-trace.
Use a hold-out matte (also known as a hold-back matte) to patch a scene to which a keying effect has been applied.
A hold-out matte is a masked-out portion of a duplicate of a layer that you have keyed. The duplicate is masked to include only the area of the image that contains the key color that you want to preserve as opaque. The hold-out matte is then placed directly on top of the keyed layer.
A. Original bluescreen image. The background for the number is also blue. B. After keying, the background for the number is also transparent. C. Hold-out matte containing the part of the image you want to remain opaque D. When the hold-out matte is placed on top of the keyed image, the background for the number is now opaque.
Don’t change Transform properties of only one of the layers after making the duplicate; keep the layers moving together. Consider parenting one to the other. (See Parent and child layers.)