Color information in After Effects is contained in three channels: red (R), green (G), and blue (B). In addition, an image can include an invisible fourth channel, called an alpha channel, that contains transparency information. Sometimes, such an image is referred to as an RGBA image, indicating that it contains an alpha channel.

Many file formats can include an alpha channel, including Adobe Photoshop, ElectricImage, FLV, TGA, TIFF, EPS, PDF, and Adobe Illustrator. AVI and QuickTime (saved at a bit depth of Millions Of Colors+), can also contain alpha channels, depending upon the codec (encoder) used to generate the images stored in these containers. For Adobe Illustrator EPS and PDF files, After Effects automatically converts empty areas to an alpha channel.

When specifying the color depth for an output image, the plus sign (as in Millions Of Colors+) denotes an alpha channel. Similarly, choosing to output to 32 bits per pixel implies an output depth of 8 bits per channel for each of four channels: RGBA.

Some programs can store multiple alpha channels in one image, but After Effects only interprets the fourth channel as an alpha channel.

The Knoll Unmult plug-in can be used to create an alpha channel from the dark areas of a layer. This works well for a layer with a light effect (such as a lens flare or fire) that you want to composite on top of another layer. For information, see the Red Giant Software website.

When you view an alpha channel in the Composition panel, white indicates complete opacity, black indicates complete transparency, and shades of gray indicate partial transparency.

A matte is a layer (or any of its channels) that defines the transparent areas of that layer or another layer. White defines opaque areas, and black defines transparent areas. An alpha channel is often used as a matte, but you can use a matte other than the alpha channel if you have a channel or layer that defines the desired area of transparency better than the alpha channel does, or in cases where the source image doesn’t include an alpha channel.

About masks

A mask in After Effects is a path that is used as a parameter to modify layer attributes, effects, and properties. The most common use of a mask is the modification of an alpha channel of a layer, which determines the transparency of the layer at each pixel. Another common use of a mask is as a path along which to animate text. (See Creating and animating text on a path.)

For more information on paths in general, see About paths.

Closed-path masks can create transparent areas for a layer. Open paths cannot create transparent areas for a layer but are useful as parameters for an effect. Effects that can use an open or closed mask path as input include Stroke, Path Text, Audio Waveform, Audio Spectrum, and Vegas. Effects that can use closed masks (but not open masks) as input include Fill, Smear, Reshape, Particle Playground, and Inner/Outer Key.

A mask belongs to a specific layer. Each layer can contain multiple masks.

You can draw masks in common geometric shapes—including polygons, ellipses, and stars—with the shape tools, or you can use the Pen tool to draw an arbitrary path.

In most ways, drawing mask paths is the same as drawing shape paths on shape layers, though the editing and interpolation of mask paths have a few additional features. You can link a mask path to a shape path using expressions, which allows you to bring the benefits of masks into shape layers, and vice versa. See Creating shapes and masks and Editing and animating shape paths and masks.

The position of a mask in the stacking order in the Timeline panel affects how it interacts with other masks. You can drag a mask to different positions within the Masks property group in the Timeline panel.

The Mask Opacity property for a mask determines the influence that a closed mask has on the alpha channel of the layer inside the mask area. A Mask Opacity value of 100% corresponds to an interior area that is opaque. The area outside the mask is always transparent. To invert what is considered inside and what is considered outside for a specific mask, select Invert next to the mask name in the Timeline panel.

Copy, cut, save, reuse, and delete masks

You can reuse masks in other layers and compositions, which is especially useful for Bezier masks you’ve spent a long time perfecting. Mask paths are stored inside a composition in a project file.

Copy, cut, duplicate, or paste a mask

• To copy or cut selected masks to the clipboard, choose Edit > Copy or Edit > Cut.
• To duplicate selected masks, choose Edit > Duplicate.
• To paste a mask onto a layer, select the layer and then choose Edit > Paste. If a mask is selected, this operation replaces the selected mask.

Save a mask

Reuse a mask

Delete masks

• To delete one mask, select the mask in the Timeline panel and press Delete.
• To delete all masks, select the layer containing the masks you want to remove and choose Layer > Masks > Remove All Masks.

Control mask path color

To help you identify and work with masks, the Composition and Layer panels outline a mask path with color, and the Timeline panel displays that same color next to the name of the mask. By default, After Effects uses the color yellow for all masks. To make each mask more distinctive, you can manually change the color of a mask using the Timeline panel, or you can set After Effects to cycle through mask colors for new masks.

When the Use Contrasting Color For Mask Path preference (under Preferences > Appearance) is enabled, After Effects analyzes the colors near the point where you start drawing a mask. After Effects then chooses a label color that is different from the colors in that region. It also avoids the color of the last mask drawn.

Change mask path color

Cycle through colors for mask paths

Mask modes

Blending modes for masks (mask modes) control how masks within a layer interact with one another. By default, all masks are set to Add, which combines the transparency values of any masks that overlap on the same layer. You can apply a mode to each mask, but you can’t animate the mode of a mask—that is, you can’t set keyframes or expressions for a mask mode property to make it change over time.

You choose a mask mode for a mask from the menu next to the mask name in the Timeline panel.

The first mask that you create interacts with the alpha channel of the layer. If that channel doesn’t define the entire image as opaque, then the mask interacts with the layer frame. Each additional mask you create interacts with masks located above it in the stacking order in the Timeline panel. The results of mask modes vary depending on the modes set for the masks higher in the stacking order. Mask modes only operate between masks on the same layer.

Using mask modes, you can create complex compound masks with multiple transparent areas. For example, you can set a mask mode that combines two masks and sets the opaque area to the areas where the two masks intersect.

The mask has no direct influence on the alpha channel of the layer. This option is useful when you are only using the path of the mask for an effect such as Stroke or Fill, or if you are using the mask path as the basis for a shape path.

The mask is added to the masks above it in the stacking order. The influence of the mask is cumulative with the masks above it.

The influence of the mask is subtracted from the masks above it. This option is useful when you want to create the appearance of a hole in the center of another mask.

The mask is added to the masks above it in the stacking order. In areas where the mask overlaps the masks above it, the influence of the mask is cumulative with the masks above it. In areas where the mask does not overlap with the masks above it, the result is complete opacity.

The mask is added to the masks above it in the stacking order. Where multiple masks intersect, the highest transparency value is used.

The mask is added to the masks above it in the stacking order. Where multiple masks intersect, the lowest transparency value is used.

The mask is added to the masks above it in the stacking order. In areas where the mask does not overlap the masks above it, the mask operates as it would alone on the layer. In areas where the mask overlaps the masks above it, the influence of the mask is subtracted from the masks above it.

Use the following keyboard shortcuts to change the mode of a closed mask while you are drawing or modifying it:

Notes:

• When you draw a new mask with the Shape tools such as Rectangle tool, and Ellipse tool, you can press these keyboard shortcuts along with the mouse.
• When you draw a new mask with the Pen tool, you can only press these keyboard shortcuts when you close the mask. To  place the pointer over the closing vertex (usually the first vertex), click and hold the mouse button to close the mask, then press the mask mode keyboard shortcut before releasing the mouse button. Open masks do not have a mode. These keyboard shortcuts do not work when drawing open masks with the Pen tool.
• When you modify a closed mask, you can press these keyboard shortcuts at any time while you hold down the mouse button.

Expand or contract the edges of a mask

To expand or contract the area influenced by a mask, use the Mask Expansion property.

Mask expansion affects the alpha channel but not the underlying mask path; the mask expansion is essentially an offset that determines how far, in pixels, from the mask path the influence of the mask on the alpha channel extends.

You can scrub the Mask expansion values up to 32,000, and scrub down to -32,000. Earlier versions of After Effects allow scrubbing these values to 1,000 and any value beyond 1000 were manually entered.

Todd Kopriva provides a visual aid and further explanation regarding mask expansion—and why it creates rounded corners—on his blog on the Adobe website.

Soften (feather) the edges of a mask

Feathering softens the edges of a mask by fading it from more transparent to less transparent over a user-defined distance. Using the Mask Feather property, you make mask edges hard-edged or soft-edged (feathered). By default, the feather width straddles the mask edge, half inside, and half outside. For example, if you set the feather width to 25, the feathering extends 12.5 pixels inside the mask edge and 12.5 pixels outside it.

You can scrub Mask Feather values up to 32,000 using the slider.

You can also extend or contract the mask edges using the Mask Expansion property to control where the mask feathering appears. (See Expand or contract the edges of a mask.)

Mask feathering takes place only within the dimensions of the layer. Therefore, the path of a feathered mask should always be slightly smaller than the layer area and should never move to the very edge of the layer. If a mask feather extends beyond the layer area, the feathered edge ends abruptly.

Variable-width mask feathering

Earlier versions of After Effects enabled you to add a feathered edge to a closed mask, but the width (extent) of the feather was the same around the mask. A new Mask Feather tool (available in the same tool menu as the Pen tool) has been added to let you define points along a closed mask that should have varying widths.

To create a feather point:

A selected feather point has a small black dot at its handle.

Feather points define both the outer feather boundary and inner feather boundary. If no feather extent handles are inside the mask, the inner feather boundary is the mask path. The mask feather extends from the inner to the outer feather boundary.

To select multiple feather points:

Using the Selection or Mask Feather tool, Shift-click the feather points' extent handles.

To toggle the selection of a feather point:

Shift-click the feather point.

To move a feather point, do either of the following:

• Using the Selection or Mask Feather tool, drag the feather extent handle.

You can move multiple selected feather points in unison by dragging one of them, and scale multiple feather extents by dragging one of the feather extent handles.

You can "sweep" feather points around corner points on a mask.

• With the Mask Feather tool active, press Left or Right Arrow to move the feather along the path, or the Up or Down Arrow to lengthen or shorten the feather's width.
  

You can move multiple selected feather points, and move longer distances/widths by holding down the Shift key.

To snap a feather point to the mask path:

Drag the feather extent handle across the mask path. It will stop at the mask path.

To control the falloff of the feather:

Choose Layer > Mask > Feather Falloff, and then select one of the following:

• Smooth (default)
• Linear

To adjust the tension (smoothness or curvature) of a feather boundary through a feather point:

Hold down the Alt (Windows) or Option (MacOS) key while dragging from a feather extent handle to adjust the tension. The Info panel shows the current tension for a handle.

To set Hold interpolation for a feather point (for constant feather radius up to the next feather point):

Hold down the Alt (Windows) or Option (MacOS) key while dragging from a feather extent handle to adjust the tension. The Info panel shows the current tension for a handle.

To set Hold interpolation for a feather point (for constant feather radius up to the next feather point):

Enable the Hold option from the context menu above the feather point. The feather point handle changes to be pointy in the direction of constant radius.

To quickly create a feathered edge for a specific mask segment:

Hold down the Shift key as you click the mask segment (between vertices, not above them). The pointer changes to indicate that you are in this mode. Drag from the segment to adjust the extent. Note the following behavior:

• If just two mask vertices are selected, the feathered edge is the contiguous segments between them.
• If both vertices of the clicked segment are selected (that is, the segment is selected), the feathered edge expands to include contiguous selected segments.
• If all or no vertices on the mask are selected, or if the first two conditions don't apply, the feathered edge is only the clicked segment.

To delete a feather point:

Using the Selection or Mask Feather tool, select a feather extent handle, then press Delete. You can delete multiple selected feather points.

Tips for variable-width mask feathering (Mask Feather tool)

• Temporarily switch between Pen and Mask Feather tools by holding down the 'G' shortcut.
  

You can turn off this behavior in Edit > Preferences > General (Windows), or After Effects > Preferences > General (Mac OS).

• View information about the number of feather points on a mask, a feather extent's length, position, and tension, and feather falloff setting in the Info panel.

• Hide the feather boundaries for a specific view by deselecting the "Mask Feather Boundaries" option in the View Options dialog box. You can still interact with the boundaries (for example, adding new feather points) where they would've been drawn.

• Control feather point counts across Mask Path keyframes by using the "Preserve Constant Vertex and Feather Count when Editing Masks" option in General preferences.

• Change the value of a feather point's tension, radius, and corner angle from a dialog box by using the commands in the context menu for a feather point.
  Hold down the Ctrl (Windows) or Cmd (Mac OS) key when hovering over a mask vertex to switch to Selection tool behavior.

• Hold down the Ctrl (Windows) or Cmd (MacOS) key when hovering over a tangent handle to switch to the Convert Vertex tool.
  

Track mattes and traveling mattes

When you want one layer to show through holes defined by another layer, set up a track matte. For example, you can use a text layer as a track matte for a video layer to allow the video to only show through the shapes defined by the text characters. The underlying layer (the fill layer) gets its transparency values from the values of certain channels in the track matte layer—either its alpha channel or the luminance of its pixels.

Defining the transparency of a layer based on the luminance of the track matte’s pixels is useful when you want to create a track matte using a layer without an alpha channel or a layer imported from a program that can’t create an alpha channel. In both cases—using alpha channel mattes and using luminance mattes—pixels with higher values are more transparent. Usually, you use a high-contrast matte so that areas are either completely transparent or completely opaque. Intermediate shades should appear only where you want partial or gradual transparency, such as along a soft edge.

A track matte only applies to the layer directly beneath it. To apply a track matte to multiple layers, first precompose the multiple layers, and then apply the track matte to the precomposition layer.

After Effects preserves the order of a layer and its track matte after you duplicate or split the layer. Within the duplicated or split layers, the track matte layer remains on top of the fill layer. For example, if your composition contains layers A and B, where A is the track matte and B the fill layer, duplicating or splitting both of these layers results in the layer order ABAB.

If you animate the position or other transformations of the track matte layer, it’s called a traveling matte. If you want to animate the track matte and fill layers using identical settings, consider precomposing them.

Convert a layer into a track matte

The TrkMat menu shares a column with the blending modes menu. To show the TrkMat menu, make sure that the Modes column is visible. (See Columns.)

If you choose an option other than No Track Matte, After Effects converts the next layer above into a track matte, turns off the video of the track matte layer, and adds a track matte icon next to the name of the track matte layer in the Timeline panel.

Using a track matte is similar to using the Preserve Underlying Transparency option, which causes a layer to get its transparency from the transparency of the composite of the layers below it in the layer stacking order. (See Preserve underlying transparency during compositing.)

Tips for working with track mattes

• Use the Levels effect to increase the contrast between light and dark parts of the matte layer. This reduces the problem of having many mid-range values, which translate to partial transparency. (Usually, mattes are most useful when they define areas as entirely transparent or entirely opaque, except at the edges.)
• To use a channel other than the alpha channel of the matte layer as a matte, use one of the Channel effects (such as the Shift Channels effect) to route the desired channel’s value into the alpha channel.
• To animate a track matte to move with the layer that it’s matting, make the track matte a child of the layer that it’s matting. (See Parent and child layers.)

Preserve underlying transparency during compositing

The Preserve Underlying Transparency option causes a layer to get its transparency from the transparency of the composite of the layers below it in the layer stacking order. In other words, the opaque areas of the layer with this option selected appear only when positioned over opaque areas in underlying layers. This behavior is similar to the behavior of a track matte, except that a track matte can only be a single layer and a track matte must be above the layer in the layer stacking order. (See Track mattes and traveling mattes.)

This option is useful for creating results such as glints or light reflecting off a polished surface.

The behavior of a layer with the Preserve Underlying Transparency option selected is similar to the behavior of a clipping mask in Adobe Photoshop.

Resources for Mocha AE masking

You can use the Mocha AE plug-in from Boris FX for Planar Tracking and Masking. It uses a planar tracking workflow to drive shapes for masking. Mocha AE Spline shapes also include variable Per-Point Edge Feathering for precise control on edge blurring.

In previous versions of Mocha AE, the mask conversion process requires a copy/paste from Mocha AE to After Effects using the clipboard and the Mocha Shape plug-in. The Mocha AE plug-in applies or converts Mocha shapes to After Effects in the current workflow.

For more information, see Planar Tracking with Mocha AE.

See the Boris FX video tutorial section for video tutorials covering Mocha AE masking for After Effects.