You animate mask paths and shape paths in much the same way that you animate other properties: set keyframes for the Mask Path or Path property, set paths at each keyframe, and After Effects will interpolate between these specified values.
Rotoscoping (or just roto in casual usage) is the drawing or painting on frames of a movie, using visual elements in the movie as a reference. A common kind of rotoscoping is tracing a path around an object in a movie and using that path as a mask to separate the object from its background. This allows you to work with the object and the background separately, so you can do things like apply different effects to the object than to its background or replace the background.
Note:
After Effects includes the Roto Brush tool, which can be used to accomplish many of the same tasks as conventional rotoscoping, but in far less time. For information about using the Roto Brush tool, see Roto Brush and Refine Matte.
If a background or foreground object is a consistent, distinct color, you can use color keying instead of rotoscoping to remove the background or object. If the footage was shot with color keying in mind, color keying is much easier than rotoscoping. (See Keying introduction and workflow.)
Rotoscoping in After Effects is mostly a matter of drawing masks, animating the mask path, and then using these masks to define a matte. Many additional tasks and techniques make this job easier, such as using motion tracking on the object before you begin drawing masks, and then using the motion tracking data to make a mask or matte automatically follow the object.
Immediately after beginning to draw a mask, press Alt+Shift+M (Windows) or Option+Shift+M (Mac OS) to turn on keyframing for that mask and set a keyframe. This way, you won’t edit a mask frame-by-frame for several minutes (or longer) and then realize that you lost all of your work on previous frames because you forgot to click the stopwatch to make the mask path animated.
Draw your masks on a white solid layer with its Video (eyeball) switch off, above the (locked) footage layer. This way, you run no risk of accidentally moving the footage layer when you manipulate the mask, and you can also much more easily apply tracking data to the mask. (You apply the tracking data to the invisible solid layer that holds the mask.) This also means that you don't lose your cached RAM preview frames each time you manipulate the mask. (See Toggle visibility or influence of a layer or property group and Lock or unlock a layer.)
Turn on the Preserve Constant Vertex Count preference. (See Designate the first vertex for a Bezier path.)
Note: Preserve Constant Vertex Count When Editing Masks is called "Preserve Constant Vertex and Feather Count when Editing Masks", in After Effects CS6.
When possible, transform (rotate, scale, move) the whole mask or a subset of the mask vertices instead of moving the vertices individually. This is both for efficiency and to avoid the chatter that comes from inconsistent movement across frames. (See Move vertices in free-transform mode.)
- Manual motion tracking is less time-consuming than manual rotoscoping. The more effort you spend getting good tracking data for various parts of your scene and object, the less time you'll spend drawing and fine-tuning masks. (See Tracking and stabilizing motion.)
- In After Effects CS6, use the variable-width mask feather feature for more control when feathering objects.
This video from the After Effects CS5: Learn By Video series shows how to combine motion tracking and rotoscoping to isolate and selectively color-correct an actor's face.
Scott Squires provides a pair of movies on his Effects Corner website that show how to rotoscope, both painting and masking:
Chris and Trish Meyer provide some tips on animating masks, including using Smart Mask Interpolation, on the ProVideo Coalition website.
Alejandro Pérez provides a script on the AE Enhancers forum with which you can use tracking data to position individual mask vertices.
Mathias Möhl provides the KeyTweak script on his website, with which you can modify many keyframes on a property simultaneously. With KeyTweak, you can modify a few keyframes manually, and the script modifies the remaining keyframes in between accordingly. KeyTweak is especially useful for Mask Path keyframes in a rotoscoping workflow.
Rich Young provides several resources for rotoscoping on his After Effects Portal website.
- To view mask paths for selected layers in the Timeline panel, press M.
- To view selected masks or shapes in the Timeline panel, press SS (press the S key twice).
- To view mask and shape paths in the Composition panel,
click the Toggle Mask And Shape Path Visibility button
at the
bottom of the Composition panel. - To view mask paths in the Layer panel, choose Masks from the Layer panel View menu.
- To hide a mask path while showing others, lock the mask
by selecting its Lock switch
in the
Timeline panel, and then choose Layer > Mask > Hide Locked Masks. - To isolate selected masks and hide others, choose Layer > Mask > Lock Other Masks, and then choose Layer > Mask Hide Locked Masks.
You can select shape layers and their components at any of four levels of selection, referred to as selection modes:
Layer selection mode
The entire shape layer is selected. Transformations apply to the transform properties for the layer, in the Transform property group that is at the same level as the Contents property group.
Group selection mode
An entire shape group is selected. Transformations apply to the transform properties for the group, in the Transform property group within the shape group in the Timeline panel.
Free-transform mode
Multiple vertices on one or more Bezier paths are selected. A free-transform bounding box is shown around the vertices in the Composition panel. By operating on this box, you can move multiple vertices with a single transformation. Transformations apply to the vertices themselves, which are contained within the Path property in the Timeline panel.
Path-editing mode
Only vertices are selected. In this mode, you can perform path-editing operations, such as adding vertices to a path and moving individual vertices.
When a pen tool is active, path-editing mode is active. To remain in path-editing mode, select the Pen tool; press V or Ctrl (Windows) or Command (Mac OS) to temporarily activate the Selection tool as needed.
Selection modes for shapes on shape layers
A. Layer selection B. Group selection C. Free-transform D. Path editing
For information on selecting masks, see Select masks, segments, and vertices.
Press Ctrl+A (Windows) or Command+A (Mac OS) with a shape vertex selected to select all vertices on that path. Press again to select all shapes. Press again to select all layers.
- Click the layer name or layer duration bar in the Timeline panel.
- Using the Selection tool, click within the layer bounds in the Composition panel.
- To deselect all shapes on a layer but leave the shape layer selected, click within the layer bounds but outside all shape paths.
- Using the Selection tool, double-click a member of the group in the Composition panel. Each time that you double-click, you descend another level in the group hierarchy.
- To activate the Direct Selection tool
, hold
Ctrl (Windows) or Command (Mac OS) with the Selection tool selected.
Click a shape in the Composition panel with the Direct Selection
tool to directly select that shape’s group, regardless of how deeply
nested the shape is in the group hierarchy. - To select a group that is contained within the same group as the group that is already selected, click the group to select.
- To add a group to a selection, Shift-click it. You can combine the Shift key with double-clicking and with the Direct Selection tool to add more deeply nested groups to the selection.
To specify the size of Bezier direction handles and vertices for masks and shapes, choose Edit > Preferences > General (Windows) or After Effects > Preferences > General (Mac OS), and edit the Path Point Size value.
- To select a vertex, click the vertex with the Selection tool. To add vertices to the selection, Shift-click them.
- To select a path segment, click the segment with the Selection tool. To add segments to the selection, Shift-click them.
- To select an entire path, Alt-click (Windows) or Option-click (Mac OS) a segment or vertex of the path with the Selection tool, or select any portion of the path and press Ctrl+A (Windows) or Command+A (Mac OS).
- To select vertices by dragging, select a path or portion
of a path to enter path-editing mode, and then drag with the marquee-selection
tool
to
draw a marquee-selection box around the vertices to select. To add
vertices to the selection, hold down the Shift key as you draw additional
marquee-selection boxes.
- Double-click a path segment while in path-editing mode or in group selection mode for a single shape.
- Select the Path property in the Timeline panel and press Ctrl+T (Windows) or Command+T (Mac OS).
Unlike layers, masks can have more than one level of selection. You can select a mask as a whole path, which is appropriate when you want to move or resize a mask. However, if you want to change the path of a mask, select one or more points on it. Selected points appear solid, and unselected points appear hollow.
To specify the size of Bezier direction handles and vertices for masks and shapes, choose Edit > Preferences > General (Windows) or After Effects > Preferences > General (Mac OS), and edit the Path Point Size value.
- To select a vertex on a mask,
click the vertex with the Selection tool
. To
add vertices to the selection, Shift-click them. - To select a mask segment, click the segment with the Selection tool. To add segments to the selection, Shift-click them.
- To select an entire mask, Alt-click (Windows) or Option-click (Mac OS) a segment, vertex, or handle of a mask with the Selection tool, or select any portion of the mask and choose Edit > Select All or press Ctrl+A (Windows) or Command+A (Mac OS). To add masks to the selection, Alt+Shift-click (Windows) or Option+Shift-click (Mac OS) them.
- To select masks by dragging, select a mask or portion of a mask to enter mask editing mode and then drag with the Selection tool to draw a marquee-selection box completely around the vertices or masks that you want to select. To add masks or vertices to the selection, hold down the Shift key as you draw additional marquee-selection boxes.
- To select all masks on a layer, select a mask on the layer, and choose Edit > Select All or press Ctrl+A (Windows) or Command+A (Mac OS).
- To deselect all masks, press Ctrl+Shift+A (Windows) or Command+Shift+A (Mac OS).
- To select an adjacent mask on a layer, press Alt+accent grave (`) (Windows) or Option+accent grave (`) (Mac OS) to select the next mask, or Shift+Alt+accent grave (`) (Windows) or Shift+Option+accent grave (`) (Mac OS) to select the previous mask.
- To deselect a mask, click anywhere other than on the mask.
- To remove a vertex or segment from a selection, Shift-click the vertex or segment.
To use the Selection tool when the Pen tool is selected, hold down Ctrl (Windows) or Command (Mac OS).
Note:
You can select only whole masks in the Timeline panel. To select individual vertices on a mask, use the Composition or Layer panel.
Locking a mask prevents you from selecting it in the Timeline, Composition, and Layer panels or setting it as a target in the Layer panel. Use this feature to avoid unwanted changes to the mask.
next
to the mask you want to lock or unlock. A mask is locked and cannot
be selected when its Lock switch is selected—that is, when the Lock
icon appears in the box.
Note:
To unlock multiple masks at one time, select one or more layers and choose Layer > Mask > Unlock All Masks.
To isolate selected masks and hide others, choose Layer > Mask > Lock Other Masks, and then choose Layer > Mask Hide Locked Masks.
You can scale and rotate an entire mask or shape path (or selected vertices in one or more paths) using the Free Transform Points command. When you use this command, a free-transform bounding box surrounds the selected vertices, and an anchor point appears in the center of the bounding box to mark the anchor point for the current transformation. You can scale and rotate the selected vertices by dragging the bounding box or its handles. You can also change the reference point from which the vertices are rotated or scaled by moving the bounding box anchor point. The free-transform bounding box handles and anchor point exist independently of the handles and anchor point for the layer.
Note:
When you animate rotation using Free Transform Points, the vertices of the mask are interpolated in a straight line from keyframe to keyframe. For this reason, the results may be different from what you expect.
- To scale the path or selected vertices, position
the pointer on a bounding box handle and, when the pointer changes
to a straight, double-sided arrow
, drag
to a new size. Hold down Shift as you drag to constrain the scale.
Hold down Ctrl (Windows) or Command (Mac OS) as you drag to scale
around the anchor point of the bounding box.
- To scale the path or selected vertices, position
the pointer on a bounding box handle and, when the pointer changes
to a straight, double-sided arrow
until
the Selection tool changes to a move anchor point icon 
. Drag
to position the anchor point.
, drag
to rotate.You can convert any manual Bezier mask path or manual Bezier shape path to a RotoBezier path. If the manual Bezier path has direction handles that have been adjusted, this conversion changes the shape of the path, because After Effects calculates the curvature of RotoBezier segments automatically.
The conversion of a RotoBezier path to a manual Bezier path does not change the shape of the path.
You can change a Bezier mask path using the Selection tool and pen tools in the Layer or Composition panel. You can change a Bezier shape path using the Selection tool and pen tools in the Composition panel.
The pen tools—Add Vertex, Delete Vertex, and Convert Vertex tools—are grouped with the Pen tool in the Tools panel. To reveal these tools in the Tools panel, click and hold the Pen tool in the Tools panel.
In most cases, the appropriate pen tool becomes active when you place the Pen tool pointer in a particular context. For example, the Delete Vertex tool becomes active when you place the Pen tool pointer over an existing vertex, and the Add Vertex tool becomes active when you place the Pen tool pointer over a path segment. To manually activate and cycle through these tools, press G.
When modifying a path, make sure that you click only existing vertices or segments; otherwise, you may create a new path instead.
.
to click
the segment between two existing vertices.
to click
the vertex.
.
To activate the Convert Vertex tool when the Pen tool is selected, hold down Alt (Windows) or Option (Mac OS).
The Adjust Tension pointer
appears
as you drag a vertex of the RotoBezier mask.Clicking a vertex instead of dragging sets the vertex to a corner point (100% tension); clicking again sets the vertex to a smooth point (33% tension). Dragging up or to the right decreases the tension of the selection, increasing the curve of adjacent path segments; dragging down or to the left increases the tension of the selection, decreasing the curve of adjacent path segments.
To view the tension value of a vertex, look in the Info panel as you adjust the tension.
To animate a path, After Effects designates the topmost vertex at the initial keyframe as the first vertex and numbers each successive vertex in ascending order from the first vertex. After Effects then assigns the same numbers to the corresponding vertices at all successive keyframes. After Effects interpolates the movement of each vertex from its initial position at one keyframe to the position of the correspondingly numbered vertex at the next keyframe. At any time during an animation, you can designate another vertex as the first vertex; this causes After Effects to renumber the vertices of the path. Renumbering vertices causes path animation to change, because After Effects then maps the new vertex numbers to the corresponding old vertex numbers still saved at successive keyframes.
When copying a closed path into a motion path, the vertex designated as the first vertex of the closed path is used as the beginning of the motion path. All motion paths are open paths.
Some shape path operations, such as Trim Paths, also use the first vertex as input to determine how to modify the path.
Note:
By default, when you add a vertex to a path, the new vertex appears on the path throughout the duration of the path but reshapes the path only at the time at which it was added. When you delete a vertex from a path at a specific point in time, the vertex is deleted from the path throughout the duration of the path. Prevent After Effects from adding and deleting vertices throughout the duration of the path by choosing Edit > Preferences > General (Windows) or After Effects > Preferences > General (Mac OS), and deselecting Preserve Constant Vertex Count When Editing Masks.
Note: Preserve Constant Vertex Count When Editing Masks is called "Preserve Constant Vertex and Feather Count when Editing Masks", in After Effects CS6.
Smart Mask Interpolation provides a high level of control for creating mask path keyframes and smooth, realistic animation. After you select the mask path keyframes to interpolate, Smart Mask Interpolation creates intermediate keyframes based on settings you provide. The Info panel displays the progress of the interpolation and the number of keyframes created.
Note:
To interrupt the interpolation process, press Esc. The Info panel indicates that the process has been interrupted and reports the number of keyframes created.
Keyframe Rate
Specifies the number of keyframes that Smart Mask Interpolation creates per second between the selected keyframes. For example, a value of 10 creates a new keyframe every 1/10 of a second. Choose Auto to set the keyframe rate equal to the composition frame rate, which appears in parentheses. Create more keyframes for smoother animation; create fewer keyframes to reduce render time.
Note:
Regardless of the keyframe rate you choose, Smart Mask Interpolation always adds keyframes at the frame just after the first mask path keyframe and at the frame just before the second mask path keyframe. For example, if you interpolate between keyframes at 0 seconds and 1 second in a 30-fps composition with a keyframe rate of 10 keyframes per second, mask path keyframes are added at frame numbers 1, 3, 6, 9, 12, 15, 18, 21, 24, 27, and 29.
Keyframe Fields
Doubles the keyframe rate. When this option is selected, and Keyframe Rate is set to the composition frame rate, a keyframe is added at each video field. Select this option for animated masking for interlaced video. If this option is not selected, the mask may slip off the object that you are attempting to key out. For more information about fields in interlaced video, see Interlaced video and separating fields.
Use Linear Vertex Paths
Specifies that vertices in the first keyframe move along a straight path to their corresponding vertices in the second keyframe. Leave this option unselected if you want some vertices to interpolate along curved paths; for example, when the desired interpolation involves rotating parts. If this option is not selected, Smart Mask Interpolation creates a natural path for the mask.
Bending Resistance
Specifies how susceptible the interpolated mask path is to bending instead of stretching. A value of 0 specifies that, as the mask path animates, it bends more than it stretches; a value of 100 specifies that the mask path stretches more than it bends.
Quality
Specifies how strictly Smart Mask Interpolation matches vertices from one keyframe to another. A value of 0 specifies that a particular vertex in the first keyframe matches only the same-numbered vertex in the second keyframe. For example, the tenth vertex in the first keyframe must match the tenth vertex in the second keyframe. A value of 100 means that a vertex in the first keyframe can potentially match any vertex in the second keyframe. Higher values usually yield better interpolations; however, the higher the value, the longer the processing time.
Add Mask Path Vertices
Specifies that Smart Mask Interpolation adds vertices to facilitate quality interpolations. In general, Smart Mask Interpolation works best when the mask paths have dense sets of vertices. Also, a vertex on the first mask path cannot match the middle of a curve or straight-line segment on the second mask path, so sometimes you must add vertices before matching to produce the desired result. Smart Mask Interpolation does not modify the original keyframes. Only the new mask path keyframes computed by Smart Mask Interpolation have additional vertices.
The value you set specifies how finely the input mask paths are subdivided. Pixels Between Vertices specifies the distance, in pixels, between vertices on the larger perimeter mask path after subdivision. Total Vertices specifies the number of vertices on the interpolated mask paths. Percentage Of Outline specifies that a vertex is added at each indicated percent of the mask path outline length. For example, a value of 5 means that a vertex is added at each successive segment of the outline that represents 5% of the total perimeter. To use only the vertices that were on the path at the first frame, do not select this option.
Note:
Smart Mask Interpolation may add vertices at existing vertex locations even if Add Mask Path Vertices is not selected. If two vertices on one mask path match a single vertex on the other, the single vertex is duplicated at the same location so that the segment between the two vertices shrinks to that location.
Matching Method
Specifies the algorithm that Smart Mask Interpolation uses to match vertices on one mask path to vertices on the other. Auto applies the matching algorithm for curves if either of the two selected keyframes has a curved segment; otherwise, it applies the polylines algorithm. Curve applies the algorithm for mask paths that have curved segments. Polyline applies the algorithm for mask paths that have only straight segments.
Note:
The mask path keyframes added by Smart Mask Interpolation are polylines when Polyline Matching Method is selected, regardless of whether the input mask paths contained curved segments.
Use 1:1 Vertex Matches
Specifies that Smart Mask Interpolation creates a vertex on one mask path that matches the same-numbered vertex on the other mask path. On each of the input mask paths, Smart Mask Interpolation matches the first vertices, the second vertices, the third vertices, and so forth. If the two paths have unequal numbers of vertices, this action may produce undesirable results.
First Vertices Match
Specifies that Smart Mask Interpolation matches the first vertices in the two mask path keyframes. If not selected, Smart Mask Interpolation searches for the best first-vertex match between the two input mask paths.
Note:
To ensure good results, make sure that the first vertices of the input mask paths match, and then select First Vertices Match.
Chris and Trish Meyer provide some tips on animating masks, including using Smart Mask Interpolation, on the ProVideo Coalition website.
You can adjust the area that is visible through a mask by either moving the mask in the Layer or Composition panel or panning (moving) the layer behind the mask in the Composition panel. When you move a mask, the Position values of the masked layer remain constant, and the mask moves in relation to other objects in the Composition panel.
When you use the Pan Behind (Anchor Point) tool to pan a layer behind a mask, the position of the mask remains constant in the Composition panel but changes in the Layer panel. The Position values of the masked layer change in relation to the composition. As you pan past the edges of the layer frame, the Mask Path values on the layer also change. Using the Pan Behind (Anchor Point) tool saves steps; without it, you would have to change the Position and Mask Path properties of the masked layer manually. You can animate a layer panning behind another layer by setting keyframes for the Position and Mask Path properties of the masked layer.
When you use the Pan Behind (Anchor Point) tool in the Composition panel, After Effects automatically makes two adjustments. In the Layer panel, the mask is moved in relation to its layer (top), and in the Composition panel, the layer is moved in relation to the composition (bottom).
Twitter™ and Facebook posts are not covered under the terms of Creative Commons.
