Managing and animating shape paths and masks

You animate mask paths and shape paths in much the same way that you animate other properties: set keyframes for the Mask Path or Path property, set paths at each keyframe, and After Effects will interpolate between these specified values.

Rotoscoping (or just roto in casual usage) is the drawing or painting on frames of a movie, using visual elements in the movie as a reference. A common kind of rotoscoping is tracing a path around an object in a movie and using that path as a mask to separate the object from its background. This allows you to work with the object and the background separately, so you can do things like apply different effects to the object than to its background or replace the background.

Rotoscoping in After Effects is mostly a matter of drawing masks, animating the mask path, and then using these masks to define a matte. Many additional tasks and techniques make this job easier, such as using motion tracking on the object before you begin drawing masks, and then using the motion tracking data to make a mask or matte automatically follow the object.

• Immediately after beginning to draw a mask, press Alt+Shift+M (Windows) or Option+Shift+M (Mac OS) to turn on keyframing for that mask and set a keyframe. This way, you won’t edit a mask frame-by-frame for several minutes (or longer) and then realize that you lost all of your work on previous frames because you forgot to click the stopwatch to make the mask path animated.
• Draw your masks on a white solid layer with its Video (eyeball) switch off, above the (locked) footage layer. This way, you run no risk of accidentally moving the footage layer when you manipulate the mask, and you can also much more easily apply tracking data to the mask. (You apply the tracking data to the invisible solid layer that holds the mask.) This also means that you don't lose your cached preview frames each time you manipulate the mask. (See Toggle visibility or influence of a layer or property group and Lock or unlock a layer.)
• Turn on the Preserve Constant Vertex Count preference. (See Designate the first vertex for a Bezier path.)
• When possible, transform (rotate, scale, move) the whole mask or a subset of the mask vertices instead of moving the vertices individually. This is both for efficiency and to avoid the chatter that comes from inconsistent movement across frames. (See Move vertices in free-transform mode.)
• Manual motion tracking is less time-consuming than manual rotoscoping. The more effort you spend getting good tracking data for various parts of your scene and object, the less time you'll spend drawing and fine-tuning masks. (See Tracking and stabilizing motion.)
• Use the variable-width mask feather feature for more control when feathering objects.

This video from the After Effects CS5: Learn By Video series shows how to combine motion tracking and rotoscoping to isolate and selectively color-correct an actor's face.

Scott Squires provides a pair of movies on his Effects Corner website that show how to rotoscope, both painting and masking:

• Rotoscoping - Part 1

• Rotoscoping - Part 2

Mathias Möhl provides the KeyTweak script on his website, with which you can modify many keyframes on a property simultaneously. With KeyTweak, you can modify a few keyframes manually, and the script modifies the remaining keyframes in between accordingly. KeyTweak is especially useful for Mask Path keyframes in a rotoscoping workflow.

Rich Young provides several resources for rotoscoping on his After Effects Portal website.

• To view mask paths for selected layers in the Timeline panel, press M.
• To view selected masks or shapes in the Timeline panel, press SS (press the S key twice).
• To view mask and shape paths in the Composition panel, click the Toggle Mask And Shape Path Visibility button at the bottom of the Composition panel.
• To view mask paths in the Layer panel, choose Masks from the Layer panel View menu.
• To hide a mask path while showing others, lock the mask by selecting its Lock switch in the Timeline panel, and then choose Layer > Mask > Hide Locked Masks.
• To isolate selected masks and hide others, choose Layer > Mask > Lock Other Masks, and then choose Layer > Mask Hide Locked Masks.

You can select shape layers and their components at any of four levels of selection, referred to as selection modes:

For information on selecting masks, see Select masks, segments, and vertices.

• Click the layer name or layer duration bar in the Timeline panel.
• Using the Selection tool, click within the layer bounds in the Composition panel.
• To deselect all shapes on a layer but leave the shape layer selected, click within the layer bounds but outside all shape paths.

• Using the Selection tool, double-click a member of the group in the Composition panel. Each time that you double-click, you descend another level in the group hierarchy.
• To activate the Direct Selection tool , hold Ctrl (Windows) or Command (Mac OS) with the Selection tool selected. Click a shape in the Composition panel with the Direct Selection tool to directly select that shape’s group, regardless of how deeply nested the shape is in the group hierarchy.
• To select a group that is contained within the same group as the group that is already selected, click the group to select.
• To add a group to a selection, Shift-click it. You can combine the Shift key with double-clicking and with the Direct Selection tool to add more deeply nested groups to the selection.

• To select a vertex, click the vertex with the Selection tool. To add vertices to the selection, Shift-click them.
• To select a path segment, click the segment with the Selection tool. To add segments to the selection, Shift-click them.
• To select an entire path, Alt-click (Windows) or Option-click (Mac OS) a segment or vertex of the path with the Selection tool, or select any portion of the path and press Ctrl+A (Windows) or Command+A (Mac OS).
• To select vertices by dragging, select a path or portion of a path to enter path-editing mode, and then drag with the marquee-selection tool to draw a marquee-selection box around the vertices to select. To add vertices to the selection, hold down the Shift key as you draw additional marquee-selection boxes.

• Double-click a path segment while in path-editing mode or in group selection mode for a single shape.
• Select the Path property in the Timeline panel and press Ctrl+T (Windows) or Command+T (Mac OS).

Unlike layers, masks can have more than one level of selection. You can select a mask as a whole path, which is appropriate when you want to move or resize a mask. However, if you want to change the path of a mask, select one or more points on it. Selected points appear solid, and unselected points appear hollow.

• To select a vertex on a mask, click the vertex with the Selection tool . To add vertices to the selection, Shift-click them.
• To select a mask segment, click the segment with the Selection tool. To add segments to the selection, Shift-click them.
• To select an entire mask, Alt-click (Windows) or Option-click (Mac OS) a segment, vertex, or handle of a mask with the Selection tool, or select any portion of the mask and choose Edit > Select All or press Ctrl+A (Windows) or Command+A (Mac OS). To add masks to the selection, Alt+Shift-click (Windows) or Option+Shift-click (Mac OS) them.
• To select masks by dragging, select a mask or portion of a mask to enter mask editing mode and then drag with the Selection tool to draw a marquee-selection box completely around the vertices or masks that you want to select. To add masks or vertices to the selection, hold down the Shift key as you draw additional marquee-selection boxes.
• To select all masks on a layer, select a mask on the layer, and choose Edit > Select All or press Ctrl+A (Windows) or Command+A (Mac OS).
• To deselect all masks, press Ctrl+Shift+A (Windows) or Command+Shift+A (Mac OS).
• To select an adjacent mask on a layer, press Alt+accent grave (`) (Windows) or Option+accent grave (`) (Mac OS) to select the next mask, or Shift+Alt+accent grave (`) (Windows) or Shift+Option+accent grave (`) (Mac OS) to select the previous mask.
• To deselect a mask, click anywhere other than on the mask.
• To remove a vertex or segment from a selection, Shift-click the vertex or segment.

Locking a mask prevents you from selecting it in the Timeline, Composition, and Layer panels or setting it as a target in the Layer panel. Use this feature to avoid unwanted changes to the mask.

You can scale and rotate an entire mask or shape path (or selected vertices in one or more paths) using the Free Transform Points command. When you use this command, a free-transform bounding box surrounds the selected vertices, and an anchor point appears in the center of the bounding box to mark the anchor point for the current transformation. You can scale and rotate the selected vertices by dragging the bounding box or its handles. You can also change the reference point from which the vertices are rotated or scaled by moving the bounding box anchor point. The free-transform bounding box handles and anchor point exist independently of the handles and anchor point for the layer.

You can convert any manual Bezier mask path or manual Bezier shape path to a RotoBezier path. If the manual Bezier path has direction handles that have been adjusted, this conversion changes the shape of the path, because After Effects calculates the curvature of RotoBezier segments automatically.

The conversion of a RotoBezier path to a manual Bezier path does not change the shape of the path.

You can change a Bezier mask path using the Selection tool and pen tools in the Layer or Composition panel. You can change a Bezier shape path using the Selection tool and pen tools in the Composition panel.

The pen tools—Add Vertex, Delete Vertex, and Convert Vertex tools—are grouped with the Pen tool in the Tools panel. To reveal these tools in the Tools panel, click and hold the Pen tool in the Tools panel.

When modifying a path, make sure that you click only existing vertices or segments; otherwise, you may create a new path instead.

To animate a path, After Effects designates the topmost vertex at the initial keyframe as the first vertex and numbers each successive vertex in ascending order from the first vertex. After Effects then assigns the same numbers to the corresponding vertices at all successive keyframes. After Effects interpolates the movement of each vertex from its initial position at one keyframe to the position of the correspondingly numbered vertex at the next keyframe. At any time during an animation, you can designate another vertex as the first vertex; this causes After Effects to renumber the vertices of the path. Renumbering vertices causes path animation to change, because After Effects then maps the new vertex numbers to the corresponding old vertex numbers still saved at successive keyframes.

When copying a closed path into a motion path, the vertex designated as the first vertex of the closed path is used as the beginning of the motion path. All motion paths are open paths.

Some shape path operations, such as Trim Paths, also use the first vertex as input to determine how to modify the path.

Smart Mask Interpolation provides a high level of control for creating mask path keyframes and smooth, realistic animation. After you select the mask path keyframes to interpolate, Smart Mask Interpolation creates intermediate keyframes based on settings you provide. The Info panel displays the progress of the interpolation and the number of keyframes created.

You can adjust the area that is visible through a mask by either moving the mask in the Layer or Composition panel or panning (moving) the layer behind the mask in the Composition panel. When you move a mask, the Position values of the masked layer remain constant, and the mask moves in relation to other objects in the Composition panel.

When you use the Pan Behind (Anchor Point) tool to pan a layer behind a mask, the position of the mask remains constant in the Composition panel but changes in the Layer panel. The Position values of the masked layer change in relation to the composition. As you pan past the edges of the layer frame, the Mask Path values on the layer also change. Using the Pan Behind (Anchor Point) tool saves steps; without it, you would have to change the Position and Mask Path properties of the masked layer manually. You can animate a layer panning behind another layer by setting keyframes for the Position and Mask Path properties of the masked layer.

You can use the mask tracker to transform a mask so that it follows the motion of an object (or objects) in a movie. For more information, see Mask Tracking.

You can also use the mask tracker to accurately detect and track human faces. For more information, see Face Tracking.

You can use expressions to read and write the x and y coordinates of path points, or vertices for:

• Layer masks
• Bezier shapes   
• Brush strokes on the Paint and Roto Brush & Refine Edge effects.

The expression methods are similar to accessing path vertices via scripting. The expression method is named points() instead of vertices() for familiarity.

You can access the expression methods using the following steps:

Listed below are example expression methods of the following path objects:

• Layer mask paths: layer("Dark Gray Solid 1").mask("Mask 1").path or .maskPath
• Bezier shape paths: layer("Shape Layer 1").content("Shape 1").content("Path 1").path
• Paint effect stroke paths: layer("Dark Gray Solid 1").effect("Paint").stroke("Brush 1").path
• Roto Brush & Refine Edge effect stroke paths: layer("Dark Gray Solid 1").effect("Roto Brush & Refine Edge").stroke("Foreground 1").path

Tips to work with these expression methods:

• Point and tangent values are returned as an array of [x,y] number pair arrays for the coordinates of the vertices. Coordinate values are rounded to four decimal places.
• Point and tangent arrays start with the first vertex on the path. You can change the first vertex of layer mask paths and Bezier shape paths using the Layer > Mask and Shape Path > Set First Vertex command.
• Coordinates for layer mask path points are relative to the layer's origin in its upper-left hand corner.
• Coordinates for Bezier shape path points are are relative to the anchor point of the path's shape group (ex., "Transform: Shape 1 > Anchor Point").
• Coordinates for brush stroke path points are relative to the start of the stroke; the first point is [0,0].
• The toComp() method, or other layer space transform methods, can be useful to convert coordinates from different layers into a common space or simply return the coordinates relative to the composition space.
• Because Bezier shape paths are relative to the anchor point of their local shape group and not the layer, their values may be offset from where you expect them to be. To work around this, set all Transform properties in the shape group to zero. Keep in mind that converting a parametric shape to a Bezier shape may affect the shape group's position.
• The createPath() method can be passed points, tangents, and isClosed methods of the same path or other paths. You can pass these methods unchanged to duplicate a path, or transform the point and tangent values before passing them to modify or animate the path.

Create Nulls From Paths scriptUI panel allows you to create intuitive animations driven by expressions, without manually writing expressions. You can directly link different paths to shapes, shapes to solids, etc. The Create Nulls From Paths panel uses expression access to path points to automatically link the nulls, so that you don't have to write the expression yourself.

The panel creates nulls for each path point on a mask path or Bezier shape path. You can use the following buttons in the panel to direct how the nulls perform:

• Points Follow Nulls: The button creates nulls that control the position of path points. The path points are attached to nulls and you can animate the nulls to drive the path points.
• Nulls Follow Points: The button creates nulls that are controlled by the position of path points. The nulls are attached to path points and you can animate the path points to move the nulls.
• Trace Path: The button creates a single null that follows the path. The position of the null is linked to the coordinates of the path. The rotation of null auto-orients to follow the path. Trace Path controls the movement of the null along the path and whether it loops. By default, keyframes are set to trace the path in 1 second, and loop is enabled.