Roto Brush and Refine Matte

Separating a foreground object from a background is a crucial step in many visual effects and compositing workflows. When you’ve created a matte that isolates an object, you can replace the background, selectively apply effects to the foreground, and much more.

Use Roto Brush to draw strokes on representative areas of the foreground and background elements. After Effects uses that information to create a segmentation boundary between the foreground and background elements.

After creating the segmentation boundary, use Refine Matte properties to improve the matte.

 The Refine Matte effect is also available separately for the improvement of mattes created using features other than the Roto Brush tool.

Traditional rotoscoping uses animated masks to isolate an element in a video clip, which can be tedious and time-consuming. Roto Brush 2 uses artificial intelligence to track your subject’s movement so you can cut down on frame by frame refinement.

The default rotoscoping option is Roto Brush 2. If you prefer the older tool, you can select it by using xxx option in yyyy dialog.

If you prefer to stick to Roto Brush 1, then you ca

If you are familiar with the Roto Brush 1, selecting objects with Roto Brush 2's propagation engine is largely the same:

• The result of the propagation can be further refined and improved via the controls in the "Roto Brush Matte" group in the Roto Brush effect, especially with the "Reduce Chatter" property. 
• Additionally, the Refine Edge Tool can be used for highly-detailed edges like hair and further controlled in the "Refine Edge Matte" property group. There are also options that compensate for motion blur and decontaminating edge colors. It is useful when dealing with mattes with soft edges or fine details like hair.
• Use the Refine Edge tool on other frames until the refinement is as precise and complete as possible. Press Alt/Option to erase Refine Edge strokes.
  
• If you've used the Refine Edge brush, the Fine-tune Refine Edge Matte option in the Roto Brush & Refine Edge effect properties is selected. Modify properties in the Refine Edge Matte property group as needed. (See Roto Brush & Refine Edge, Refine Hard Matte, and Refine Soft Matte effect reference.)
  

Use the toggles at the bottom of the Layer panel to view the matte results, or open the Composition panel to view the result in-context with other layers. For more information, see Layer panel view options.

If you want more precise refining around the edges, you can use the Roto Brush 1, or combine it with the Roto Brush 2. The controls available to you in Roto Brush 1 by default can be activated in Roto Brush 2 by selecting the Enable Classic Controls option in the Preferences. The procedure to use it is largely the same as Roto Brush 2.

You can choose these view modes from the Show Channel menu in the Layer panel, by clicking the buttons in the Layer panel, or by using keyboard shortcuts. You can use the controls at the bottom of the Layer panel to change the color and opacity of the overlays used in Alpha Boundary and Alpha Overlay mode.

This effect is applied automatically when the first Roto Brush or Refine Edge stroke is drawn in the layer panel. Use this effect to control the settings for the Roto Brush & Refine Edge tools. After you have created a segmentation boundary and the boundary edges that need refining, use the Roto Brush Matte and Refine Edge Matte properties to improve the matte.

The Roto Brush effect properties in the Propagation property group affect segmentation between foreground and background and how that segmentation information is used for contiguous frames in a span. Other properties of the Roto Brush effect affect the matte that is generated based on the initial segmentation. The Refine Matte effect includes all of the properties of the Roto Brush effect outside of the Propagation property group (except for Invert Foreground/Background).

• Version: To select from the two Roto Brush versions - 1.0 (Classic) and 2.0.

• Quality: Select the quality of detailing around the edges. Standard is faster and doesn’t have quite as much detail around the edges. Best is refined around edges.
  

• Propagation: Properties in the Propagation property group (except for View Search Region) affect all Roto Brush calculations. Any change to these properties requires a recalculation and propagation of the segmentation information from a base frame. Also, the results on a base frame itself are not affected by changes to these properties; therefore, it’s best to change these properties when the current-time indicator is a frame or two away from a base frame, so that you can see the result of the changes.
  • Search Radius: The radius of the area within which After Effects searches when looking for pixels that match from one frame to the next. You can change how the search radius adapts to regions with more or less motion using the Motion Threshold and Motion Damping properties. If the search radius is too small, some motion may be missed; if the search radius is too large, extraneous motion may be detected.
  •  View Search Radius: When selected, displays a green box around the subject. It indicates where the Roto Brush algorithm is searching for the subject. If the subject moves out of that frame, the  Roto Brush effect is lost. You can increase the percentage to a higher number to cover the entire area. It is slower but more accurate and covers more area.
  • Enable Classic Controls: In Roto Brush 2, Propagation also has this checkbox. This is only available when Version is set to 2.0 and passes the propagation result from the new algorithm through the Classic one for finer control of propagation and more edge details. It turns the old Roto Brush options such as Motion Threshold, Edge Detection, and Motion Damping back on. In the version 1.0, these options are on by default but in version 2, you need to enable them manually. If you're not getting the desired line segmentation results with Standard or Best, try enabling classic controls. Doing so creates a hybrid approach, using the best of Roto Brush 2 and Roto Brush 1).
    • Motion Threshold and Motion Damping: (Available in Roto Brush 1 by default, and when you enable classic controls in Roto Brush 2). These two properties control how the search region is constrained based on motion. Change Motion Threshold to set the motion level below which is considered no motion, where the search region will shrink to nothing. Motion Damping affects the remaining areas that are considered to be in motion. As you increase Motion Damping, the search region is tightened, with slow-moving areas tightening more than fast-moving areas. Constraining the search region in areas with little motion can reduce edge chatter in these regions. Constraining the search region too much will cause the automatic boundary detection to fall off the edge of the object.
    • Edge Detection: Choose whether to favor the segmentation boundary calculated for the current frame in isolation or the segmentation calculated based on the previous frame when determining the edge between foreground and background. The Balanced option considers the current frame and surrounding frames equally. Foreground objects with colors that match the background will usually benefit from Favor Predicted Edges.
    • Use Alternate Color Estimation: Subtly changes the process by which the Roto Brush effect determines what is foreground and what is background. Sometimes checking it helps with segmentation; sometimes it doesn't.

• Invert Foreground/Background: Inverts which strokes are considered foreground strokes and which strokes are considered background strokes in the segmentation phase of the Roto Brush effect.
• Fine-tune Roto Brush Matte: Use this to refine edges, reduce chatter, shift edges, work on contrast, and feather the selection.
• Refine Edge Matte: 
  
  • Smooth: Increasing this value reduces the sharpness of the curves in the segmentation boundary by smoothing along the edge. Leave this number low when isolating a object with sharp features, such as hair.
  • Feather: Softness of the segmentation boundary. This property does nothing if Smooth is 0. In contrast to Smooth, Feather applies across the edge.
  • Choke: The amount of choking (contraction) of the matte relative to the value of the Smooth property. The result is very similar to that of the Choke property in the Matte Choker effect, but the value is given from -100% to 100% (instead of -127 to 127).
  • Reduce Chatter: Increase this property to reduce erratic changes to edges from one frame to the next. This property determines how much influence the current frame should have when performing a weighted average across adjacent frames to make the matte edges not move erratically from one frame to the next. If the Reduce Chatter value is high, the chatter reduction strong, and the current frame is considered less. If the Reduce Chatter value is low, the chatter reduction is weak, and the current frame is considered more. If the Reduce Chatter value is 0, only the current frame is considered for matte refinement. Tip: If the foreground object isn’t moving, but the matte edges are moving and changing, increase the value of the Reduce Chatter property. If the foreground object is moving, but the matte edge isn’t moving, decrease the value of the Reduce Chatter property.
• Use Motion Blur: Check this option to render the matte with motion blur. The high quality option is slower, but generates a cleaner edge. You can also control the number of samples and the shutter angle, which have the same meaning as they do in the context of motion blur in the composition settings. (See Motion blur.)
• Decontaminate Edge Colors: Check this option to decontaminate (clean) the color of edge pixels. The background color is removed from foreground pixels, which helps to fix halos and the contamination of motion-blurred foreground objects with background color. The strength of this cleaning is determined by Decontamination Amount (In Roto Brush 2, this is named as Amount).
  • Extend Where Smoothed: Only functional when Reduce Chatter is greater than 0 and Decontaminate Edge Colors is selected. Edges that are moved in order to reduce chatter are cleaned.
  • Increase Decontamination Radius (In Roto Brush 2, it is named as Increase Radius): Amount (in pixels) by which to increase the radius value for the cleaning of edge colors, in addition to any cleaning that covers feather, motion blur, and extended decontamination.
  • View Decontamination Map (In Roto Brush 2, it is named as View Map): Shows which pixels will be cleaned by decontamination of edge colors (white pixels in the map).

When the View menu in the Layer panel is set to Roto Brush, a Freeze button appears in the lower-right corner of the Layer panel. Click this button to cache and lock segmentation for all Roto Brush spans for the layer within the composition work area. This preserves the matte and saves it with the project, preventing the Roto Brush effect from recalculating the segmentation when you open the project again or make changes.

If After Effects has already calculated Roto Brush segmentation information for a frame when you click the Freeze button, then this information is cached. If the segmentation has not been calculated for a frame within a Roto Brush span, then After Effects must calculate the segmentation before freezing.

Frames with frozen (cached and locked) segmentation information are represented by blue bars in the Roto Brush 1 span view in the Layer panel (and purple bars in the Roto Brush 2 span view).

To unfreeze Roto Brush segmentation, click the Freeze button again.

When Roto Brush segmentation is frozen, you can place the pointer over the Freeze button to see a tooltip that tells you when the cached information was created.

When Roto Brush segmentation is frozen, the pointer for the Roto Brush tool has a slash through it.

The information that is cached and locked is the result of Roto Brush strokes and the properties in the Propagation property group of the Roto Brush effect. Making changes to any of these items (for example, by drawing new Roto Brush strokes or modifying properties in the Propagation property group) has no influence on the result of the Roto Brush effect until you unfreeze segmentation. The properties in the Matte property group are not frozen.

Frozen Roto Brush segmentation information is cached and locked while the application is running, and the cached information is saved with the project.

• When drawing strokes to define a foreground object with the Roto Brush tool, begin by drawing strokes along the center of the object’s features. For example, draw a stroke along the skeleton rather than along the outline of an arm. Unlike conventional rotoscoping, which requires precise manual definition of boundaries, using the Roto Brush tool works by defining representative regions. After Effects can then extrapolate from those regions to determine where the boundaries are. Before you draw a stroke along a boundary to attempt to get a precise segmentation, be sure that you've drawn foreground strokes down the center of the object and made at least some rough background strokes on the other side of the boundary.

• If you draw a Roto Brush stroke over the wrong area of the image, undo that stroke. (See Undo changes.) However, if After Effects misinterprets your stroke and includes or excludes too much of the image, don’t undo; further teach Roto Brush by drawing additional strokes to include or exclude regions.

• Work with resolution set to Full when using the Roto Brush tool. Fast Previews modes, such as Adaptive Resolution, don’t work well with the Roto Brush tool, because switching resolutions requires a full recalculation of the segmentation information. For this reason, Fast Previews modes are turned off when you draw a Roto Brush stroke. This setting is shared by the Composition and Layer panels. (See Resolution.)

• Use the Roto Brush tool in a composition with a frame rate set to match the frame rate of the layer's source footage item. A warning banner appears at the bottom of the frame in the Composition panel if the frame rate of the composition doesn't match the frame rate of the layer's source footage item. (See Frame rate.)

• When you've gotten everything as good as you can with the Roto Brush effect, you can touch up the matte further using other compositing features in After Effects—such as by painting on the alpha channel. (See Compositing and transparency overview and resources.)

• Choose your starting frame wisely. The more precise and stable the frame is, the better the propagation is.
• The Roto brush selection can only be made in the Layer panel so make sure you have your footage open there.
• Older projects that used Roto Brush should still open with Roto Brush 1. If you open a project created in versions before v17.5 that used the older Roto Brush & Refine Edge effect, the version is set to 1. So, old projects render the same even when opened in the new version.