I'm going to show you how to use the Pen tool to create masks.
Here we have three comps in this project. These bowls, this thing which I call 'making waves', and this portrait. We're going to work with masks and all three of them. So back to the bowls. What I want to do here is I want to isolate the bowl from the background. So we're going to put a mask on it, and then display this little background behind it.
Just as a reminder: you make a shape of a Pen tool by clicking on the Pen tool, and not having a layer selected, and then clicking anywhere, and you start making a shape. When you're done you will close it off. There you go. We made a shape. It is a shape layer right there. It's not what we want to do here of course. So I'll click on that, and delete it.
What we wanted here is make a mask, I mean make a mask of the Pen tool on a layer. So I select this layer. And we're going to make a mask on here. I'll be working with RotoBezier, this little option here when we use the Pen tool. RotoBezier automatically creates curves when you have more than two points. Which is a nice thing, but it gives you a little bit less control over your work than making it without RotoBezier where you control the curves more exactly with the Pen tool. But we use RotoBezier here just kind of as a reminder.
And what I want to do is I want to click here around the edges here to just select the edges of the bowl. And we're going to stop short of doing the whole thing, because that'll take too long. And the way this thing works with RotoBezier is that you typically want to click at a high point of a curve, and a low point, and a high point. Like that. So we'll start down here at the low point right there, then click here. Now we've got a straight line. The third point will then turn those guys into a curve. Like that. And the curves are set automatically depending on how high up this thing is relative to these of the two points, and how far apart they are. I click here, another curve, down there, up here, down there. Up here.You're getting the sense of how this works, right? After there, down here. Here we go. We're kind of just making some progress here. And well, let's go up to the top there. And we'll call it a day.
I'm going to come back down here and close it off. And you can close it off by going back to the starting point. You can tell it's a starting point, because it's got that double box around it. When you hover over it you get that little circle next to your Pen tool. And now you click here, and close that off. There we go.
And now we can see the mask and how the mask then protects that area of that clip, and then reveals everything else behind it in the transparency behind it. And now I want to adjust these little points like that. And it's kind of hard to adjust them here, because we have nothing to look at relative to what's missing here. This is when you want to work inside the Layer panel. So let's double-click on this. And that opens up the Layer panel.
And we have Render turned off here. If Render were on, then you'd have the same kind of problem to deal with. So it's Render off. Now we can see the bowl relative to where we put those points there. As long as the points are not solid squares, we can work with them one point at a time.
I'm going to work with them with the Selection tool. I'm a little more comfortable working with them on the Selection tool than with a Pen tool, because there's only a couple things you want to do here. You want to move a point, perhaps we want to adjust the curve. So let's just take a look at what you got here. I think this curve could be made up a little better. So I'll put that one up there and wield just the left a little bit. So, you can drag these guys around like that.
Now, let's say I didn't see all these little empty squares. If it looks like this, if I click down here and click on the word for the mask layer, just click on that by itself, click away from that, now click on that. You get these squares here like that. And if I could drag them like now, they're going to move the whole thing around. That's the big difference. So, if you just click on the name of the layer, not on the mask, then you get those little points like that. That's why we're going to pull on here. And once you click on one, then you get the hollow squares. So we can adjust these guys accordingly.
And what I want to do is adjust the curve a little bit. And so if you go over to the Pen tool, you'll see this option here for Convert Vertex Tool, which is what you would use to adjust the curve here when you're working with a RotoBezier. But when you've got the Selection tool selected- there's a keyboard shortcut for that. If you want to adjust the curved nature of this thing right here, you hold down the Control Alt in Windows or Command Option on Mac. And then you get the Convert Vertex Tool there like that, when you've got the Selection tool active. Now if you click and drag, it then makes the curve. See how that works? It makes it get sharp or rounder like that. That's how you control the curve here.<
Now look at this thing. I think I could pull this a little bit further to the right. That would be a little more effective. I can pull this one to the right too, and make that a little more effective as well. Let's say though I want to add a point now. I don't have a keyboard shortcut here with my Selection tool to add a point. It's much easier to go to the Pen tool to do that. So I'm going to click on the Add Vertex Tool here. I just click on this line, I add a vertex. I can drag it wherever I want. Add another vertex here, and drag it down to here. So I can always add more vertices if you want to do that. Not a problem. All right.
So let's go on down to the next comp here, the making waves comp. I couldn't help but call it 'making waves'. The goal here is to highlight this little curve here, and then come back and highlight it coming back, and create this kind of wave inside here. Behind you we've got this little animated gray wave here, which you can see in the Comp panel. There you go.
There is a little animation back there. I'll turn this back on. The little issue here with this one though is that this top layer is actually animated left to right. So if I want to take my Pen tool and start selecting this thing along here, I have to click fairways, and then I have to pull it forward, and click again and pull it forward. Not very convenient. So it's easier to work inside the Layer panel.
Now, the Layer panel shows the entire width of that graphic instead of just the portion that's inside the frame in the Comp panel. So this is the whole thing rather than going back to the Comp panel, which is truncated, and have to go all the way to the right to see the rest of it like that. So let's go back to the Layer panel. This time let's dive in deep, take off our life vest, and use the Pen tool without using RotoBezier.
So select the Pen Tool, and turn off RotoBezier. The way Pen tool works is that you need to click and drag. If you just click and then drag, it's too late. You're just going to drag around the point. See the click? Hold down your cursor, and then drag to add handles. That's the process. And if you're going left to right here - like here -, you're going from this place to that place, you want to click and drag to the right.So we're going to click and drag at all these various high and low points here. They add those handles. The handles let us control the shape of the curve very exactly. Now if I get out here, if I just click, it's going to add a point without handles. And go down here and click - no handles there as well. As I go back here and I'm going to click and drag it. Now I want to click and drag to the left. If I click and drag to the right, I would create kind of a looping thing there. That just would look weird. So we're not going to do that.
So click and drag here to the left, the left. There you go. And you can follow any line you want to follow. I'll show you how this works. There we go. Going down the line here, and I'll click out here. No dragging necessary. And click on the end there to wrap this thing up. So there is our mask. If you want to see how it looks in action, go back to the compound. There we have it. It looks a little rough around the edges. Let me go back there and make some adjustments before we fix the rough edges.<
Let's say it's not exactly where I want it to be. I want to move some of these points. So I go get my Selection tool. Now as long as the name is selected here and not just the mask. If I click on the mask directly, then I get those little squares. And if I move them around, I move the entire mask. I don't want to do that. So I need to click away from the word Mask here, and just select the layer. And I get those little dots again. If I want to click on one of those dots, then you get that square, but the rest of these cuts are hollow. When you click on the dot you get the handles for that dot. And then you get one handle for each of the adjacent points.
So let me click someplace where it might be a little easier to see that. How about right there? There are those two handles right there. And yellow is not working here, because of the background. Let's change that color to something else. Change the color to green. There's no green in this picture, right. The green there. There you go. You can see that. One that's selected with its two handles, and then one handle for the one to the left, and one handle for the one to the right. If I want to control the shape of that curve, I can pull the handle around, and make it obviously horrible like that. Or I can bring it down, and not make it quite so horrible. Like that. That's how it works. And on the adjacent one I can adjust as well. Just that. It affects the curve going this way. So that's what those handles do.
If I want to add handles to this one, there are no handles there, and I do the old convert vertex tool thing- hold on Control Alt on Windows or Command option on Mac -,if I click on that guy, and then drag, and now it'll add handles to it. This guy has no handles either, just the one going out, because this one had a handle on it. So now I'm going to go Control Alt or Command Option,click on there, add handles there as well. So that's how you can convert the vertex.
If I want to break the handles, Control Alt or Command Option and break. Now I can let go of everything, and those guys are broken for good now. Unless I do the hold Control Alt or Command option thing again, and then we got those guys back, and they're not broken anymore. They now work together nicely like that.
So working with the Pen tool can be a little tricky, and takes a little bit to getting used to, but I think you can see that handles will help you make nice curves, and you can find two things as you go along like this ,and you can use the keyboard shortcuts. And if you don't really like the idea of using keyboard shortcuts you can always go back to the Pen tool, and just click to these guys, and add a vertex or delete a vertex or convert a vertex. You can just go to these guys one at a time.
All right. Let's go back to the comp side of this thing. Right now it looks a kind of jagged. That's caused by aliasing, because of the little square pixels that go on inside HD TV or even standard definition TV. So you want to smooth that out. So we're going to use the old Mask Feather to do that. Open up Masks. There's Mask Feather there. So let's feather it up; maybe 250 or something like that. It looks much nicer now.
Since we just feathered it we might want to expand a little bit. So drag it out like that a little bit, and kind of expand it. It's not critical of course what we're doing here, but nevertheless. Now if we had to play this it wouldn't go all that smoothly, because that background layer has got three effects applied to it, plus some motion as well. So we're just going to kind of drag our way through it here for the moment. Just to kind of go along with the theme of this kind of wavy thing I add an effect to it too.
So what's kind of nice is that you've taken the trouble to mask out this area here. Got it all masked out. It would be nice if we can apply effects to it while we're at it.So I've got this guy selected. And I go to the Effect Controls panel. And I added Wave Warp to it. If you want to mess with Wave Warp, be my guest. I've already created some settings there that kind of make it look sort of smooth as it goes from left to right there.
We'll do a little ramp preview here, but not too much, because this guy is working pretty hard to get this thing to work. But we'll give it a second or two, and then we'll let you play maybe a half a second of this thing so you can see how it looks. I think we're almost to half a second here. Let's just take this thing to the beginning to get a sense for how it works. There you go. It's the object of the game to get that little wave going there alone with that background.
Let me show you something else here. I want to show you a little problem I want to resolve here. We're going to use masks to resolve the problem. The Pen tool is the best way to make these masks, because they're not going to be in the standard shape like a square or an ellipse or something like that. But they're very easy to make. They don't take a lot of precision here. It just helped us solve this problem. And what I want to do is I want to get rid of the gray, so I can show something else in the background. And the way you get rid of a color frequently is to use a Key effect.
In this case we're going to use Keylight. And the Keylight is a super powerful, excellent keying effect that comes with After Effects called Keylight there. So when you use something like Keylight, you pick a color, in this case gray, and say make that transparent. Well, the trouble is there's gray here and gray is also throughout her face, around her eyes. So I turn this on. You're not going to like what you see. It got rid of all that gray. It turned to transparent. So you see the black background here. But oh my gosh! It just does horrible things to her face. And we just don't want to use this key on her face. We want to use it only there.So if I were to, let's say, take a Pen tool and try to select right along her hair there, it just wouldn't work, because the hair is too fine. So we did this really hard edge to it, now it looks just awful. So what I want to do is I want to just take the Pen tool, and create a shape here, a mask right there, and have that be the only thing visible, and apply the key to that, so then it affects this gray, and the rest of the stuff I don't want to have effect.
So let me show you how that works, because it's not quite that simple as I explained it. Now this first layer here has a mask on it. I'm going to turn the mask visibility off for the time being, just so you can see the border though. Turn on that layer. I've got an effect on there as well, turn the effect off. So here's this area that we've masked, the very easy mask. Look at how easy it was to make that. Nothing precise about it all. Just make sure we just got the hair covered, got the place where there's some holes that are covered. And we're going to just mask this area away from the rest of the image.
So I'll turn the mask on now so you can see that. That's just the part that we're going to work on. Then we apply the effect to that part. In this case Keylight. And we're not working with effects yet, so I took care of all the settings for you. Applying Keylight there then got rid of the gray; and the hair is in pretty good shape. It looks pretty good. It doesn't change much when we put that key on it, which is remarkable, right? Turn it off for a second; turn it back on. Now look to see how different it is. And basically it looks the same. You get to see through the spots there to what's below it. That's pretty great.
I'll turn that off for the moment here, and show you the next layer down. The next layer down is her again, but we need to just mask, we need to make sure that just this part of her is visible, and not this part over there. You can see I've got a mask here, but if I don't have the mask there, if I turn that off for the time being, you see that the gray would be there, because whatever's above her is transparent there.If I put this thing above her, this area is transparent above her. So if we have her down below there, then this will be visible. This gray is visible again; we've wasted all of our time. So if we add a mask to what's down below here, - if I turn that mask on now -, then this area has now been masked out of this layer right here. Turn that off. You can see that it's masked out now. This mask is visible, and the one above it. Let me click down here, so you see the mask this time here. This is the mask we applied to the one below there.
So this area here is no longer part of that second layer down here. We'll call this one the empty layer, but this one layer, two layers, this layer here has that mask now such that you don't see this area. And the one above it you see just this area with a little bit of overlap. And to make sure the overlap goes smoothly I did mask feather on both of those guys. So they overlap nice and smoothly.
So now I've got those two guys together. They went from this to this. And you know, look at this face. It's untouched, it's not changed. The highlights in her hair look the same. It's really remarkable. Now we can put any kind of background we want behind her, and it won't be affected in terms of how it touches her hair or whatever.
You can see right through her hair and see it.
So here we used Pen tools to create these very simple masks. And they solve the problem for us. They helped us remove that gray color without affecting the rest of the image. So that's how to use the Pen tool to create masks, and also a couple of different uses for those masks.
- In Adobe After Effects, select a layer and select the Pen tool.
- Click on key points along the shape that you want to mask. Close off the mask by clicking again on the start point.
- Adjust the mask point by using the Selection tool. Click and drag points to adjust them.
- Select the Convert Vertex Tool in the Pen tool set and click to adjust the curve around each point.
- Select the Add Vertex tool in the Pen tool set and click to add extra points.
Contributors: Infinite Skills, Jeff Sengstack