Photoshop layer effects

Harnessing the Power of Layer Masks

Advanced Photoshop Tutorials by Jay Kinghorn, April 2009 (updated August 2009)

Layer Masks | Smart Objects | Advanced Masking | Image Sharpening Burning and Dodging
Layer masks are one of Adobe Photoshop’s most powerful, yet least understood features. In this tutorial, I’ll demystify layer masks by highlighting the benefits of incorporating layer masks into your image correction workflow and demonstrating the use of these tools in common, real-world situations. This tutorial is designed for photographers who are comfortable working with layers in Photoshop and have at least a basic familiarity with Photoshop’s painting and drawing tools, including the Marquee selection tools, the Brush and Gradient tools.
You can use layer masks most effectively if you first understand what they are and what they do. Layer masks allow you to selectively hide, or mask, portions of a layer while allowing other portions of the layer to show through. For example, you may use a mask to make the background of an image transparent. This allows a designer to combine the photo with text or graphics in a page layout program.

Jay Kinghorn

Jay Kinghorn

More commonly, photographers use layer masks for applying changes to specific portions of an image without affecting the rest of the photo. Similar to burning and dodging in the wet darkroom, these changes give you tremendous control over your photos. These selective corrections can bring a photo alive, add depth, improve contrast and shape your photo in subtle, yet powerful ways.
Photographers also use layer masks to composite multiple images for an entirely new creation, or, blend multiple exposures taken of the same scene to expand dynamic range, blend multiple white balances, or extend the depth of field.
Understanding how to use layer masks is one of the keys to unlocking the true potential of Photoshop. Once you’ve mastered the use of Layer Masks, the doors to Photoshop are wide open for you to explore.

What are Layer Masks?

Layer masks are key to a versatile, non-destructive workflow. As mentioned earlier, a layer mask is used to mask, or hide, the contents of a layer. A mask can hide the corrections on an adjustment layer, or can hide pixels on a pixel layer. It is important to remember that the masked pixels aren’t eliminated, they are simply hidden. This allows you to refine your layer mask an infinite number of times without damaging your image.

I. Masks vs. Selections

Many photographers learn to adjust specific areas of a photo by creating selections with the selection tools such as the Marquee tool, Magic Wand or Lasso. Unfortunately, selections are difficult to create, difficult to adjust and even harder to go back and refine later. Layer masks are superior to selections in virtually every way.
Fortunately, the tools you’re accustomed to use for making selections form the basis for building layer masks. In fact, selections and layer masks are essentially two sides of the same coin. A selection can be thought of as a temporary mask, a mask can be thought of as a permanent selection. To clarify this point, let’s look at an example.

If you’d like to follow along, click on the flower.jpg image link and right click on the image to save to your desktop. Open the image in Photoshop.

I: Step 1

Step 1

Using the Rectangular Marquee tool, create a selection in the center of the image. From the Image > Adjustments menu, select Hue-Saturation, then decrease the Saturation amount to -100. This removes color from the selected area, the center of the rectangle.
In this example, our change, removing color from the photo, is applied to the selected area but not applied to the deselected area. The same principle applies when working with layer masks, only instead of thinking about selected and deselected areas of the photo, you’ll want to think in terms of masked or unmasked areas.
Undo the last two steps (Edit > Undo or use the history panel) to just before you removed color from the photo. Your selection marquee should still be active. If not, go ahead and draw a new rectangle with the Rectangular Marquee tool.

I: Step 2

Step 2

Now, instead of going to the Image > Adjustment menu for your Hue-Saturation adjustment, create a new Hue-Saturation adjustment layer using the Create New Fill or Adjustment Layer at the base of the Layers panel.

I: Step 3

Step 3

Decrease the Saturation amount to -100, removing the color from the area inside the rectangle. Visually, the effect is identical to the previous method, only this time, your initial selection was transformed into a layer mask as soon as you created the adjustment layer.
If you look in your Layers panel, you’ll see your Hue-Saturation layer now has a layer mask associated with it. A small thumbnail of the mask is visible next to the Hue-Saturation icon.
The white areas of the mask correspond to the area inside the selection. The black areas correspond to the deselected areas within the image. The white areas of the mask show the Hue-Saturation adjustment, while the black areas do not. The important rule to remember when working with layer masks is: “White reveals, and black conceals.”
The most important difference between selections and masks is corrections made with masks are not permanently applied to the image. Let’s say you wish to leave the color of the flower intact and desaturate the background. With selections, this involves undoing several steps and starting over. With masks, we can use our existing mask as a starting point and begin refining the mask.

I: Step 4

Step 4

Press CMD-I (mac) CTRL-I (win) to invert the layer mask. Now, instead of a full-color image with a desaturated rectangle in the center, you should have a desaturated background with a full-color center.

Step 5

Select your Brush tool (B) and, using the Brush Preset Picker in the Options Bar, set your Master Diameter to 35 and your Hardness to 20%. Press “D” to set your default foreground and background colors (white and black) then press “X” to reverse the foreground and background colors. To restore the color to the flower, you will need to paint over the flower with black to hide the Hue-Saturation correction. Begin painting over the flower to return the color to the flower.
Tip: If you encounter any difficulties in this step:

• Be sure your Hue-Saturation layer is targeted (highlighted in blue) in the Layers Panel.
• Make sure your Brush Opacity is set to 100% in the Options Bar
• Make sure your Brush’s blending mode is set to Normal in the Options Bar.

If you paint over the background, simply press “X” to switch the foreground and background colors and paint over the area a second time. Press “X” again to return to painting with black to add to the mask. As you get closer to the edges of the flower, decrease your brush size by using the left bracket “[” key. Increase the brush size by pressing the right bracket “]” key.
While this is a lot less tedious than trying to select the flower with the lasso tool, masking with a mouse can be difficult. If you find yourself regularly creating masks, consider investing in a graphics tablet like the Wacom Intuos series. It makes masking significantly easier and more precise than working with a mouse.
Below is the finished flower (left) and the final mask (right). Take note of the white and black regions in the mask. The white areas reveal the Hue-Saturation change (the color removal) and the black areas hide the change, returning the color to the flower.

Finished Flower

Final Mask

Saving this as a layered TIFF or PSD document will retain the layer mask allowing you to edit, alter or remove the mask at a later date. This gives you full flexibility over the corrections you make to your image.

II. Using Layer Masks

While the above example is useful for demonstrating the techniques used for layer masking, most of your layer masks will be used to apply subtle changes to your photos. In this example, I’ll walk you through the process of selectively adjusting the contrast and brightness in a photo using adjustment layers and layer masks.

II. Climber

In the Climber image, we face a common problem. The separation in the shadows in the trees is insufficient to maintain shadow detail in a fine-art print on cotton-rag or similar paper. Because these fine-art papers are highly absorbent, the heavy shadows often become muddied and indistinct. This can be remedied by is to adding additional contrast to the shadows to ensure your shadow detail reproduces cleanly. Using layer masks in conjunction with a Curves adjustment layer is a perfect way to open up the shadows without impacting the rest of the image. Not only is the technique quick, but it is very effective and serves as a perfect example of the types of corrections you can perform with layer masks and adjustment layers.

To follow along, click on the climber.jpg image link and right click on the image to save to your desktop. Open the image in Photoshop.

II. Step 1

Step 1

Begin by creating a new Curves adjustment layer. Bend the shadow half of the curve up and left to apply a steep contrast correction to the shadows. This temporarily makes the midtone and highlight portions of the photo too bright, but you’ll remedy this momentarily.

Step 2

In Photoshop CS3 and earlier, press OK to exit the Curves dialog box. Press CMD-I (Mac), CTRL-I (Win) to invert the layer mask from white to black. This hides the curves correction from the entire image.

II: Step 3

Step 3

Select your Brush tool and select a brush approximately 20 pixels in diameter with a low hardness setting and a brush opacity of 50 percent. When brushing in corrections, it helps to have a soft brush with a medium-opacity setting to help disguise your corrections.

II: Step 4

Step 4

Press “D” to set your default foreground and background colors (white and black, respectively) and begin painting over the pine trees in the foreground. This lightens and adds contrast to the trees creating visual separation from the foreground trees and the pine trees on the distant slope. You may wish to zoom into the image to make the painting process easier. If your brush strokes are visible, or if you’ve painted in the wrong location, press “X” to reverse the foreground and background colors and paint with black over the error. If your correction is too strong, decrease the layer opacity to reduce the overall intensity of the Curves correction.
Adjustment layers allow you to quickly apply any type of correction (Hue-Saturation, Contrast, Brightness, Vibrance, Selective Color etc.) to specific areas within a photo. From a workflow perspective, it is best to perform your global corrections first (overall color balance, brightness, etc), before tackling specific portions of your image.

III. Masking Pixels Instead of Corrections

Layer masks aren’t exclusively used for masking adjustment layers. When compositing multiple images, you will use a mask to make portions of an image transparent, allowing the background layers to show through. Although the general techniques are exactly the same as those presented above, there are a few key differences. We’ll address these differences while blending two images to balance the brightness of foreground and background elements.

Step 1

The Blending image illustrates a common problem when photographing outdoors—the camera records the scene very differently than our eyes see it. Regardless of your exposure settings, if the sky is correctly exposed, the foreground will be too dark. If the foreground is exposed correctly, the sky becomes too bright. One solution is to export two versions of the same camera raw file, each balanced for a specific portion of the image. In this case, I’ve exported one version for the foreground and a second for the sky. Your job is to blend the two layers to create a balanced composition.

If you’d like to follow along, click on the blending.psd image link and save to your desktop (it’s a psd file). Open the image in Photoshop.

III: Step 1 (a)

III: Step 1 (b)

Tip: When performing this technique, it is often preferable to export both versions of the file as Smart Objects, which allow you to return to the raw file to make slight adjustments to fine-tune the balance of each file.

III: Step 2

Step 2

Unlike adjustment layers, pixel layers do not automatically contain a layer mask. Add a layer mask to the light layer by targeting the Light layer in the Layers panel then, clicking on the add Layer Mask icon at the bottom of the Layers panel.

III: Step 3

Step 3

Once you’ve added the layer mask, take notice of the small black lines around the corner of the mask. These are your only indication you are painting on the mask instead of the layer itself. As the complexity of your composites grows, it is easy to accidentally click on the layer thumbnail causing you to paint on the layer instead of the mask. Should this happen, undo your last brush stroke CMD-Z (Mac), CTRL-Z (Win) and single click on the mask thumbnail.
When masking a pixel layer, black hides the pixels on a layer, making them transparent. White areas of the mask preserve the pixel’s opacity. In this case, you want to preserve the opacity of the foreground pixels and make the sky transparent allowing the darker sky to show through. You can accomplish this manually with the brush tool, or, since this is a large area, you use the gradient tool for quicker results.

III: Step 4

Step 4

Press the “D” key to your ensure your foreground and background colors are at their defaults, then select the Gradient tool from the toolbar (G). Be sure the linear gradient option is selected in the Options bar and you’ve selected the Foreground to Background gradient from the Gradient Picker.

III: Step 5

Step 5

Create your gradient by clicking and dragging the gradient vertically across the horizon line. For this image, drag the gradient from the back of the semi truck to the beginning of the deep blue in the sky. This hides the transition between the foreground and background images to occur along the natural transitions in the horizon line.
If you turn off the layer visibility of the dark layer, you can see how the gradient mask preserves the opacity of the foreground pixels, then makes a smooth transition to transparency allowing the pixels of the dark layer to show through in the sky.
The end result is a seamless composition of the two images.

III: Blending Final Image