Layer masking provides a non-destructive way to hide or show selected areas of a layer.
1. Let's start with an image. In this case I've chosen a violet on a contrasting background.
2. I add a layer mask by going to the Layers Pallette and clicking on the Add Layer Mask icon. This will add the layer mask and show a plain, white thumbnail next to your image thumbnail. (Note: Layer masks cannot be added to locked layers or to the "Background" layer. Notice in this example I've renamed my Background layer to unlock it. Rename a layer by double-clicking on the name.)
3. Alt-clicking or Option-clicking on the layer mask thumbnail will display it in your editing window. As you can see it is 100% white. Anything white on a layer mask will show the layer. In this case it's all white, so we can see the entire layer. Anything black on a layer mask will hide, or "mask", that part of the layer. Gray tones will partially mask the layer.
4. We could paint on the layer mask, either while it's visible, or while viewing the original image. Remember: painting black will hide, painting white will reveal. Since I want to mask just the flower, I need a black flower on a white background. I'll inspect the color channels to see if there's one that can give us a headstart. We're in luck, the green channel has the high contrast between the flower petals and the background that I'm looking for. I hit Ctrl-A or Command-A to select all, then Ctrl-C or Command-C to copy the green color channel. On low-contrast images I usually just paint directly on the mask, but any technique you know to isolate your masking subject will work for this.
5. Going back to my main image, I Alt-click or Option-click on the layer mask thumbnail to show it in the editing window. Then I press Ctrl-V or Command-V to paste my green channel information into the layer mask.
6. This has way too much information, plus I want good blacks and whites, not a lot of grays, so I use the Threshold command (Image>Adjustments>Threshold) to isolate just the flower.
7. I clean up the area I don't want masked by painting with white. I also go over the flower with black to give a good, solid mask. To provide a less severe mask around the edges, I also apply a small amount of gaussian blur.
8. Clicking on our main image thumbnail reveals the result of our efforts.
9. You can disable a layer mask at any time by shift-clicking on the layer mask thumbnail. Repeat to enable.
10. Here I've clicked on the layer mask icon and inverted (Ctrl-I or Command-I, or Image>Adjustments>Invert). This exchanges the black and white values, therefore reversing the layer mask.
11. Layer masking provides an excellent opportunity to experiment with your images, all the while remaining completely non-destructive and reversible. Here I've copied the mask we just made and applied it to a Hue/Saturation adjustment layer.