You can paint transparency just by erasing on layer 0 or erasing on a layer and hiding layer 0. When you export the transparency map will be in the alpha of the exported Color texture.
When you hide Layer0 every color you paint in any of the layers is considered the transparency map.
Think opacity as RBGA color alpha value. To get opacity variation (add semi-transparent material, etc) you only need to paint with alpha value. Then when you hide layer 0 now you see semi transparent or full transparent with the layer color alpha value.
Or you can set layer opacity , eg if you set layer 1 opacity value as 0.3, then hide “layer 0” now you should see 0.3 alpha color of the layer. (semi transparent)
The reason why 3DC coat keep “layer 0” as visible: it make user easy to color (overwrite layer 0) mesh. (layer 0 keep alpha as 1.0 (no transparent) and color perfect 0.5 0.5 0.5 RGB). If there is not layer 0, you may not see any surface until you import map, or actually paint on surface in another layer. it make difficult to paint.
After paint or add smart material with alpha value, hide layer 0, then export color (diffuse/albedo) map as RGBA (png, tiff etc). Then the exported color map keep alpha value. Or you can choose option to export “Black and white opacity” map with RBG color map.
Use alpha value of color or export as opacity map (black and white) depend on which render application you may use.
If you can not get any transparent, even though you change layer 1 opacity as 0.5 etc, maybe you have another top layer which paint over. Or you may change “color blend “way (in layer panel , you can choose how color will blend with below layer. Default = Standard blend)
Also can add color with opacity (alpha) with use stencil freeze mask, or use Smart Material with add mask.
If it is only view render issue, you only need to set “backface Culling as OFF” and “Correct alpha ordering” in view menu. Like this:
Mask with layer
Dont forget about some interesting layer attributes such as “LOCK LAYER TRANSPARENCY” and “MASK WITH LAYER…” plus actions such as “FREEZE PAINTED PIXELS” or “FREEZE TRANSPARENT PIXELS” under Layers menu (you can tweak the brush transparency as well).
Example: first “ERASE” Layer zero, draw a few grass strokes (a grass material or maybe just a silhouette) and then “LOCK LAYER TRANSPARENCY”: now every other spec/color stroke (in the same layer) will only affect the pre-painted grass.
Plus you could use this layer as a mask for subsequent Layers. This way the “Grass Layer” will be the transparency layer for all the others. Don’t forget to play with “FREEZE PAINTED PIXELS” or “FREEZE TRANSPARENT PIXELS” (& remember to export as a psd!. Extract an alpha in photoshop or in Maya you just need to set the PSD file attribute “Use as alpha” to “transparency”)
The leaf was painted with only one stroke (notice that there is a MASK texture and material: height,color and roughness)
