Thanks for testing, I've tried numerous texture, anything with a soft gradient result in a pretty noisy depth, even one of the default mask (mask sample 1 and 3 here)
I tried to use 16 bit mask but those don't work at all in 3d coat.
I will keep testing and if I find a solution will post, maybe there's a setting that increases sampling of masks somewhere?