[Solved] Calculate Curvature and AO not working for Vertex Painting


Recommended Posts

  • Member

Hello,

I am trying to get conditions to work for vertex painting. However, whenever I calculate Curvature or AO, the maps don't come out as expected.

The AO is way to weak and the Curvature just stays black. Does it rely on UVs even though jus wanna I vertex paint?

I would really love to vertex paint in 3D Coat but it is a bit sobering without masking conditions. I guess it is supposed to work, as it asks me specifically to bake those maps when I select conditions (at the top , where it defaults to "Always") .

I am trying to get similar painting feel to "Automask" by Cavity in ZBrush and I am a bit surprised that it says you need to bake those maps, as with vertex painting, there are of course no UVs.

Would appreciate any help.

Link to comment
Share on other sites

Hello, I am testing 2022.25 but cant find any trouble baking AO or Curvature on vertex paint

Could you please enumerate step by step your current workflow ? Or share the model to take a look ?

Link to comment
Share on other sites

  • Member
Posted (edited)

Hi Carlosan,

Yes it appears that the new update v25 seemed to work and fixed the occlusion and curvature problems, at least for the few models I've tested.  Amazing work from the 3DCoat team! If I run into this on any models, I'll be sure to flag it here for awareness.

Thanks for everything you guys do!

-Alex

Edited by alexn007
Link to comment
Share on other sites

  • Member

Thanks for the replies guys!

@alexn007 Unfortunatley, updating to 2022.25 did not solve the problem for me.

@Carlosan Of course:

When using this model here that I exported via FBX export from ZBrush, when I bake AO its super gray'ish, not sure if it looks correct. When baking Curvature, the result is simply just black:

https://www.dropbox.com/s/bevx3aox6abh4fe/BabyET_2022_001_forCoat.fbx?dl=0
 

Here are the steps:

image.thumb.png.63879a51e539b9c2e0254d1d6fc2f25c.png

And this below is the result after the curvature baked, obviously not including any curvature data:

image.thumb.png.7e47fa8a21aa7c2086c8cbf754a10f40.png

The AO bake result also seems too subtle for me, not what I expect usually from Coat. I mean its always a bit grayish, but there seems to be missing punsh:

image.thumb.png.fecaec6e07d16361873d082766033833.png

Another idea: Could this have to do with scale discrepancies? I will maybe try to import into Blender first and then send via 3D Coat Appllink, will see if that makes any difference.

Maybe this is some incompatibility when directly importing FBX from ZBrush'S FBX exporter plugin.

I am supposed to bake via "Textures->Caclulate Curvature" and "Caclulate Occlusion", even for Vertex Paint, correct?

Link to comment
Share on other sites

  • Member

Ah one thing I forgot to add:

The model is already polypainted in ZBrush, so I bring those Vertex Colors in on import into Coat, which works great and it inserts it as new layer.

Not sure if that could maybe have anything to do with the problem.

Link to comment
Share on other sites

  • Member

And as mentioned, the problem is not existing when I use one of 3D Coats starter base meshes from the menu.

It seems internal data does not have this problem. Until now, I could not get any externally created sculpt to work with baked AO and Curvature.

Be it imported as FBX or OBJ. All behave the same for me.

Link to comment
Share on other sites

  • Member

Wow! Thanks a lot you two for looking into it! Curvature map looks good. Thanks @alexn007! Glad you like it.

I am a bit surprised though, as for me, when I import it over via "Startup Screen->Vertex Painting", the model scale relation to the grid is the exact same as in your second picture, where you scaled up by a 1000%.

When I import over menu "File -> Import -> Import for Sculpt/Vertex Painting" , my mesh seems still the appropriate size, at least the grid looks the same as in your second shot @Carlosan:

image.thumb.png.e87c8068cefd6a167227f6ec28f59963.png

Same when imported via Applink from Blender. I vaguely remember that I had the situation that the model was so small a few times in  between tries. Now I struggle to reproduce the model being so small on import. It seems that I may lack some basic understanding on how to appropriately import models?

However, I now went to sculpting room, and set my scene scale, which was at some number like 2226 or something, to "1" , and now I am able to bake again. But I can not for the life of me figure out anymore how you get the model to be so small on import :P. I guess that might be similar top what you did @Carlosan.  My Unit Scale in preferences is set to meter. I wonder why the scene scale gets off, though.

image.thumb.png.2def6686d1bb5358d1e32a6fd38288c1.png


P.S.: Would you guys know of any way to visualize these conditions? It seems you have to just try? Or can one preview them somehow inside the viewport prior to painting? The baked curvature does not get affected by it it seems:

image.thumb.png.510cff084c6e801beb189aec3b0c3e35.png

Link to comment
Share on other sites

  • Member

@Carlosan I know it is recommended now to use the RGB Curvature, right? But I wonder how you got that crisp greyscale curvature. I tried the several options: "Local Curvature" and "Light Based" Curvature, but those are not so high frequency as yours.

 

It looks like this for some reason with local curvature, which makes sense, as its the same as with local RGB jus without the colors.

image.thumb.png.63d622fe8528b3ce7b0981e4bd050de2.png

Looks completely different from yours though. Have you used some special settings, gamma correction maybe ?

 

image.jpeg

Link to comment
Share on other sites

  • Member

Any idea also maybe how I could use the conditions for fine grain masking? Like only painting the indents of pores?

image.thumb.png.793ac4323f6d5c23e5b34a6ee8eed94c.png

I have tried with various conditions and different "Degree", "Contrast" and "Cavity Width" settings. All giving me broader results without fine grain.

Does this have to do with the above problem that my map is not fine grained enough? Or even the sculpt itself...  I assume....

My Curvature Map seems to generally include these fine details though it seems, here is a screenshot from flat shaded view with active Cavity layer:

image.thumb.png.da84b8bbefa54adafd7deb22a48930fb.png

I also noticed that everything I paint comes out with sharp pixelated edges. Also a result of the curvature map not being fine enough? Or would there be a way to get smoother results ? :

 

image.thumb.png.e0ed05d14ff711bf4e7095ff5588520a.png

Link to comment
Share on other sites

  • Member
On 5/13/2022 at 11:12 PM, Carlosan said:

No, i use the default settings.


Try this, hope it help

 

Thank you! Does not really help unfortunatley, as I already know how to bake Cav and AO for Smart Material usage.

What I struggle with is getting decent results with them when Vertex Painting, using the "Conditions", to get something similar to what I have in ZBrush using various "Mask by" options.

 

Link to comment
Share on other sites

  • Member

Thx again @Carlosan

I am aware of the freezing being equivalent to mask in ZBrush.

But, can you also "freeze by Cavity" or "freeze by peaks and valleys" or "freeze by AO"? IYKWIM? 

For these, I thought I need to use the conditions menu "more an concave" and "more on convex", right? 

I thought 3D Coat works differently, by storing bakes into vertex color layers to drive that conditions, instead of using masking(freezing).

How does freezing come into play with this in 3D Coat? 

I would also love to know if we could visuslize the condition tweaks (cavity width, contrast) in the viewport, jus as we can visualize our "mask by" mask intensity levels in ZBrush.

I feel very comfi Polypainting in ZBrush, but the option to fully PBR paint on sculpts in 3D Coat is jus insane and tempting to work with it. I also like the renderer. The option to bake it down to several maps when done or transfer it to Blender nodes via applink is the sweet cherry on top.

Sadly currently, for me, I am not really getting it to function as I expected, probably I am doing something wrong. I I think it sould be possible to have a similar experience to ZBrush painting, if you know how.

Link to comment
Share on other sites

  • Member
1 hour ago, Carlosan said:

With Conditions you can using Fill tool and selecting Fill w/Freeze Mode

image.png

 

That's a great way to get the preview of the conditions!!! Great! thx ! Didn't think of that. You could also do it with a solid color instead of freeze and then simply undo, jus the way you do in ZBrush with settign up mask by options. Really like this system with the 3 parameters "Cavity width", "degree" and "contrast".

I meanwhile imported a model via file menu instead of the starting dialogue, not sure why that would give you different results for the import, but it somewhat does I think.

I would advice anyone who reads this against directly using FBX exported from ZBrush, as Carlosan said, their scale is super tiny. But this won't show when importing via start screen -> Vertex Paint. But only when you go over the menu it seems. I think that these are seemingly not delivering the same looks in regards of scaling, can potentially confuse users.

For me it worked when I put my model to a reasonable scale in Blender first, which most of the time scaling from "0.01" which is the scale coming from ZBrush FBX, to "1" in Blender, which gave me the measurments in meters the model was build with in mind (1,4m). When then bringing it ouf of Blender to 3D Coat via FBX, baking generally seems to work as expected. The AO looks terrific now:

image.thumb.png.941175df338b1398b33512cc2c534603.png

Even though I have some problems when baking the head, which I don't know where it could come from right now:
image.thumb.png.711593c2944a382413afbb0b36943d97.png

image.thumb.png.e35abdcbe149a2551b17c31f6f77173f.png

I think I will figure it out from here. Thanks much for the support! Eager and excited to do some full PBR vertex painting in 3D Coat in future! Amazing what this program offers

  • Thanks 1
Link to comment
Share on other sites

Sorry i forgot to mention

Setting up mask by options AO, cavity... all baked textures are stored in this location

SavedToDisk.jpg

You can use this maps (copy it to another layer if need to use later) as condition mask creating a new Smart Material (or duplicate and edit any old one if needed)

image.png

-----------------------------------------------------------------------------

Layer Blending let you to assign AO and CV as mask

image.png

Link to comment
Share on other sites

  • Member
7 hours ago, Carlosan said:

Sorry i forgot to mention

Setting up mask by options AO, cavity... all baked textures are stored in this location

SavedToDisk.jpg

You can use this maps (copy it to another layer if need to use later) as condition mask creating a new Smart Material (or duplicate and edit any old one if needed)

image.png

Ah thanks! Thats good to know. Makes sense for Smart Material usage. I mean an auto assignment of the maps, like in Substance Painter, which makes the maps available from within the project, would be even better.

I was specifically looking for painting though, not necessaraly applying Smart Materials. But now as you say it... ok, that is also super poweful. You can basically procedurally texture your sculpt also during vertex painting. Another incredible opportunity, I just realize...

  • Like 1
Link to comment
Share on other sites

  • Carlosan changed the title to [Solved] Calculate Curvature and AO not working for Vertex Painting

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share