Normal map baking issue

[MaxC]

Hello,

is there a way to fix this distorted normal map baking issue, without adding edge loops?
I'm baking with hard edges, so I don't understand why I am getting this issue.
Thanks!
-Max

[Carlosan]

Try changing this set please

 

[MaxC]

Thanks for your quick reply @carlosan

I just double-checked by baking with both "closest along normals" and "outer surface". they give the same result.
(I also tried reimporting the low-poly from maya, to make sure it doesn't have issues. same results.)
Any other suggestions?

[MaxC]

bringing this mesh into Marmoset Toolbag for baking, unchecking "smooth cage" fixes the issue.
is there an equivalent to this in 3d-coat?

[Emi]

You are getting this issue because that's how baking works, "Skew" is what is called by programs, and the only way to fix it is through the baker. It's not technically the baker's fault but that's why Bakers should have a Skew correction feature, to correct this problems when they arise, which is pretty common in situations like yours.

The reason you didn't get Skew when having more edges is because by having more polygons, the cage is going to have more vertices that would project better rays for thecage to catch the details better depending on the Shape. This has to do with the vertex normals of the cage, but the automatic cages these bakers create are going to use average vertex normals, so for example, in a box, where every corner is 90°, the average vertex normals would be 45° (which is the exact middle average between the 2 edges), so the baker will shoot the rays in a 45° catching everything from left to right, but of course the problem is that by being 45°, if you have details close to those edges, it will get more skewed because the ray that tries to catch the details would be inclinated 45° and not perpendicular to the face where those details are.

That's why in in Marmoset, you didn't get skew when you unchecked the smooth cage because the smooth cage is the cage with average vertex normals, and when you unchecked it became a cage with Hard vertex normals, that means that the rays of the baker would travel straight from the vertex normal creating something like an L if you see the right edge on a side view, but the problem with that is that if you have rounded corners or something like that, the hard vertex normals would not catch what is in the corner becuase the rays only traveled straight unlike when you use average vertex normals.

Marmoset has a feature where you can use smooth cage but paint the skew correction btw, so pretty much what it does is show you how the normals are shooting from the cage and you can paint where you want straight lines.

Which also will help you to understand how bakers work, yo could go to Maya or some software and look for the vertex normals and check yourself how the rays display depending how they are, Soft, Hard or Average. But an easier way is to use this skew correction inside Marmoset, all you need is to go to the Low mesh baking group, and click "Show Skew", it will show green lines that represent how the cage would be looking around for high details, and then if something looks off you can paint the skew and you will see in realtime how the rays go from green to red and they also go perpendicular to the face.

So yes, all bakers will have this same problem. Because the only way to make sure the cages catch all details is using average vertex normals, Marmoset of course is great and has realtime correction and all, you can even use the hard cage and fix it like it worked on your test. Substance Designer also has a skew correction but I dont know how that works if it's not something you can visualize in real time, 

About 3DCoat? I don't think 3DCoat has something like this, I checked and didn't find any settings for skew correction. You might be able to add edges and then delete them after the baking, as long as they are edges that don't change the UVs, so it's the same UVs but with fewer edges. But a direct way to correct the skewing? I didn't see anything for that. I would guess that more edges like in your first pic or Marmoset is your only option here if you want to avoid this. Unless I am wrong about 3DCoat and skew correction feature I might have missed. 

 

I hope you understand what I tried to say. It's not a complicated thing but it's hard to see it unless you see it, so I hope Marmoset and the Show Skew can make you see what I tried to say in all my paragraphs, but yeah, didn't see anything about fixing skew normals in 3DCoat. 

 

Edit: I added quick images in Marmoset to illustrate better the cages and and the difference between how the baker will look for details between average vs hard vertex normals. As you can see every "ray" is shooting straight to the face on the hard vertex normal one, adding a red square to one of the corners show you what would not get baked and why the average one is the best one that works in most cases, because it would nor miss anything when baking, but stuff will get skewed because as you can see that the center of the faces are the only ones with straight ones and then the others would average to meet the vertex normal of the corner, which is 45° in this case. So that's why the objects further for the center of the polygon would get more skewed and then happens what happened in your bakes when having fewer polygons.

 

Edited April 12, 2019 by Emi

[MaxC]

@emi

thank you so much for that explanation!  I better understand what is going on in the baking process now.
I see there is no easy solution for this issue. for now I will just bake with some edge loops.
Thanks again for the detailed explanation Emi!!

[Emi]

18 minutes ago, Max Cederroth said:

@emi

thank you so much for that explanation!  I better understand what is going on in the baking process now.
I see there is no easy solution for this issue. for now I will just bake with some edge loops.
Thanks again for the detailed explanation Emi!!

No problem! I am glad I can help.

Since 3DCoat doesn't allow custom cages or non-smooth cages like Marmoset does, the only workaround I found to fix this in 3DCoat is to explode the mesh or an easier way is to use the option that says "bake selected only". so you select each invidual faces, so in the the case of a cube, you can do 3 bakes, back, front and then sides if they dont have any detail, but you would have to export the normal map on each barking process to be merged in photoshop or whatever program you use, making sure everything is like it should be in a normal map, and then import them back to 3DCoat. With that bake selected only, if your issue is only that small portion of your baking, you could easily, bake everything normal and skewed, then do a baking with the faces that have really bad skewed problems, export the normal map, and replace the skewed objects in your map. So it can be done, but it is still a pain.

But as long as they are the same UVs, it will work fine, things will be baked in the same place, so the cube with everything welded can display what you baked separatedly no problems. The only thing that this changes is the way the cage is perform and that's what 3DCoat doesn't allow you to do today.

The easiest workaround in 3DCoat would be to do what I said, make the loops that you are planning to do anyway and then delete them and import the normal map and all that, if the loops don't change the UVs then it would work really good as well. You would give more vertex for the cage to work and catch details not so skewed. 

 

But to be honest, if you have access to Marmoset, why do you not use it? it's one of the best today for the many features and posibilities you have when baking, smooth cage or hard cage in a checkbox and skew correct without any magical power, be able to visualize how you are going to bake without the need of changing names and making everything perfect (names can be help with the loader but it doesn-t mean you are limited to them) and you just import it to 3DCoat and problem solved. 3DCoat is good for many things but the baker, like many other programs is not the best thing it has in my opinion, especially if your low mesh is a cube.