normal [Solved] The Normal Map Preset issue

[toomanydemons]

I baked some mesh maps in Substance Painter for an old project. It's an atlas of small items for detailing an environment (bottles, books, cans)

In Substance, the OpenGL maps bake perfectly. Here are screenshots from Substance Painter:

 

 

I decided that there were a lot of cereal boxes, logos and things I would need to paint in an external image editor, and I liked Coat's linking option (for Affinity). So I ported the project over to Coat.

I imported the normal map with the correct settings. The normal map doesn't show up correctly. I double-checked and toggled the green channel to be sure. The dark halos around the triangles are horrible in 3D Coat. Here's a screenshot:

 

 

So I tried re-importing the normal map and even creating a new project and changing the UV smoothing settings. To no avail.

Then I discovered the problem.

 

 

The software preset Blender and Unity yield different results, even though they should both be exactly the same (an OpenGL normal map with an inverted green channel). SO, I created my own normal map preset and called it OpenGL. I didn't do anything at all, I just made a new one.

Now everything shows up great! This is a screenshot from 3D Coat:

 

There should not be a Unity normal map preset vs a Blender normal map preset. They should not be different. There should only be an OpenGL and a DirectX setting for normal maps. Also, imported OpenGl maps using those presets show up wrong, and it will anger a lot of new users coming from other software. I'm also happy to share the model and normal map for anyone to verify my findings here. I'm using 4.8.32-SL(GL64)

 

 

 

Atlas_1_Bake.obj

[toomanydemons]

Edit: After some testing, I still can't get an imported normal map to show up correctly. there are always shading issues, however subtle. Also worth noting is that the normal map looks fine in every other software I try it in.

This is a serious show-stopper for me. I have been looking for a viable replacement since Adobe bought Allegorithmic. But there are normal map issues in 3DC that don't exist in any other software I port to. I can bring the normal maps into Unity, Blender, UE4, Substance (Painter & Designer), even sketchfab. 3D Coat is not reading normal maps correctly.

I wonder how long this has been an issue and no one noticed...

Edited February 7, 2019 by toomanydemons
More information acquired.

[toomanydemons]

Also, here is the high poly model if you would like to bake directly in 3D Coat. The normal map shows up fine in this case, but doesn't look as perfect in any of the other aforementioned softwares.

https://www.dropbox.com/s/rk4fj6stn0s43xo/Atlas 1_high.obj?dl=0

I hope these normal map issues get fixes asap. 3D Coat's front page looks juicy and may draw a large market, but normal maps have to be flawless for that to stick.

[digman]

This will help. Lock your normals in the import dialog box. I tested both Unity and Blender import settings.

Without locking your normals  3DC will recompute them. This is important when using an imported normal map. Not in all cases but this one for sure.

The only trouble I had were the rectangle boxes. That could be solved by unwrapping them a little different or adding edge loops to soften the 90 degree normals of the edges.

I generally do not have problems between 3DC, SP, Blender and Unity though I tend to add edges loops so the normal map does not have to try and completely fake the edge plus to help shading issues.

The above are suggestions only which made or made not work for your workflow.

[toomanydemons]

52 minutes ago, digman said:

This will help. Lock your normals in the import dialog box. I tested both Unity and Blender import settings.

Without locking your normals  3DC will recompute them. This is important when using an imported normal map. Not in all cases but this one for sure.

The only trouble I had were the rectangle boxes. That could be solved by unwrapping them a little different or adding edge loops to soften the 90 degree normals of the edges.

I generally do not have problems between 3DC, SP, Blender and Unity though I tend to add edges loops so the normal map does not have to try and completely fake the edge plus to help shading issues.

The above are suggestions only which made or made not work for your workflow.

Locking normals also does not work. In fact, I've tried everything in every single menu that relates to importing or normals. The fact is, 3D Coat's viewport renderer simply does not see normals correctly.

Things in your screenshot may seem correct, but take a look over here. Here is your screenshot with an indicator: 

 

[toomanydemons]

Let me better show you. Here is a link to a gif, of me trying to fix the issue: 

https://gyazo.com/e90a0ab735832b55b22858159f314095

The triangles leave a dark halo no matter how perfect you may seemingly get. This is the reason no one has really noticed this error until me. Although if you are experienced in 3D Coat you may have noticed this whole time that your normal maps seem a little off in other software.

So, this is what your box objects look like:

 

You see the problem?

 

 

Edited February 8, 2019 by toomanydemons

[toomanydemons]

In all other software, those surfaces are perfectly flat:

https://gyazo.com/afcb33cb529cbc8ac887d3ec1e330462

 

https://gyazo.com/93fd4f7a6f01a5f7e008e0b64825caf3

 

This sort of error is hard to spot, especially on organic models like most 3DC users would be accustomed to creating. Factor in that most users only use 3DC for a few of its rooms in their workflow, it may be that fully half or more don't even paint in 3DC. So you can see how this sort of error can escape detection for years.

Edited February 8, 2019 by toomanydemons

[toomanydemons]

Now, if you bake in 3D Coat, your objects will look perfect. However, the exported normal map will seem a little off in other software.

In a nutshell:

 

When I bake normal maps in 3D Coat, they look a little off in other software, even though they look fine in 3D Coat.

When I import normal maps into 3D Coat, they look a little off, even though they look fine in other software.

 

The lack of continuity here cannot be fixed by any option from any menu from my thorough (thorough...) examination. I spent nearly a whole day trying to figure this out. And even if I did find some sort of workaround, software shouldn't work that way. It should work as intended and not rely on any sort of hacky fixes or workarounds that would snag newer users and force a google search. If the most obvious path isn't default then there's a problem with the software.  If "lock normals" did in fact make a difference, then it should've been checked by default since this workflow is common.

Edited February 8, 2019 by toomanydemons

[Carlosan]

I am reporting this issue to support@3dcoat.com

[Ascensi]

8 minutes ago, Carlosan said:

I am reporting this issue to support@3dcoat.com

I'd also like to request a C4D preset for normals that can be selected in a new PPP project.  I would of created it myself If I knew the operations of C4D. I don't have it but I work with  AAA content/textures created by it and would like absolute compatibility with the libraries I have then export to other pipelines as needed.

[digman]

As stated I already mentioned the rectangle boxes and their problems. I gave my solution for the rectangle boxes. Also mentioned that how I work might not be what you needed. 

I did not fix the rectangle boxes as a normal map was already created for them. 

I hope you find the solutions you need...  

[toomanydemons]

19 minutes ago, digman said:

As stated I already mentioned the rectangle boxes and their problems. I gave my solution for the rectangle boxes. Also mentioned that how I work might not be what you needed. 

I did not fix the rectangle boxes as a normal map was already created for them. 

I hope you find the solutions you need...  

Thanks  I hope it gets solved too.

As for the rectangle boxes, I should not have to do anything to make those work. I should not have to add edge loops or unwrap differently. The main reason is that those normal maps look great in Substance Painter, Designer, Unity, Unreal, Sketchfab, Marmoset, and Blender. 

I shouldn't have to change the vertices and UV's of a model to make it work in 3D Coat. That is an obvious sign that things are broken. 

I've been keeping an eye on 3D Coat as a viable replacement for PBR painting. I hope that happens in the future  but in order for that to work, the normal map continuity between software must be addressed. It is extremely important, for 3D Coat as well (and everyone who tries the software in the future).

Given the tantalizing advertisement on 3D Coat's home page, it all looks very appetizing!! This normal map issue must be addressed ASAP, and put at the very top of the list, while people are still jumping ship from the Adobe acquisition of Allegorithmic. 

[Emi]

I am sorry to tell you, but the problem is your model, not 3DCoat. But Maya and Marmoset Toolbag present the same issue.

I will explain why the best I can, but it starts for the way you are exporting your model from Blender, but it will happen the same on any other software if you do the same.

Yes, normal map might fix a little the shading issues, for how the normal maps are read by software, and that might make the issue a little less too noticeable, but the issue it is still there, you are just ignoring the real issue and then it seems you aren't applying some of the rules that would help you to make easier the job and life easier by baking better normal maps.

 

By opening the model on maya, right away the model looks terrible, I noticed that most normals were average and few ones were hard, which was weird to see. But that's clearly the the main reason of your problems here, how you are exporting the smoothing groups for your model and how they are being read in other software. First, the boxes can't  be soft edges or average, they are 90° angles, and no matter what you do, they will always have this lighting issue if they are not set to hard edges. In the past, before more advanced baking programs like Marmoset appeared to make it easier,  you had to always make a cage, and the cage would be the one that had the average vertex normals, so the cage would be used to do the average 'searching' of the high poly, without causing errors on the baking since the box would remind hard edges. Of course now Marmoset has the checkbox "smooth cage" doing the same and now you don't need a cage unless you really need a cage.

 

So there are two 'rules' here and they are linked as well. 1. when you are doing models with 90° angles, they should be always hard. Of course you can also do a little beveling and avoid the 90° angle edges. this will not only follow the shape closely of the high res model, but also, you can use the soft edges on the box and make it look nicer and with less issues, and rounded and prettier and all that.

2. Hard edges (which pretty much should be 90° angles) should always be split on the UV to avoid issues as well, you might not really notice it on your model, but it is there, and it has to do with how the pixels meet at the 90° angle and how pixels might get blended and they are just different so there is not enough space like a padding would do to fix the small issues that might happen. Yes, it depends on the resolution and UVs and all that and that might make it more noticeable than others, but yes, in a case like this where you are doing boxes, you would have to split all those 90° edges, and yes, that means 6 box sides = 6 islands to avoid issues.

Talking about 3DCoat and how it works with presets and all that. What I understand about it is that by using the presets like Blender or Unity, you are actually using a predefined file with not only how it will treat the Normal map, which you are right, it will be Green flipped or not. But also Tangent Space, the triangulation, and how it exports the normals and some other options. If you use the Unknown option, you will be able to use the ones on preferences, the ones you set. 

If you go to C:\Program Files\3D-Coat-V4.8XXX\ToolsPresets\ExtTools  you will see the XML files and how Unity differs from Blender, because Blender preset uses MikkTSpace while unity is set to LengyelAreaAngleWeightedSpace. That is the only difference. but apparently enough for you to see it because they are being interpreted differently. Technically they shouldn't affect much anything and only the way it reads the normal map if it is 'DirectX' or' OpenGL',  but the root of the shading problem is not about the normal map, it is about the model and it will always be the model.

 

I hope I explained decently what the issues are here so it might help you on your future normal map bakings. Good luck and have a good day!

Edited February 9, 2019 by Emi

[toomanydemons]

20 minutes ago, Emi said:

I am sorry to tell you, but the problem is your model, not 3DCoat. But Maya and Marmoset Toolbag present the same issue.

I will explain why the best I can, but it starts for the way you are exporting your model from Blender, but it will happen the same on any other software if you do the same.

Yes, normal map might fix a little the shading issues, for how the normal maps are read by software, and that might make the issue a little less too noticeable, but the issue it is still there, you are just ignoring the real issue and then it seems you aren't applying some of the rules that would help you to make easier the job and life easier by baking better normal maps.

 

By opening the model on maya, right away the model looks terrible, I noticed that most normals were average and few ones were hard, which was weird to see. But that's clearly the the main reason of your problems here, how you are exporting the smoothing groups for your model and how they are being read in other software. First, the boxes can't  be soft edges or average, they are 90° angles, and no matter what you do, they will always have this lighting issue if they are not set to hard edges. In the past, before more advanced baking programs like Marmoset appeared to make it easier,  you had to always make a cage, and the cage would be the one that had the average vertex normals, so the cage would be used to do the average 'searching' of the high poly, without causing errors on the baking since the box would remind hard edges. Of course now Marmoset has the checkbox "smooth cage" doing the same and now you don't need a cage unless you really need a cage.

 

So there are two 'rules' here and they are linked as well. 1. when you are doing models with 90° angles, they should be always hard. Of course you can also do a little beveling and avoid the 90° angle edges. this will not only follow the shape closely of the high res model, but also, you can use the soft edges on the box and make it look nicer and with less issues, and rounded and prettier and all that.

2. Hard edges (which pretty much should be 90° angles) should always be split on the UV to avoid issues as well, you might not really notice it on your model, but it is there, and it has to do with how the pixels meet at the 90° angle and how pixels might get blended and they are just different so there is not enough space like a padding would do to fix the small issues that might happen. Yes, it depends on the resolution and UVs and all that and that might make it more noticeable than others, but yes, in a case like this where you are doing boxes, you would have to split all those 90° edges, and yes, that means 6 box sides = 6 islands to avoid issues.

Talking about 3DCoat and how it works with presets and all that. What I understand about it is that by using the presets like Blender or Unity, you are actually using a predefined file with not only how it will treat the Normal map, which you are right, it will be Green flipped or not. But also Tangent Space, the triangulation, and how it exports the normals and some other options. If you use the Unknown option, you will be able to use the ones on preferences, the ones you set. 

If you go to C:\Program Files\3D-Coat-V4.8XXX\ToolsPresets\ExtTools  you will see the XML files and how Unity differs from Blender, because Blender preset uses MikkTSpace while unity is set to LengyelAreaAngleWeightedSpace. That is the only difference. but apparently enough for you to see it because they are being interpreted differently. Technically they shouldn't affect much anything and only the way it reads the normal map if it is 'DirectX' or' OpenGL',  but the root of the shading problem is not about the normal map, it is about the model and it will always be the model.

 

I hope I explained decently what the issues are here so it might help you on your future normal map bakings. Good luck and have a good day!

There are no hard edges on the model. I'd be happy to jump on call with you personally and explore the situation. (https://gyazo.com/3386ce721adff5f6ead5849bdbed9373)

According to its documentation and settings explained by Allegorithmic staff, Unity uses MikkTSpace.

I have never had a problem with hard angles on a model if the cage is constructed correctly. Also, I have built at least a thousand hard surface models. (https://sketchfab.com/toomanydemons) (https://www.artstation.com/toomanydemons)

My dicord handle is TooManyDemons #2125. Cheers!

All of that being said, I have run some very recent test bakes using just cubes on very new geometry, considering the fact that those previous models are rebuilt from ones downloaded on Blendswap. I still ran into issues.

Additionally, those issues do not exist in Substance or Sketchfab, sorry.

 

 

 

Edited February 9, 2019 by toomanydemons

[toomanydemons]

All of that being said, I am aware of the hard angle rules for baking normal maps. Considering that these are super tiny low-poly cereal boxes, I didn't think it would be an issue. I will take your advice and investigate further. 

All in all however, I'm getting some pretty nice results everywhere else:

https://gyazo.com/184d39716173d4e68588122d26205d5c

Edited February 9, 2019 by toomanydemons