Jump to content
3DCoat Forums

[Solved] The Normal Map Preset issue


toomanydemons
 Share

Recommended Posts

  • Member

I baked some mesh maps in Substance Painter for an old project. It's an atlas of small items for detailing an environment (bottles, books, cans)

In Substance, the OpenGL maps bake perfectly. Here are screenshots from Substance Painter:

 

85a98b72bd4ac61a66d5807d652045e7-png.jpg

 

I decided that there were a lot of cereal boxes, logos and things I would need to paint in an external image editor, and I liked Coat's linking option (for Affinity). So I ported the project over to Coat.

I imported the normal map with the correct settings. The normal map doesn't show up correctly. I double-checked and toggled the green channel to be sure. The dark halos around the triangles are horrible in 3D Coat. Here's a screenshot:

 

365516fe8648d1766f7f3341c543535f-jpg.jpg

 

So I tried re-importing the normal map and even creating a new project and changing the UV smoothing settings. To no avail.

Then I discovered the problem.

 

bc9d5f7b7b028981f7bb3a7cf160456b-png.jpg

 

The software preset Blender and Unity yield different results, even though they should both be exactly the same (an OpenGL normal map with an inverted green channel). SO, I created my own normal map preset and called it OpenGL. I didn't do anything at all, I just made a new one.

Now everything shows up great! This is a screenshot from 3D Coat:

eebfd6ffc693d91070d7a926e47209b4-jpg.jpg

 

There should not be a Unity normal map preset vs a Blender normal map preset. They should not be different. There should only be an OpenGL and a DirectX setting for normal maps. Also, imported OpenGl maps using those presets show up wrong, and it will anger a lot of new users coming from other software. I'm also happy to share the model and normal map for anyone to verify my findings here. I'm using 4.8.32-SL(GL64)

 

 

 

Atlas_1_Bake.obj

MAT_RandomProps_Atlas_1_normal_base.png

Link to comment
Share on other sites

  • Member

Edit: After some testing, I still can't get an imported normal map to show up correctly. there are always shading issues, however subtle. Also worth noting is that the normal map looks fine in every other software I try it in.

This is a serious show-stopper for me. I have been looking for a viable replacement since Adobe bought Allegorithmic. But there are normal map issues in 3DC that don't exist in any other software I port to. I can bring the normal maps into Unity, Blender, UE4, Substance (Painter & Designer), even sketchfab. 3D Coat is not reading normal maps correctly.

I wonder how long this has been an issue and no one noticed...

Edited by toomanydemons
More information acquired.
Link to comment
Share on other sites

  • Member

Also, here is the high poly model if you would like to bake directly in 3D Coat. The normal map shows up fine in this case, but doesn't look as perfect in any of the other aforementioned softwares.

https://www.dropbox.com/s/rk4fj6stn0s43xo/Atlas 1_high.obj?dl=0

I hope these normal map issues get fixes asap. 3D Coat's front page looks juicy and may draw a large market, but normal maps have to be flawless for that to stick.

Link to comment
Share on other sites

  • Reputable Contributor

This will help. Lock your normals in the import dialog box. I tested both Unity and Blender import settings.

Without locking your normals  3DC will recompute them. This is important when using an imported normal map. Not in all cases but this one for sure.

The only trouble I had were the rectangle boxes. That could be solved by unwrapping them a little different or adding edge loops to soften the 90 degree normals of the edges.

I generally do not have problems between 3DC, SP, Blender and Unity though I tend to add edges loops so the normal map does not have to try and completely fake the edge plus to help shading issues.

The above are suggestions only which made or made not work for your workflow.

unity.jpg

  • Like 1
Link to comment
Share on other sites

  • Member
52 minutes ago, digman said:

This will help. Lock your normals in the import dialog box. I tested both Unity and Blender import settings.

Without locking your normals  3DC will recompute them. This is important when using an imported normal map. Not in all cases but this one for sure.

The only trouble I had were the rectangle boxes. That could be solved by unwrapping them a little different or adding edge loops to soften the 90 degree normals of the edges.

I generally do not have problems between 3DC, SP, Blender and Unity though I tend to add edges loops so the normal map does not have to try and completely fake the edge plus to help shading issues.

The above are suggestions only which made or made not work for your workflow.

unity.jpg

Locking normals also does not work. In fact, I've tried everything in every single menu that relates to importing or normals. The fact is, 3D Coat's viewport renderer simply does not see normals correctly.

Things in your screenshot may seem correct, but take a look over here. Here is your screenshot with an indicator: 

 

unity.thumb.jpg.6135cf85267eb09678557ee528f15b27.png

Link to comment
Share on other sites

  • Member

Let me better show you. Here is a link to a gif, of me trying to fix the issue: 

https://gyazo.com/e90a0ab735832b55b22858159f314095

The triangles leave a dark halo no matter how perfect you may seemingly get. This is the reason no one has really noticed this error until me. Although if you are experienced in 3D Coat you may have noticed this whole time that your normal maps seem a little off in other software.

So, this is what your box objects look like:

f748c251b484042113c42e680df5c291-jpg.jpg

 

You see the problem?

 

 

Edited by toomanydemons
Link to comment
Share on other sites

  • Member

In all other software, those surfaces are perfectly flat:

https://gyazo.com/afcb33cb529cbc8ac887d3ec1e330462

 

https://gyazo.com/93fd4f7a6f01a5f7e008e0b64825caf3

 

This sort of error is hard to spot, especially on organic models like most 3DC users would be accustomed to creating. Factor in that most users only use 3DC for a few of its rooms in their workflow, it may be that fully half or more don't even paint in 3DC. So you can see how this sort of error can escape detection for years.

Edited by toomanydemons
Link to comment
Share on other sites

  • Member

Now, if you bake in 3D Coat, your objects will look perfect. However, the exported normal map will seem a little off in other software.

In a nutshell:

 

When I bake normal maps in 3D Coat, they look a little off in other software, even though they look fine in 3D Coat.

When I import normal maps into 3D Coat, they look a little off, even though they look fine in other software.

 

The lack of continuity here cannot be fixed by any option from any menu from my thorough (thorough...) examination. I spent nearly a whole day trying to figure this out. And even if I did find some sort of workaround, software shouldn't work that way. It should work as intended and not rely on any sort of hacky fixes or workarounds that would snag newer users and force a google search. If the most obvious path isn't default then there's a problem with the software.  If "lock normals" did in fact make a difference, then it should've been checked by default since this workflow is common.

Edited by toomanydemons
Link to comment
Share on other sites

  • Contributor
8 minutes ago, Carlosan said:

I am reporting this issue to support@3dcoat.com

I'd also like to request a C4D preset for normals that can be selected in a new PPP project.  I would of created it myself If I knew the operations of C4D. I don't have it but I work with  AAA content/textures created by it and would like absolute compatibility with the libraries I have then export to other pipelines as needed.

Link to comment
Share on other sites

  • Reputable Contributor

As stated I already mentioned the rectangle boxes and their problems. I gave my solution for the rectangle boxes. Also mentioned that how I work might not be what you needed. 

I did not fix the rectangle boxes as a normal map was already created for them. 

I hope you find the solutions you need...  

Link to comment
Share on other sites

  • Member
19 minutes ago, digman said:

As stated I already mentioned the rectangle boxes and their problems. I gave my solution for the rectangle boxes. Also mentioned that how I work might not be what you needed. 

I did not fix the rectangle boxes as a normal map was already created for them. 

I hope you find the solutions you need...  

Thanks ^_^ I hope it gets solved too.

As for the rectangle boxes, I should not have to do anything to make those work. I should not have to add edge loops or unwrap differently. The main reason is that those normal maps look great in Substance Painter, Designer, Unity, Unreal, Sketchfab, Marmoset, and Blender. 

I shouldn't have to change the vertices and UV's of a model to make it work in 3D Coat. That is an obvious sign that things are broken. 

I've been keeping an eye on 3D Coat as a viable replacement for PBR painting. I hope that happens in the future :) but in order for that to work, the normal map continuity between software must be addressed. It is extremely important, for 3D Coat as well (and everyone who tries the software in the future).

Given the tantalizing advertisement on 3D Coat's home page, it all looks very appetizing!! This normal map issue must be addressed ASAP, and put at the very top of the list, while people are still jumping ship from the Adobe acquisition of Allegorithmic. 

Link to comment
Share on other sites

  • Member
20 minutes ago, Emi said:

I am sorry to tell you, but the problem is your model, not 3DCoat. But Maya and Marmoset Toolbag present the same issue.

I will explain why the best I can, but it starts for the way you are exporting your model from Blender, but it will happen the same on any other software if you do the same.

Yes, normal map might fix a little the shading issues, for how the normal maps are read by software, and that might make the issue a little less too noticeable, but the issue it is still there, you are just ignoring the real issue and then it seems you aren't applying some of the rules that would help you to make easier the job and life easier by baking better normal maps.

 

By opening the model on maya, right away the model looks terrible, I noticed that most normals were average and few ones were hard, which was weird to see. But that's clearly the the main reason of your problems here, how you are exporting the smoothing groups for your model and how they are being read in other software. First, the boxes can't  be soft edges or average, they are 90° angles, and no matter what you do, they will always have this lighting issue if they are not set to hard edges. In the past, before more advanced baking programs like Marmoset appeared to make it easier,  you had to always make a cage, and the cage would be the one that had the average vertex normals, so the cage would be used to do the average 'searching' of the high poly, without causing errors on the baking since the box would remind hard edges. Of course now Marmoset has the checkbox "smooth cage" doing the same and now you don't need a cage unless you really need a cage.

 

So there are two 'rules' here and they are linked as well. 1. when you are doing models with 90° angles, they should be always hard. Of course you can also do a little beveling and avoid the 90° angle edges. this will not only follow the shape closely of the high res model, but also, you can use the soft edges on the box and make it look nicer and with less issues, and rounded and prettier and all that.

2. Hard edges (which pretty much should be 90° angles) should always be split on the UV to avoid issues as well, you might not really notice it on your model, but it is there, and it has to do with how the pixels meet at the 90° angle and how pixels might get blended and they are just different so there is not enough space like a padding would do to fix the small issues that might happen. Yes, it depends on the resolution and UVs and all that and that might make it more noticeable than others, but yes, in a case like this where you are doing boxes, you would have to split all those 90° edges, and yes, that means 6 box sides = 6 islands to avoid issues.


Talking about 3DCoat and how it works with presets and all that. What I understand about it is that by using the presets like Blender or Unity, you are actually using a predefined file with not only how it will treat the Normal map, which you are right, it will be Green flipped or not. But also Tangent Space, the triangulation, and how it exports the normals and some other options. If you use the Unknown option, you will be able to use the ones on preferences, the ones you set. 

If you go to C:\Program Files\3D-Coat-V4.8XXX\ToolsPresets\ExtTools  you will see the XML files and how Unity differs from Blender, because Blender preset uses MikkTSpace while unity is set to LengyelAreaAngleWeightedSpace. That is the only difference. but apparently enough for you to see it because they are being interpreted differently. Technically they shouldn't affect much anything and only the way it reads the normal map if it is 'DirectX' or' OpenGL',  but the root of the shading problem is not about the normal map, it is about the model and it will always be the model.

 

I hope I explained decently what the issues are here so it might help you on your future normal map bakings. Good luck and have a good day!

There are no hard edges on the model. I'd be happy to jump on call with you personally and explore the situation. (https://gyazo.com/3386ce721adff5f6ead5849bdbed9373)

According to its documentation and settings explained by Allegorithmic staff, Unity uses MikkTSpace.

I have never had a problem with hard angles on a model if the cage is constructed correctly. Also, I have built at least a thousand hard surface models. (https://sketchfab.com/toomanydemons) (https://www.artstation.com/toomanydemons)

My dicord handle is TooManyDemons #2125. Cheers!

All of that being said, I have run some very recent test bakes using just cubes on very new geometry, considering the fact that those previous models are rebuilt from ones downloaded on Blendswap. I still ran into issues.

Additionally, those issues do not exist in Substance or Sketchfab, sorry.

 

 

 

Edited by toomanydemons
Link to comment
Share on other sites

  • Member

All of that being said, I am aware of the hard angle rules for baking normal maps. Considering that these are super tiny low-poly cereal boxes, I didn't think it would be an issue. I will take your advice and investigate further. 

All in all however, I'm getting some pretty nice results everywhere else:

https://gyazo.com/184d39716173d4e68588122d26205d5c

Edited by toomanydemons
Link to comment
Share on other sites

  • Member

It seems when I change the lighting on Sketchfab, the normal map errors are becoming apparent. So you may be right, this may be an issue with this model. Thank you. I've had several people take a look at this issue on Discord, I will run some more tests with them. Cheers!

PS. I am curious if you can get anything to look good using the import texture button to bring in a normal map, using the Unity (or Blender) preset. If you can, please tell me how you did it! 

Edited by toomanydemons
Link to comment
Share on other sites

  • Member

OK so yea :) it does look like the import normals option for Unity and Blender are broken. However, a newly created blank preset does read correctly.

That's a relief. So as long as I avoid 3DC's presets I will do fine.

Thanks for your insight about the hard angles @Emi After running more tests I can confirm this the problem with the box objects. 

Edited by toomanydemons
Link to comment
Share on other sites

  • Member

Leaving aside the UVs rules, the issues it cause are minimal, if you are aware of that, then it's your choice saving some UV space and ignoring the edges and the issue that might happen vs doing them correctly since you would have to get too close to notice the issues. sometimes there is a big gap between the edges, has happened to me but in this case they are not that noticeable. 

 

But looking at your screenshot, it was exactly my point. you can't have those boxes as average vertex normals or soft, they have to look like hard edged boxes because they are 90° angle edges. if you want to use soft edges you have to create a bevel. Shading error exists on your screenshot and until you don't fix it the problem it will only be covered by your normal map, but the error is still there. Boxers should be 100% hard edges, Cylinders should be Top and bottom Hard and the sides Soft.
When I opened the model in Maya, it seems even worse what what I thought because all sides were average and others hard. so it is a mess for any engine to deal with it. 

The lowest poly are usually the models with more issues because you are dealing with few polygons that can't deal with the high res topology that well, especially if you go the lowest possible poly count and you do a 6 faces 90° angle edges boxes like in this case, that's why I suggest making bevels, just tiny bevel will make a big difference on many cases, especially in 2019 where there is no excuse to have a box with no bevel at all for something like this.

 

And no, 3Dcoat presets like I said, are just small variations that you can even manually select by using unknown, you dont need the presets, they are just to help 3DCoat interpret the normal map and the model, but I never trust the presets, because like you say Unity preset should be set to MikkTSpace. But my point is that it's not just about OpenGL vs DirectX normal map. so that's not the issue here.

So, when I use 3DCoat which is usually to paint, everything works fine, but that's because I make sure everything on my low poly is good as well, I mean, to make good normal maps and make them work everywhere you need a good low poly that can catch the whatever high poly details you made. There is no magic, it just works because I make sure of everything, like the smoothing groups and UVs and all that.

  • Like 3
Link to comment
Share on other sites

  • Applink Developer
Just now, Emi said:

Leaving aside the UVs rules, the issues it cause are minimal, if you are aware of that, then it's your choice saving some UV space and ignoring the edges and the issue that might happen vs doing them correctly since you would have to get too close to notice the issues. sometimes there is a big gap between the edges, has happened to me but in this case they are not that noticeable. 

 

But looking at your screenshot, it was exactly my point. you can't have those boxes as average vertex normals or soft, they have to look like hard edged boxes because they are 90° angle edges. if you want to use soft edges you have to create a bevel. Shading error exists on your screenshot and until you don't fix it the problem it will only be covered by your normal map, but the error is still there. Boxers should be 100% hard edges, Cylinders should be Top and bottom Hard and the sides Soft.
When I opened the model in Maya, it seems even worse what what I thought because all sides were average and others hard. so it is a mess for any engine to deal with it. 

The lowest poly are usually the models with more issues because you are dealing with few polygons that can't deal with the high res topology that well, especially if you go the lowest possible poly count and you do a 6 faces 90° angle edges boxes like in this case, that's why I suggest making bevels, just tiny bevel will make a big difference on many cases, especially in 2019 where there is no excuse to have a box with no bevel at all for something like this.

 

And no, 3Dcoat presets like I said, are just small variations that you can even manually select by using unknown, you dont need the presets, they are just to help 3DCoat interpret the normal map and the model, but I never trust the presets, because like you say Unity preset should be set to MikkTSpace. But my point is that it's not just about OpenGL vs DirectX normal map. so that's not the issue here.

So, when I use 3DCoat which is usually to paint, everything works fine, but that's because I make sure everything on my low poly is good as well, I mean, to make good normal maps and make them work everywhere you need a good low poly that can catch the whatever high poly details you made. There is no magic, it just works because I make sure of everything, like the smoothing groups and UVs and all that.

Wau. I learned a lot from you. Thanks so much for taking time to explain these things. 

Link to comment
Share on other sites

  • Member
1 hour ago, Emi said:

Leaving aside the UVs rules, the issues it cause are minimal, if you are aware of that, then it's your choice saving some UV space and ignoring the edges and the issue that might happen vs doing them correctly since you would have to get too close to notice the issues. sometimes there is a big gap between the edges, has happened to me but in this case they are not that noticeable. 

 

But looking at your screenshot, it was exactly my point. you can't have those boxes as average vertex normals or soft, they have to look like hard edged boxes because they are 90° angle edges. if you want to use soft edges you have to create a bevel. Shading error exists on your screenshot and until you don't fix it the problem it will only be covered by your normal map, but the error is still there. Boxers should be 100% hard edges, Cylinders should be Top and bottom Hard and the sides Soft.
When I opened the model in Maya, it seems even worse what what I thought because all sides were average and others hard. so it is a mess for any engine to deal with it. 

The lowest poly are usually the models with more issues because you are dealing with few polygons that can't deal with the high res topology that well, especially if you go the lowest possible poly count and you do a 6 faces 90° angle edges boxes like in this case, that's why I suggest making bevels, just tiny bevel will make a big difference on many cases, especially in 2019 where there is no excuse to have a box with no bevel at all for something like this.

 

And no, 3Dcoat presets like I said, are just small variations that you can even manually select by using unknown, you dont need the presets, they are just to help 3DCoat interpret the normal map and the model, but I never trust the presets, because like you say Unity preset should be set to MikkTSpace. But my point is that it's not just about OpenGL vs DirectX normal map. so that's not the issue here.

So, when I use 3DCoat which is usually to paint, everything works fine, but that's because I make sure everything on my low poly is good as well, I mean, to make good normal maps and make them work everywhere you need a good low poly that can catch the whatever high poly details you made. There is no magic, it just works because I make sure of everything, like the smoothing groups and UVs and all that.

I still don't understand why you say my normals have hard edges. Am I missing something? can I see a screenshot of this, because according to my Blender file here, there is only one smoothing group.

Link to comment
Share on other sites

  • Reputable Contributor

Placing the information here for others interested in this thread not that you need this information tomanydemons.

I do not think EMI was saying that you created a hard edge smoothing group but a 90 edge is consider a hard edge. Now it is not a real hard edge smoothing group and will not be treated as  a real hard edge smoothing group. The term hard edge can cause some confusion.

My workflow. I would bevel a low-polygon  box. Yes it adds a little more geo but removes any shading issues as you soften the transition between the normals.  This is what I was talking about. I also model to reduce the wavey normal map areas that can appear in a low polygon model. 

With today's games engines and powerful  computers plus mobile devices, the vertex count in the low polygon model can be greater. This would include the added uv set vertices.  This does not mean you go hog wild.

I include 3 pictures. 

1. Left cube with just the 90 degree normals. (hard edge), Middle cube (bevel), Third cube. All edges set to a smoothing group with hard edges.  The first picture in Wings3D has smooth shading turned on.

Left cube has shading problems.

Middle beveled cube much better.

Third cube, set all edges to a hard edge smoothing group. I am just showing here for others interested in the thread. Normally depending upon the model you will have some soft and hard edge smoothing groups.

Second picture. Shown in flat shade. Orange edges show on 3rd cube I have set the edges to a hard edge smoothing group.

3rd pictures shows normals. 

For those interested, a couple of links

http://wiki.polycount.com/wiki/Normal_Map_Modeling

https://polycount.com/discussion/81154/understanding-averaged-normals-and-ray-projection-who-put-waviness-in-my-normal-map

 

first.jpg

second.jpg

Three.jpg

  • Like 2
Link to comment
Share on other sites

  • Member
23 minutes ago, Emi said:

Yes, what I mean is that a model with 90° edges should be set automatically to hard edge, there is no other way around and that's it. it doesn't matter if the Cube has 6 sides like in this case or 1000 subdivisions Width, height and Depth. If you don't set the 90° angle of a box as hard edge, it will always have the weird shading on those edges that are 90° but are set to soft or whatever.

You are applying a single smoothing group (like you said it) to the whole entire set of objects and that's wrong, that's the issue, because you can't make soft edges or whatever Blender is doing to Boxes (which in my test looks more like the average ones), especially if they objects are only cubes with 6 faces (the lowest possible cube you can have) So the whole box will look wrong. I don't know why you are doing it, or why you think it is a good idea, you even showed me the screenshot and the issue is there, but then you say i is okay and blame 3DCoat, when you are setting things incorrectly before you export it.

You can't "smooth" something like, especially if you have a simple 6 polygons box, if you want to use a box like in your models, set all the sides to Hard Edges. Or the other only way around is to add a bevel around the edges to break the 90° angle harsh and then set it to soft. Baking the normal map to hide the issue is not the solution, it might look a little better but the shading error still exists on the model. Even the cylinders in your scene should have soft edges on the sides and on the top and bottom faces they should be set to hard edges unless you add a bevel.

So the question is why do you still want to keep everything as a single smoothing group? unless you work on it properly and set things that should be hard edges hard and soft edges soft or add bevels on every 90° angel then nothing will fix your issue, and you have to just hide the issue with the normal map and pretend no problem exists.

To answer your question, I wanted to avoid the extra vertices involved in flat shading, and thought to make it all a single smoothing group. Thanks for the input ^^ so in this case I'll use hard edges for the cereal boxes and things, that should work.

It's very easy to blame 3D Coat if things look off (simply because the software is so buggy). But I'm glad that normal maps are working as intended. I would have shelved the software otherwise.

Ty reminding me about angles! It's something I learned about (particularly for making cylinders look decent) but forgot along the way. 

Link to comment
Share on other sites

  • Carlosan changed the title to [Solved] The Normal Map Preset issue

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...