Polycount shouldn't be a problem, the mesh has only around 34k vertices. It was created in Maya, uv-packed in 3d Coat, fixed in Maya and than painted in 3d Coat. I also gotten some packing problems, when some UV shells are overlapping. These UV shells gotten somehow reconnected and crumbled to zero. So I needed to delete these overlapping shells, than packing, reimport to Maya and recreate them. Perhaps this is creating some strange Problems, but i'm not sure.
When i'm loading the file, I can see how my memory is going up to the max and then the swapping starts, so possibly more memory would be the solution.
My mesh is also using only 1 uv set, but I have far more layers. AO, Curvature, around 15+ smart Material Layers and 30+ other layers, but most of them only have some small details.