Right again. You do lose the lighting information embedded in a genuine .exr panorama. So, converted images are only good as backgrounds that will not influence the lighting of your 3D object.
Here are some links for genuine .exr panoramas available online for free:
And here is a "master list" of free .exr and .hdr panorama sources: