I realize this is an old thread, but I wanted to add the solution that worked for me in case anyone else stumbles across this. I was also having trouble.
In order for Applink to work properly, there's one simple fix that people might be overlooking. Your C4D scene must first be saved. Boom, that's it. If it's not saved, the exchange between apps doesn't work properly, and 3dcoat will open without any geometry. Once the C4D scene is saved, everything works perfectly..
To transfer the back to C4D, go to File > Open in Original App in the 3dcoat menu. Then click Ok and give it a few seconds to export the FBX in the background. Then in C4D, click on Import in the Applink menu.
That's worked for me, so hopefully someone finds it helpful. I'm using C4D r20 and Applink 2.0 by the way.