Wait, sorry, cancel ... I figured it out.
1. Clickthe [Camera] button at the viewport's upper right
2. Click the third option 'Background'
3. From the flyout click the last option 'Edit image placement'
4. Make sure 'Show all planes' is NOT checked.
5. Rotate your view so that the reference image is showing
6. Click 'Choose' in the Reference images dialog
7. Do no select an image. Simply click the 'Open' button .. effectively you're opening no image and that's how you delete the image.
Cheers,
Gamal