If an (ex) friends picture stays on your profile page even after you have deleted them, apparently the trick is to block them. Then, when their picture has disappeared from your profile, you can unblock them (if you wish).
(The only other reference to this tip I can find, is from spdemon, so thanks to you sp if it was your idea originally).