You are right.. as has been mentioned a number of times before.
In the meantime, before something like this is implemented, you can always change the EVERYONE notice to NO ONE while in a game so others can't see you on-line. I know this is a bit cumbersome, but it allows you to finish your game without needing to decline invites from friends and relatives.
Simply click on the EVERYONE and choose NO ONE. Don't forget to put it back when you want your friends, and the general public, know that you are available.
Have a nice day!