I find it valuable to have a large friends list.
The meter is only affected when you have your alerts sound on.
When the sound is on, you can really feel it in the meter when they're around.
In the game client, hit menu, options, audio, invitation alerts off. This not only gets rid of the sound when friends come and go or invite, but keeps your meter from hiccups when they do.