Yes, the headphone is just a listening device but Skype has a lot of stuff going on that could affect performance. Things like friend alerts, who logs on, messages, etc. There are options to turn that stuff off and that helps a lot. It isn't like there's a lot of background computing going on, you're basically on the phone playing.
Ventrillo is what 7th is talking about.