A large part of the game lies in loading gross amounts of data (high-quality images, etc.) over your internet connection. If that connection does not have sufficient bandwidth or otherwise solid performance, then you're in a world of hurt.
The swing itself, right up to the moment of impact, is strictly locally controlled. Despite that, the game has multiple threads running. If any of those threads is busy doing something else, particularly if one if pre-loading information from the servers, then even your swing can be affected.
Whether or not you can improve your connection in any way is impossible to say, as you don't give any details more substantial than "it sucks."