It seems obvious now I've worked it out, but it wasn't when I started, the amount of effect the wind will have depends on the club you are using. The longer the ball is in the air the more chance of the wind shifting it. So if you're using a 3W or 3I there will be a lot more swing than a 9I or wedge.
You won't believe how long it took me to realise that!