I personally think it is related to the view and not to the straight line between the ball and the flag. At first, I thought that it was related to the Skyview at the bottom right (like in those cartoonish ones), but if you change camera (which is the view and the little man is not always there), the wind arrow will change accordingly.
But I maybe mistaken, hope not!
Rob