To the original poster, the simple fact is the game will be too easy if timing the click is not involved, I know its the one MAJOR thing stopping me from scoring much lower.
As others have mentioned, it cant get much more realistic than it is except IMHO, by bringing it to a 3 click meter, yes 3 clicks. Why, well when you take a backswing in real golf you have to stop it at a certain point during motion, right now you have the luxury of time to adjust that for power so feel lucky for this as it is now, more realistic and harder will be 3 clicks.
As for the forward swing click very simple, when you take a real golf swing, you have to HOPE you come down and hit the ball exactly where you intend, ie usually the sweet spot in the middle but sometimes you may hit off center a little to draw/fade and sometimes you dont hit the ball where you intend and end up slicing, hooking and just pushing the ball to the right etc. When you are clicking the meter right now this is what it is replicating, so yes the forward swing is realistic the way it is and retains some challenge for a video game, not clicking there would be too easy.
Happy hitting