I hadn't played this game for months, largely due to this issue. I just finished a full round and it seems much less frustrating. If I missed on the swing meter it missed in the appropriate direction, and went awry in proportion to how badly I missed the swing.
Can't say it's perfect (only one round today) but it seems improved.