All great answers, so I have a different twist.
Are you expecting too much is the key.
What are your expectations?
If it is about wining and being the best(or even just doing well in RG;s), walk away.
Yes it is a quitters attitude but sadly it is the truth, you will never be one of the elite.
If your expectation is to have fun and enjoy the game then stay.
I am in the last category, I suck most days but still have as much fun regardless of the score, sometimes more so when I card an over par score.
I do not quit and I am too a golfer, not a gamer,
When I score well, it is due to my putting, nothing else. I do not take notes, and calculate furiously, it is by feel so some days it works, others (most days) it doesn't.
If I took notes and calculated each shot, I would be a lot better score wise, but this is a pastime, not a life or death scenario, so I will bumble along. If you want to improve, take detailed notes on each green (where to play the approach to) and notes on the break etc - and you will improve, especially the more putts you take.
My advice is to enjoy the game for what it is, if it isn;t fun, it is not worth spending any time on it.