YankeeJim: To my way of thinking, VEM is an error factor that magnifies your mistakes.
If you make a mistake (and VEM can determine you did make a mistake), that should reduce your skill level. VEM won't do a thing. And if it reduces your skill level enough to go past the threshold, VEM would actually help you get it closer.
But how does VEM know you made a mistake? There are many ways to get the ball close. In one case, club down and add topspin, or don't club down and use backspin. In a different case, don't aim so far right and use the slope left of pin to bring the ball back to the pin, or aim further right and don't use the slope, or miss the ding right and don't aim so far right...etc. So how can VEM tell you made a mistake?
Maybe as an example, it looks at your preliminary shot result and thinks, "uh oh Jim's 15ft from the pin and his average DTP is 11, that's a mistake. Let's update his skill level and check against the threshold. Nope, still below the threshold so nothing for me to do here. Let's go with Jim's shot untouched." OR "Yep, that's above the threshold, Jim's having a bad couple of days I gotta help him out. Let's go with this improved result 3ft from the pin."
And I do agree the skill level has to be based on more than only the shots in the current game. You need a wider view, but not too wide.