I have the same experience RM, I get booted often. Many times I think it is because the play who set up the game doesn't want to deal with the tournament green speed and other times it is just because he wants to be the 'Big Dog' in the group.
I play a good bit of alt shot and generally set my search at Tour Pro and above. Playing in a match with 2 legends and 2 tour pro, 1 legend, 2 masters and a tour pro, etc. is a lot of fun. Many of the lower tier players enjoy the chance to play with a Legend and it is always fun to play from some of the spots that you end up in.
I did have a game with another Legend (who will go unnamed) the other day in an alt shot game with 2 tour pros. He had the better of the two in terms of equipment and average but his partner was having a bad day. He complained and berated his partner on every bad shot, finally raising the ire of the other three players. Three down with four holes to go he conceded the next hole after the tee shot. Players like him are better off staying in level rather than playing with lower players, of course the sad thing was in my opinion he was the weak link. You expect your Tour Pro partner to hit some bad shots and you as a Legend have to make up for them, he didn't have the ability to do that.