yes, happens at least fifty percent of the time. You get an invitation, you accept, (sacrificing your current game) and pfffft. - there you are at the main menu, with a little box advising you that the game has been cancelled. I've invited that person to another game immediately, but whether it's wgt's fault or whatever, he never accepts - probably gets booted too.
Being an intermittent fault, it's going to be a tough one to solve.