If someone invites you to a game and you don't see it and actually hit the button to decline it, then it will forfeit you.
The bug has been around for a few months.
The work around is to either pay closer attention or do what I do and play as invisible when doing coin games.