Yes, once you've RSVP-ed go. Because you accepted that invite, someone else didn't get invited. Preparations have already been made for you, food may have been bought, etc.
As a hostess, very little annoys me more than people who cannot be bothered to attend an event that they've already Reserved a Place for. It's very rude.
On the other hand, if someone declines an invite right off the bat, that's not a problem.
Plus, what Tecoyah said!
