Gah! I hate credit cards so much! Can someone offer me some advice?

I am young and I have three credit cards which I almost NEVER use, and when I do use them I pay them off in full. I only have them because they are my only lines of credit for building a credit score.

My problem is one of the cards randomly switched the terms of service on me without me noticing and now there is a $50 annual fee bill which showed up in the mail.

I was livid and called them and complained, and they said that I had the opportunity to revert to my old terms of service but the expiration to notify them had passed already. The thing is I was out of the country when that piece of mail came! :(

Now I have a credit card bill with a $50 charge for the fat bankers and I don’t know what my options are.

1. I can close the account and lose my oldest line of credit which will hurt my score, but
2. Even if I close it, I would still have to pay that $50 fee, right?!

I feel so cheated by this crappy bank!

Any advice would be appreciated.