Some interesting answers here for sure.
I've been in this situation and for me it was very simple, I explained to Company A that I had, obviously, been looking for a new opportunity when I interviewed and even accepted their position. Company B came in, late to the game, after accepting the position, I've given this some serious consideration and feel, at the time it's a better fit for me and my career.
It really is that simple, forget about grudges, etc. You know how many people leave companies (sometimes not on great terms) and go back later on for bigger, better positions; TONS!!
When you seek new employment and interview with people/companies, it's a two way street; both parties have to believe it's going to be the best fit or it just won't work in the long term. If you feel you Company B is the better choice for you then, this is your career, so just move forward with Company B.