Your boss is being unreasonable; however, the best thing you can do is let it go and leave with dignity. You never know when you might encounter him (or someone else who's watching) in the business world, and you should never burn bridges.
He is not being unreasonable, he is doing his job and trying to retain a valuable employee, by appealing to your emotions and using a bit of guilt.
What the hell is suppose to do congratulate you and wish you well?
He went to bat for you with his boss and argued that you were underpaid and matched their offer. The natural reaction of any boss who has relatively new employee <2-3 years is to be annoyed when employee does the "I got a higher offer somewhere else". When you match their offer you expect a least year or two of loyalty. He now looks bad with his boss.
It sounds like this isn't a guy you want to piss off, giving him 4 weeks is a good start, So a little humble pie is in order.
Before you leave a conversation with him that goes something like this.
I am sorry, I realize that what I did so soon after you got me a raise, wasn't very professional. It was such big increase in salary, and I have gotten lots of students loans (insert reason I really need the money). The recruiter called and I reacted instead of thinking. If who ever takes over has question feel free to have them give me a call. (This is pretty much an empty promise)
If you have an opportunity to talk to your boss's boss let him know that this wasn't related to your boss.
If you don't do this DO NOT USE YOUR BOSS as a reference, and even if you do I'd be hesitant.
While its true that most companies don't allow bosses to give formal reference, LinkedIn changes things. Its way easier now then its ever been to use your network to get informal references, and yes I'd think twice before making an offer to employee who did this.