I think it depends on your entire financial picture, but I'd lean toward staying put. You've got the house to where you want it, and the difference in taxes is not that great. If it was several thousand, it would be more of a factor in my mind, but $3100 ain't bad! And if you take the transaction costs of selling and buying and divide that out into years of tax differential payments, it cancels out.
$50/month difference, basically. Ten years of paying the additional property tax is only $10K.
And you already know your neighborhood and your neighbors - your next neighbors could be jerks or there could be loud dogs, or...the list goes on.