Would applying the extra money allow you to go down to a 15 year mortgage? Your monthly payment will increase some, but you are looking at total amount paid, and it will be way less.
Bankrate shows a 30-year at 3.88%. Having said that...
$364k x 3.19% x 15 year mortgage = $458,480, monthly payment $2,547 (your 30-year payment is $1,751)
My worry about points is that, should you be wrong about this being your forever home, that money is gone. If you put it into equity, then you can at least get it back out, assuming you aren't in a local real estate bubble.
The average person in the US lives in 13 places over their lifetime. That shocked me, until I realized that, at 45, I had already clocked in at 12. (and am in #13 now--and not in the US) Plans change, unexpected things happen, and the best solutions aren't necessarily the ones optimized for the "happy path," but rather the best total expected outcome.