There's no reason why you can't do both. With the same monthly payment the lower interest loan will be paid off sooner.
Some example numbers (assuming $10k student loan starting balance, scale appropriately):
4%, $137/month. Paid off in ~7 years. $1434 total interest paid.
4.5%, $137/month. Paid off in ~7.2 years. $1655 total interest paid.
4.65%, $137/month. Paid off in ~7.2 years. $1723 total interest paid.
4.65%, $52/month. Paid off in ~29.3 years. $8232 total interest paid.
Insert your own numbers into
unbury.us if you want to experiment for yourself.
I personally wouldn't bother refinancing from 4.65% to 4.5% unless you have a very high balance. Not enough difference if you're aggressively paying it down. Refinancing to 4% could make a bigger difference.