There's no reason why you can't do both. With the same monthly payment the lower interest loan will be paid off sooner.

Some example numbers (assuming $10k student loan starting balance, scale appropriately):

4%, $137/month. Paid off in ~7 years. $1434 total interest paid.

4.5%, $137/month. Paid off in ~7.2 years. $1655 total interest paid.

4.65%, $137/month. Paid off in ~7.2 years. $1723 total interest paid.

4.65%, $52/month. Paid off in ~29.3 years. $8232 total interest paid.

Insert your own numbers into

unbury.us if you want to experiment for yourself.

I personally wouldn't bother refinancing from 4.65% to 4.5% unless you have a very high balance. Not enough difference if you're aggressively paying it down. Refinancing to 4% could make a bigger difference.