Definitely don't pay for points.
In fact, do a no cost re-fi. This will give you a slightly higher interest rate. Some places will advertise "no cost" but they are just moving the closing costs into the balance due, that's not what you want. You want a true no cost, and to get that you'll pay a slightly higher rate.
When rates hit rock bottom in '21-22, I refinanced four times in a year and a half, each time doing a no cost.
I ended up with a 2.75% 30 year. If I would have paid a little I could have gotten a 1.95% 30 year, but my balance is low enough and the rate is low enough that it didn't move the monthly payment much, and the break even point was something like 8 years.
I was using online brokerages. I found I could pull $2k out of my equity without it being considered a cash back re-fi (which didn't have as good terms).
When rates are falling it makes sense to keep refinancing with them, it's just a paperwork issue.