The interest saved would be exactly as much as you’d earn if you invested the same payments into bonds with the same interest rates. So, e.g. putting a lump sum $10k toward your 3.625% mortgage has the exact same impact on your net worth as putting those $10k into a bond yielding 3.625% (aside from considerations of taxes, liquidity, and risk). So would you buy a very safe but illiquid bond at that interest rate? If yes, might as well prepay the mortgage.
To be specific, a $100k lump to the 3.625% mortgage will earn you $3,625 per year until the mortgage is paid off. The payoff date will be earlier by the amount of time it would take to pay off the last $100k of principle. This early payoff is equivalent to getting your principle back from a bond.
Should you refi? Hell yea! Even with the $100k slug your mortgage will still have $290k remaining. Looks like you could shave about 0.6% in interest right now, which would be worth (290k x 0.006) $1,740/year in your first year, declining in subsequent years as principal gets paid down. That should pay for closing costs in about 3 years. There are also deals out there where you can exchange a higher interest rate for zero closing costs.