It's totally up to you. The best option depends on your goals and financial situation.
A longer term means a lower payment for a longer period of time, and usually a higher interest rate.
A shorter term means a higher payment for a shorter period of time, and usually a lower interest rate.
I agree it gets hard to compare options when you have a non-standard number of years left on your existing mortgage (say you have 22 years left and are considering refinancing into a new 20 year mortgage, for example). When I did it I usually built spreadsheets to help me figure out what was best. If you do that, be sure to account for the entire period (22 years in this case) so that you capture the effect of not having a mortgage payment in years 21 and 22 in the refinance case. I also usually considered the effect of taxes, but now mortgage interest is not as deductible as it used to be with the new tax law, so that is probably not too big of a consideration. Also, account for any transaction costs in the analysis (closing costs, escrow costs, etc.).