At the very least, if you still decide it's not worth the "stress" (and I get that), at least realize how much that perception is costing you. It's for you to decide if that's worth it.
Personally, I pay the minimum on my 3.625% fixed 30yr, and happily. Even if I was very debt-averse, I wouldn't even think of paying extra unless I was, at a minimum, maxing all available tax-advantaged retirement accounts (401k/IRA/HSA). Otherwise, at the very absolute minimum, I'm paying a "penalty" of my marginal tax bracket (all of which are much higher than 3.625%), not even including investment gains or compounding, both of which are substantial.
EDIT: Realized this is confusing. What I mean is that if you're not maxing your tax-advantaged accounts, you're paying extra in taxes in order to pay your mortgage. So every $1000 extra you pay on your mortgage and don't contribute to a retirement account instead really costs you $1000 + your marginal tax rate. So for example, if you're in the 15% bracket, it costs you $1150 ($1000 plus $150 in taxes) to pay your mortgage down by an extra $1000.