IWBF,
I think you're avoiding the unavoidable by not refinancing your house to a cheaper rate/shorter term, even if that means dipping into your taxable account and bumping your monthly mortgage to do it. The math is pretty straight forward: currently, you pay roughly $1380 a month, or $16.5K a year, of which roughly 12K goes toward interest. But putting 30K down to refi at 15 years for 3.25% drives your monthly payment up to $1540, or $18.5K a year, yet pushes your yearly interest payments to below $7K, and saves you over 200K in interest over the duration of the loan. You go from paying $350 a month in principal to around $1000, allowing you to build equity at a much quicker pace.
The extra 2K a year for the increased mortgage payment should be trimmed from your everyday non-essential expenses, like cable and eating out. I think with a little work your monthly utes can be trimmed by $100 and your grocery bill by $200-$300 a month.
What about non-essentials, like clothing, hobbies, gifts, etc? I'm trying to get my wife to be a more mindful shopper, which has actually freed up a couple hundred every month.
Keep on maxing out the retirement accounts and find creative ways to trim spending while plowing those savings into rebuilding your depleted brokerage account. For example, check out some travel reward cards in order to get some free hotels/rentals/airfare to trim your travel expense. My family for the past couple of years was spending a very unmustachian $1K a month on vacations -- hockey game in the big city, trips to Florida, the Bahamas, etc; don't think you're the only one failing in areas you know you can do better. But now that we've addressed our psychological need to travel/escape in style, coupled with some travel rewards cards, that number will be chopped down to around $400 a month while increasing our actual vacation time (less Disney, more visiting friends and family out West, deep South, etc).
Trust me, I listen to your financial situation, from the house to the savings to the expenses, and I see my own. Early retirement means tackling that mortgage head on, and it starts first with a lower rate/shorter term.
Best of luck.