You have done well to this point, which says that your basic lifestyle and instincts and habits are good. But a big part of your fear is coming from not knowing. And that's a reasonable fear, because you
don't actually know. You have been planning on one path, and now you are fed up and are debating a complete 180, but you don't know much of anything about your current or future costs to tell you whether that new path is even doable. IMO it would be unreasonable
not to be scared at this point.
But that doesn't mean you just say, oh well, guess I'd better go back to the original path -- that would just be stupid (hmm, if I fail and have to go back to work, I will be miserable, so better just to lock in that misery and stay at my current job). What you want to do is start converting those "unknowns" to "knowns": use your time to investigate and learn all the stuff you don't know now -- how much you spend, what the medical insurance options are, what college is going to cost, what your future SS/pension will be at different times, etc. etc. etc.
So by all means, take the new job, for now. Unless you are on the point of starvation or homelessness, no job is worth daily misery. And then commit to a minimum of say 6-9 months there to see how you like it. And then over the course of that 6-9 months, teach yourself what RE would actually look like for you:
- You track your expenses, using YNAB, Mint, a ledger book -- whatever works for you.
- You investigate medical insurance options and costs
- You talk to your wife about how long she wants to continue working
- You investigate your options for withdrawing your 401(k), and you run spreadsheets and models where you can evaluate the effect of taking SS/pensions/401(k) withdrawals at different ages
- You run some net cost calculators for your daughter's college options. And conveniently, if you commit to another 9 months of work, by the time you approach your target date, you will know what college will cost, because she will have offers and financial aid packages in front of her, and you can revise your plan to take that into account.*
I suspect that at the end of that 6-9 months, you will discover that you have sufficient assets to FIRE immediately. But the best news is that you also won't be nearly as afraid of making that leap, because you will have many fewer unknowns -- and you will have the confidence that you have sufficient assets and skills to manage whatever comes up. I mean, sure, you can never know if health insurance premiums are going to go up. But if you have flexibility in your budget, you know that you will be able to manage if they do.
*FWIW, our RE plans coincide with the spring of our youngest kid's senior year in HS,
not when he graduates from college. Because by May of his senior year, we will know where he is going to college, how much money he needs to go there, and how much we have saved to cover that. Worst-case, if we're short, we work another 6 months or so to make up the difference.