Thanks everyone, I was wondering if posting here, would help me get feedback through my thinking and it really did.
I should have made a few things clearer than I did in my original post.
I totally agree that a 6,000 square foot house is too much, and I my wife and I feel a 2,500 or 3,000 square foot house with a finished basement is more than enough room. We see, that in Fort Collins, CO, with good public schools, we can get a new house, for half the price ($500,000), and 80% less property taxes (a $20,000/year savings...). We like the idea of getting a house that is just big enough for a family of 4, but also small enough that when my wife and I become empty nesters we can stay in the same house and 2,500 - 3,000 square feet with a finished basement should do just fine.
To those who asked about hiring someone to run the business, or get help to take some of the work off my shoulders - good point. My wife and I go back and forth on that. Hiring someone to "run" the business - well, I really would probably be OK to have someone handle a lot of the sales follow up/admin tasks. I do have someone who does a lot of that now, though they are not a salesperson. I suspect if I can get a salesperson to take some of the more routine/day-to-day sales stuff, I'd be a lot happier.
My wife does all the financial stuff. The MMM "thing" made me look at our spending, and we realized that our "required" spending is about $9,800/month, with about $2,000 of that being our property taxes - and that is why the whole "property tax thing" sprang up. So nearly 20% of our required spending was property taxes, whereas if I lived in other places it would be about 5%--similar house, similar schools ect.
Now, after seeing what other people were reporting with property taxes elsewhere in the US, I was surprised. Illinois is high (or at least my school district is high), though I can see there are plenty others that are as high as what I'm dealing with - and, surprisingly, not as frustrated about it as me.
It seemed to me that if I could live somewhere near the mountains, good public schools, and perhaps half the house - and near a good public university, I could really stretch my money further and enjoy my life more. Getting half the house returns $500,000 to savings (or about that), and reduces my property taxes by about $20,000/year, and then going to a town with a public university in town, reduces my kids college expense by half. I believe the deal I'll make with my kids are: mom and dad have X amount of money for your college. If you complete your bachelors or masters degree, whatever is left - you keep, ideally, to use to start a small business of their own. This is my current thinking anyway...
Boulder seems CRAZY expensive - I mean, NUTS expensive. Boulder has really excellent schools, looks very pretty and is a big college town, but 800 square foot houses go for $800 - $900k. Crazy. Boulder seems like going right back into the rat race to me.
And by the way, this whole thing started because my wife and I got frustrated with our business a few months back and though, well, what if we retired? What could we generate in income, and what would our expenses be? Another thing happened, was that this year was the first year that our dividends alone in our savings came very close to $100,000 by themselves -not even counting the gain in value. Again, this is when I though...what the F am I still grinding things out with my business for? I'm 44, is there something else I'd like to do?
I asked myself the question - would 70 year old me be annoyed with the 44 year old me for not taking a little risk and changing course, trying something else at this state, especially with finances in good shape? I think 70 year old me would. 44 year old me likes hunting/fishing/skiing/hiking, ect.
My business, though while generating a comfortable income and a nice "stache", is probably worth at most $500k to $1M and I wouldn't get the money all at once.
There are things I do enjoy about the company. I get to work with my wife, we built it together from nothing. She has been with me the whole way and we get along fine running the business together (people always ask how it is running a business with my wife- assuming it is not good). There are certain employees that we have that really cause me to want to hard, and create opportunities for them because they are loyal and good at what they do. Challenging employees and challenging clients make it awful, though, I'm trying to be much more selective on both these days.
The thing about downsizing my house and staying in the same school district is that I'd have a house, half the space, older, with an older layout, but still $12,000/year in property taxes. It seems nuts to me that a $500,000 house should have $12,000/year in property taxes.
Thanks again everyone - super helpful input. I read everyone's posts 3 times and am going to share with my wife.
We are already bought tickets to do our family summer vacation in Colorado, specifically to visit Fort Collins, Colorado Springs, and Vail (just to get a few days in the mountains in the summer during off season). We plan to visit some of the schools (if possible) and a few other things that are important to us. We want to rent Airbnb and "live" there for a few days each. From there we want to determine if what we see, we like better than what we have. Is the pull, off an early retirement lifestyle in Colorado, stronger than staying where we live in the community we live in. We just won't know unless we go. If we are going to move, we realized it has to be after this next school year (sell the house, deal with the business, ect.), so we either get moving on this, or we are just all talk.
One other issue is neither my wife or I know what we would do if we were retired. We worry we'll have deep regret letting go of a profitable business that we built together. Neither of us has real hobbies. Though, the idea of clearing my head and letting other things comes in, really sounds very intriguing.