We crested way over the $5m mark, selling stock that was held in a company stock account. It added another $300k or so of value to our net worth. Unsurprisingly, this gives me a lot less satisfaction that paying down what we owe (mortgage). I know this is unreasonable & illogical, yet debt causes me more emotional angst that our portfolio gains generate. We paid our mortgage down to the half way point today (facepunches graciously accepted). It's a bay area mortgage, so we've paid off just over $1.8m of house in a 4 years. Feeling good about that. Eventually, we'll recast our mortgage to a point that I'll feel comfortable quitting my job. I did the math today, and ... it's not far off. Now that, that feels "rich". The $5m of net worth, or the $1m+ of mortgage paid off? It doesn't feel the same. But, financial flexibility & the ability to do whatever I want in the future? That feels pretty rich.
(Sidenote: our kids have four years left of school, and after that, we'll be ditching the VHCOL house & moving somewhere cheaper. This is a plan that helps us get through the next 4-5 years.)