I guess the real question is: what are we counting net worth for?
I count it to see how close I am to being able to live off of it. So if I have 400K in stocks, and have decided that I'm comfortable with taking 4% a year of that forever, then I have 16K a year that I can live off of. I compare that to how much I'm spending now, and say, hmm, I'm X% there! If I put half that money into a house, now I have 200K in stocks. So if I have and 200K in a house. Now I have 8K I can live off of, and a house. I compare that to how much I'm spending now minus the housing expenses saved and think, hmm, I'm X% there.
So if I put 25K into a house, I would decrease my net worth that much, but hopefully, that would decrease my spending some, and those are the 2 sides of the coin that I care about. That's what I'd be thinking about when deciding whether to put the down payment on the house anyway.