We also live in a HCOL. We decided to make a large (very large) down payment when we bought our house, even though that's not the most mathematically advantageous. I am the breadwinner, my DH earns very little, and I work in a cyclical industry. I needed to know that if I got laid off and had to fall back on freelance work or less than full-time work, I would still be able to make the payments. So keeping the payment moderate was a bigger priority for us than maximizing the return on the down payment amount.
I did get laid off (some years later) and didn't get a new job for 5 months, and I was glad I didn't have to come up with an extra 1-2K per month for the mortgage.
You need to assess your comfort level with the alternatives and do the sleep test. Could you sleep at night knowing you had a big brokerage account along with a big monthly nut? Or would feel happier with less invested and a lower payment, plus more monthly saving to replenish your investments?