I married in 2007 and bought a house in 2010. In 2007 I didn't know we would buy a house so soon, so I was putting money into a Roth IRA, and a minimal amount into my 401(k). I was using the rest to pay extra on student loans. The housing crash hit, the government was giving free money towards a house down payment, and we realized it was cheaper to buy a larger house than continue renting. The last part is only partially true as the house we ended up buying added a 45 minute highway commute replacing an 8 minute in town commute.
Roth IRAs allow you to withdrawal contributions tax and penalty free for any purposes. That is what we did. If I had any gains (I'm not sure any remained), I left them in there.
What I don't like about brokerage accounts is if you move your investments around a little it triggers gains even if you don't remove the money from the account. If you move from S&P 500 to bonds, you get taxed, and have to report the sell. If you move from either to a company stock you get taxed and have to report the sell. Whereas I can do as I like in an IRA account without needing to report it.