You're not crazy. My husband did exactly this, except in another HCOL area. He bought a pricey 2 bedroom, 2 bathroom condo with his best friend. Their agreement was, whoever gets married first buys the other person's share out at the market value. Sounds like you both have family who can bail you out financially if the deal goes awry, which will be a great safety net. As long as you're open to losing out on a bit of money here and there in order to keep your friendship intact, you should be good. Be sure you have your plans for exiting out of the arrangement clarified well in advance. My husband also took a loan from his parents to buy his share of the place, and then a second loan from them to eventually buy his friend out. The property value increased a lot after that transaction, which his friend didn't mind.
The main concern of mine would be you both moving in with your SOs. You two are best friends, but your SOs aren't. If there's a disagreement between one of your SOs and anyone else in the house, will you or your friend be forced to choose sides? If it comes down to it, would you both choose your friendship over your relationships?