Do you already own the home you live in? If not, I'd buy that first. Best interest rates and loan products. Then after a couple of years buy another to live in and convert yours to a rental. Then keep going. That's how I did it, "laddering up". Yes, my family lived in 8 homes in 18 years, and I did purchase straight investments as well, but it all started with that first one.
If the Brokerage account is extra, left over money, then I would cash out what you want to use and use it instead of borrowing against it. But I like to keep things dead simply and not try to get smart or tricky with money like that. Even if it makes sense "on paper".