Sound plan, leave the shares to continue growing, borrow the $1.6m and spend the next however many years paying it down. Try and take a mix of split and variable loans so you can take advantage of cheap rates as well as filling up a big offset account, when in future you might need to buy out the other half.
Hopefully by then the shares will also have grown to such an amount that you’ll have the capital to redevelop the block if you need to.
Appreciate the points about family risks but I feel those will be there regardless of what you do, so you might as well carry on.