Selling shares from a taxable account to buy a house

[Aardvark]

This story spans investments/real-estate/case-study. If I should repost it somewhere else, please let me know.

My wife and I were only planning on buying a house in 12-24 months from now, but instead we found a great house in an absolutely dream location. It's within our budget, but we dont have cash saved up so we would need to liquidate shares from a taxable portfolio.
Complicating this further is:

(1) An absolutely wild and difficult to comprehend dose of priviledege: a finanically secure family member is offering us a loan of up to $500k at 4.2% interest. This would come out of a family trust and would be formalized. He has a strong preference for us to minimize the amount that we loan from him. So, he wants us to sell most of our taxable shares and then loan the remainder from him.
(2) I don't actually know how to calculate/minimize the capital gains tax associated with liquidating our shares. Most of what I would liquidate come from SCHB that I've consistently been putting about $5k per month into for the past 18 months and IVV that was purchased well over a year ago. There is also some PRGFX, ICLN, SCHZ, and SCHC in the account that could help with tax loss harvesting (but I don't actually know how to do this, haha).

It seems like the best route forward is to leverag the priviledge by selling the majority of the taxable shares (and paying the cap gains tax) and then taking the 4.2% loan from the family member for the outstanding amount. BUT - this would leave us feeling seriously undiversified. We would have most of our eggs in one basket - the house - AND we would be living in that same house! My rational mind thinks this is okay... But my stomach is not happy with this.

More personal details:
Married filing jointly
Combined annual income is around $245k pre tax
Annual expenses around $80k
Annual savings around $90k
Taxable shares account: $280k
Sum of both of our IRAs and 401k: $76k
HYSA cash position (current account plus rainy day fund): $25k
Both of our credi scors are above 775

House:
Location: Boise, ID
Expected sale: $590k


Key questions:
1 - Should I listen to what my stomach is saying about having too many eggs in one basket, or should I listen to my head and chill out?
2 - Any potential red flags around taking the 4.3% loan from the family member?
3 - How do I minimize my capital gains tax. I don't want to go too deep down this rabbit hole, but I do want to get the easy gains if possible
4 - what else should I be thinking of?

[Rob_bob]

I don't know if I'll be of much help but I'll through out some thoughts.

I'm guessing you are young based on the amount of retirement savings you have.  You also have a good income and savings rate so you should be able to rebuild your account fairly quickly.

For many people their home is their largest asset.  I don't think of my home as part of my retirement assets since I can't sell it and live in it too, so I really don't think of it as too many eggs in one basket.  You will insure the house against loss right?  You can't buy replacement insurance on your taxable account.

Buying a home reduces liquidity, that might be what causes you to think of it as too many eggs in one basket.  What kind of emergency fund will you have after buying the home?

4.3% sounds like a good rate.  The question is what kind of person are you borrowing from?  Are they easy going and will stick to the loan agreement or will they at some time lord it over you and expect little favors from you?

I use tax software, gains and losses are entered in and it does it's magic.  Basically $1 of long term capital loss will offset $1 of long term capital gains.  As I recall only $3000 of losses can be used in one year, if you have more than that it will be carried over to offset income in the following years.

[Aardvark]

Thanks @Rob_bob
You make great points.

I am 33, my wife is 28.

You are right that it's the lack of liquidity that gets my stomach in knots. Also that our net worth is so closely tied to the local housing market.

We will keep our emergency fund at a comfortable enough size (as you can hear - we are not keen for risk on the liquidity side of things).

They guy we are borrowing from is solid gold. There is nobody that I'd be more comfortable borrowing from. It is still awkward doing money things with family. But the situation could not be better than it is.

BIG follow up question:
I assume that the tax software that you use is too expensive for me to buy for the same of this one transaction? Is this the type of thing that is worth going to a tax consultant for?

Additional question:
What about the following strategy... Sell the shares that are more than a year old and pay long term cap gains on that. Instead of paying short term cap gains tax on the shares that are less than a year old, I take out a Pledged Asset Line from Schwab for like 50% of the value of the shares that were purchased in the past year. I am trying to connec with Schwab regarding what my interest rate on the loan would be (what my Secured Overnight Financing Rate ("SOFR") would be). If my interest rate from schwab is low enough, then this seems like an attractive option. What should I beware of here?

Update - I found out that the SOFR has nothing to do with me. It's a Federal rate that can be found here:
https://www.newyorkfed.org/markets/reference-rates/sofr
Currently around 5.3% which would leave my PAL interest rate around 9.95%... Which is much higher than I was expecting.

[VanillaGorilla]

I'm frequently very grateful that I bought my first house a few years before I ever discovered the FIRE literature. I know for sure that I would never have pulled the trigger after reading things like JL Collin's book, and it would have been far less efficient in the long run.

A few random thoughts:

Look up your cost basis on your taxable shares. It's probably negligible since the market has been down and flat for the last bit. Taxes are easy. Your brokerage should show you the long and short term gains on your shares before you sell. Calculating your tax liability is trivial from there. You don't need any software, just a calculator.

Tax loss harvesting is complex words for "selling shares at a loss, only to reinvest immediately". Not what you're talking about. Any shares you sell at a loss will offset other gains, but they're still a loss.

IMHO that is a reasonably affordable house for your income. Your incomes should increase with inflation. Your mortgage payment will not.

Stop worrying about "all your eggs being in one basket". You have relatively few eggs right now and you're going to have a lot more in the future. Buy a house if it matches your life now and in the near future, if it makes sense, and you can afford it. Your net worth being "tied to the local housing market" is entirely the wrong place of thinking about it, unless you're planning on leaving the area in the near future. A house is not a way to build your net worth, it's a way to provide security, shelter, safety.

After a decade of home ownership I adore my house. I think of it as a hedge against my equities. It provides a stable imputed rent no matter what's going on in the equity markets. It's like an inverse bond that I live in. It's freaking awesome.

I'm not going to touch the generous family member aspect - that's out of my expertise.

[nereo]

Question: are you trying to purchase this house “in cash” (without a traditional mortgage), and if so, why?  I understand rates are high at the moment, but you seem to be skipping over the “traditional” option which would leave most of your diversification in tact. At your reported income and credit scores there’s nothing to suggest this wouldn’t be a viable option.

At your earnings and savings levels you can “bounce back” from almost any path you choose assuming your life circumstances do not change greatly.

[Aardvark]

Thanks @VanillaGorilla - super reassuring reading your post :)

@nereo - I think you're asking the question more directly than I phrased it. Which is great. My question is basically: "is the combination of diversification and liquidity important enough to pay the 7-8% interest rate assocaited with a regular mortgage - or should I give up the liquidity and diversity of the shares portfolio in exchange for the 4.2% mortgage rate?" - and then in addition to that I suppose I have a question about whether there are other creative instruments like Pledged Asset Lines that might be useful in this situation.

One update in my thinking is definitely around viewing the house as a home. An investment in quality of life rather than an investment for a maximum return.

[mistymoney]

I don't the like strings on that 'privledge'.

You are making very good salaries, why rely on a favor from family? You can sell just enough for a traditional down payment and get a mortgage like every one else. Yes, it will be more than 4.2%. It's the cost of independence, and my advice is to think about how much that might mean to you.

If frist time buyers, maybe FHA loan would work? need much less down payment, but I don't know how it works.

[Rob_bob]

Thanks @Rob_bob


BIG follow up question:
I assume that the tax software that you use is too expensive for me to buy for the same of this one transaction? Is this the type of thing that is worth going to a tax consultant for?

Additional question:
What about the following strategy... Sell the shares that are more than a year old and pay long term cap gains on that. Instead of paying short term cap gains tax on the shares that are less than a year old, I take out a Pledged Asset Line from Schwab for like 50% of the value of the shares that were purchased in the past year. I am trying to connec with Schwab regarding what my interest rate on the loan would be (what my Secured Overnight Financing Rate ("SOFR") would be). If my interest rate from schwab is low enough, then this seems like an attractive option. What should I beware of here?

I use Turbo Tax Premiere, it looks like federal plus state software will be about $140.

Yes, if you have capital gains sell long term, longer than one year, holdings first, those have the more favorable tax rate.

[reeshau]

4.2% is below the current IRS long-term Applicable Federal Rate. (AFR)  This could generate imputed income foe them, and the difference considered a gift (and income) for you.  Many people could manage this with a little paperwork and an expected lowering of their estate tax exemption, but if they have enough to lend this to you, maybe it's important to you both.

https://www.investopedia.com/terms/a/applicablefederalrate.asp

Cashing out your investments is a dumb move, in isolation.  If that's definitely a string for this offer, maybe you can live with it.  Putting away $5k / month will definitely build that back up quickly.  Is this your first house?  A new house comes with other expenses, too: repairs / refreshes, furnishings, moving, supplies / endless misc. items.  Make sure you have money available for that, too.

[MustacheAndaHalf]

Your brokerage account should show a gain/loss or "cost basis" amount somewhere.  The gain/loss is what you pay taxes on if you sell that entire investment.  Vanguard has a "cost basis" method called "HIFO" which stands for highest-in, first-out.  The most costly shares are sold first, because they generate the smallest gain (or a larger loss).  For simplicity and lower taxes, that's a decent combination.

Getting a loan from family members for half the market rate (4% vs 8%) seems like something the IRS would notice.  But for a $500k loan, their "gift" is about $20k/year to you and your spouse.  Since a financially secure family member can gift each of you $17k/year, that could fit under the gift exemption - I think.  You might check with someone who has more experience first.

https://www.irs.gov/businesses/small-businesses-self-employed/frequently-asked-questions-on-gift-taxes