Have you worked it out mathematically, by checking out the
NY Times Rent vs Buy calculator? Once you know the numbers, all that's left is an emotional decision. Is buying
now worth whatever changes this will trigger in your financial future? Will this push back retirement by 6 months? 6 years?
Deferring expensive decisions has a nice side effect of allowing investment compounding to take effect. Obviously, leaving that $50k out in a (1%?) savings account means it's not going to compound so much. Are the numbers convincing enough to put buying a house on hold at least 5 years? If so, maybe invest it understanding the risks - the money may not have grown yet since it's a short timeline. If that happens, you have two options - start rebuilding the down payment fund
now while the $50k moves into invested assets, and in 5 years time, see what the rent vs buy picture is, as well as how the two of you feel emotionally about the decision at that point.