Just to address the "having to pay back immediately if you lose your job", that isn't true if you have Vanguard. You do have to continue making payments to it, but a former co-worker left last year (took out a 401K loan to buy his house) and Vanguard allowed him to continue with his regular payments. Not sure if this is a special case or not, but it is not an absolute truth. Obviously something to confirm if you do go through with it (for the record I wouldn't).
We took out 401K loans when we bought our house 4 years ago. We had to carry two houses at once and should have paid them back when the old house sold, but it was very pre-MMM days for us, and instead we put in a pool. We love the pool and all, but we should have paid off the damn loans ASAP instead. Big regret!! It is our only non-mortgage debt and a big source of stress. We have made some adjustments and are treating the loans as "hair on fire" debt now. I would only use a 401K loan as a last resort.