We have a separate emergency fund (approximately 2-3 months worth of expenses) in our brick and mortar bank, and I have about 1 year's worth of expenses (to be used in the event of a crash) in an online savings account earning better interest than the brick and mortar bank.
I know what our basic level of expenses are (with all the creature comforts that we require), and I then added 5K to that amount to cover for things like major house repairs, car repair/replace fund, and vacation. I came up with that amount based off of past performance; how much general house repairs cost us, how often we had cars worked on and that cost... it's basically a guesstimate, but a pretty good one I think.
We saved up enough in our investments to hit that total number, and now we're going to just pull out of investments on an as needed basis. So every month, we pull out enough money to pay for that month's expenses. If we have something unexpected come up, it will get lumped in with the monthly expenses and paid off at the usual time. We might dip into the emergency fund, but if so, come bill paying time, we'd just top it back up to the level we had it before.
Since we know a "high number" and planned for it, then the idea that many years we likely won't even use that amount means our portfolio should be just fine and the extra is just a cushion.
Even if you're worrying over the penalties with pulling from an IRA or the like... if it is a true emergency and you have to come up with the money, it's only 10% I think to pull the money you need out.
I'm lucky in that I have a large inherited IRA that is penalty-free to pull money; I just have to pay taxes on what I receive. And being in the married filing jointly/below the 15% taxable bracket, I won't actually ever pay taxes again unless we get really crazy with the money.