Except pom, then you wouldn't be tracking the index, and you would have to pay more attention to it.
How many hours are you willing to spend doing it? How much is that time worth to you? If you choose wrong with your investments, and some of them go sour, are you willing to take an entire-portfolio view of your investments and hold the course vs. holding the index? Can you sleep well at night with those choices?
The moment you step away from an index you -have- to ask yourself these questions, and .05% is NOT worth making these considerations for. I don't primarily index invest because I enjoy doing what I do. Even then, I'm starting to look into value and dividend funds, but those do have higher ER's then .05%, but they would probably provide me as much piece of mind and I will be putting much of my further funds towards such ends.