I think it's useful to draw a distinction between passive investing and index investing.
At it's heart passive investing is about identifying assets you are comfortable with holding for a very long time period and then sticking with them through the whole variety of market conditions that will occur. This approach when executed properly results in low fees, reasonable returns and fewer investor mistakes. It's a philosophy of how to act in an investment environment rather than what to buy.
The indexing approach then builds on this by saying that in the long run buying the market index is guaranteed to give you average performance. To the average layman without huge resources and skill to devote to investing it's very difficult to identify assets with superior 'risk adjusted returns,' so you might as well take all of the benefits of a passive approach and combine it with maximum diversification by owning a fund which simple buys the whole market.
As you've mentioned this can feel uncomfortable for two reasons: either the market is seen as expensive or some of the assets within the market are seen as expensive. Myself I am a value investor but not an index investor for both of these reasons. I hate feeling like I'm overpaying for assets. For this reason I only buy stocks which both seem cheap (bottom 20% of companies by P/B, P/E, CAPE), have highly profitable underlying businesses (top 20% of companies by Operating Profit/Assets), are shareholder friendly (high payout ratios), and are small (within the bottom 10% of total market cap).
It's important to note that many of the strongest results against active management are based on two very strong assumptions. Firstly, their alphas are calculated using regressions based on some or all of the factors mentioned above. Secondly their returns are then scaled by volatility. What does this mean in plain English? Well it means you can expect to increase your returns by 'tilting' towards these factors. This is exactly what I have done above. I don't buy 'expensive' companies because I don't feel confident in the fact that I will hold them through thick and thin. They give me an uneasy feeling through what I perceive as a low margin of safety.
It's important to note that the returns of these tilts may not work again in the future. The modern market might not reward you for buying the cheaper than average companies, or the small ones, or the more profitable ones. This is a very big debate in itself.
You are also losing a bit of diversification when you move away from the index. How much is up to debate. I personally think not much, others will probably strongly disagree with me. I read recently that stocks which have been on the market or less than 1, 3, 5 and even 10 years under-perform their more grizzled peers. I can dig that out if anyone is interested. This is why I think index rebalancing is less efficient that doing it yourself.
What I've attempted to show is that it can be perfectly rational in an efficient market to be something other than an index investor. Index investing though is still pretty awesome as it is so simple and easy. It let's you get on with the rest of your life! If you are interested in more check out this post by Joshua Kennon
http://www.joshuakennon.com/mail-bag-buying-stock-when-valuations-are-high/
