If your portfolio is diversified you'll be able to pull out your 4% annually with some confidence.
http://dqydj.net/sp-500-return-calculator/Just cherry-picking some dates for illustrative purposes:
1965-1985 total average annual return adjusted for inflation: 1.474%
2000-2015 total average annual return adjusted for inflation: 2.502%
There have been long periods when the S&P500 did not manage the magic 4% annually.
1965-2015 total average annual return adjusted for inflation: 5.325%. Even over 50 years the S&P500 did not manage the often touted "7%".
Current S&P500 PE is about 22, an earnings yield of 4.5%. Profit margins are at historically very high levels. The market is being driven up by just a few mega-cap tech stocks. Interest rates are at very low levels.
In my opinion, now is not the time to be sinking one's entire nest-egg into a market-cap weighted index fund.