To be honest, it does SERIOUSLY outperform over time. But you cannot just go buy anything, in any area, and put a manager on it. And you have to qualify what defines passive investing.
My one friend has 27 single families right now, in an area that will not appreciate beyond inflation, and averages 21.5% annual return (before depreciation). I know quite a few examples of this. He manages himself, but would still beat the market average with paying for management.
If you buy, improve, and then rent the returns can be enormous, but now you are getting further into the gray area of "what is passive?" Is RE passive in the true sense, ever? What if you hire out all the rehab work, and the management, etc?