I think it can be a good idea. I have other investments but would be quite comfortable with real estate alone.
I'm not worried about the eggs in one basket at all because:
1. If you understand real estate well it is not a crap shoot. You can control for most of the risks through insurance, long-term mortgages, good credit and good practices.
2. If prices decline you don't have to sell.
3. A smart RE purchase beats the 6% or 7% return investing could bring hands down.
Ex. We invested $150,000 for a $700,000 multi-family. The rents pay the mortgage and expenses - allowing us to live for free and save the equivalent of est. $900 of mortgage interest each month that we would otherwise pay for a SFH which is then invested.
I'm not as good with math as many others here so there may need to be some adjusting, but here is a rough comparison:
$150,000 invested for 20 years with 7% return compounded turns into $580,452.67 at year 20.
$150,000 invested into a multi-family rental for 20 years with rental income creating in additional $900 a month to invest for 20 years results in that $900/month compounding over 20 years to be $462,848.17, plus principal pay down of $350,000. This does not account for house appreciation which in our area has historically averaged 5.4%, but let's put this at 3.5% instead. This means at year 20 your home may be worth $1,392,852.
Now, there are a lot of variables, but if we compare these rough calculations you can see that it is possible that you could turn $150,000 safely into $580,452 through passive investing, or turn $150,000 into $1.455 million through a home purchase plus investing the surplus income with slightly more risk and a bit more work.
Taxation of proceeds and rents and investment income is a more complicated country and situation specific calculation, but still favours owning multi-family.
Once the entire mortgage is paid off in five years the income will be the equivalent of $3000 per month after expenses, free rent and underlying asset that will likely continue to appreciate at 3.5%. You will also have your approx. $500,000 in additional investments that could spin off another $20,000 a year at the SRW.
This is only one strategy. Some of you live in markets that are cheap to buy in and give great ROI. You could potentially buy a bunch of properties and hire a manager and live on the income stream while the equity increases.