I had a VERY small "empire" consisting of 4 houses at one point, and you'd be surprised at how little work it is overall. If you are buying slum housing and dealing with crack addict tenants or something, I could see the hassles adding up fast. But I had houses rented to middle class professionals and graduate students and people like that, which I had done all my due diligence on before buying. I also very carefully selected tenants in stable positions who stuck around, in all but one case, for the entire time I owned the property.
So there's a bunch of up front work (probably a full 40 hour work week per property when all is said and done), but if you do that well, the ongoing work is negligible - it ended up being *maybe* 4 hours of work a month for me. Many months it's zero, and occasionally you get a washer-blew-up day and spend all day fixing the problem.
That said, I sold all of them as of last year and put the money in the market, because RE prices were through the roof and I got lazy.
-W