My starter house is currently a rental and has been for the past 2 years. I paid cash for it so it would be nice to move into it for retirement since the expenses are so low and it is very foot, bike, transit and car friendly (easy access to major freeways. Not the greatest neighborhood but not bad either, it's not a ghetto.
My only gripe was really the traffic noise. It's on a side street with no sidewalks, bus lanes or stores and it is not a major street but there are stop lights at each end of the road about a mile or so down so you still get a fair number of cars every few minutes, 3-5 at a time during the morning and even commutes as people use it as a short cut. Outside of that, it's maybe a car every few minutes After 7 or 8 pm it's fairly dead.
Of course the house is about 20 yards from the road, and it's an older house. Back in the 40's they didn't have much traffic on this road. In any case, the swoosh of cars passing drove me nuts after about 2.5 years. Actually I could not stand it after the first month and ended up having to live with white noise or a fan on all the time until I found my next house, though I eventually moved because in the summer, all the boom cars and motorcycles come out which is aggravating as well.
One thing I've considered doing is getting triple pane windows (currently double) and some soundproofing material/soundproofing that is done by a professional subject matter expert. Also, I've considered that if I move back into it, I could turn the back bedroom into the living room. It faces the back yard with a nice view of a huge yard and has lot's of windows. There is less road noise in that room, which is where I would spend the majority of my time.
Alternatively, I could just keep it as a rental. My tenants don't seem to mind the traffic noise. I could also sell it and use the money to buy a condo (top floor not too bad noise wise, right?) or a house far enough from the city/jobs that it is easily purchased with the proceeds from the sale of this one.