We are actually just in the process of buying one as our post-FIRE home. Our thinking is to rent out the garage apartment on those occasions where we want a bit of extra income and are willing to put up with neighbours. We don't want anyone in there permanently because we are buying this property for peace and quiet (i.e. for its lack of neighbours).
If you are still paying off a mortgage I think it would be a great way to get some extra income. What are your downside concerns?