The only deterring factor for me is the rent controlled place you're at now. For that reason alone, buying would be a bad idea. You're much better off with those funds invested, from a financial perspective.
Your cons, honestly, sound pretty typical and even the sum of the parts don't add up to counter that rent controlled pricing in any way. There will always be vagrants and punks in any large city, and any neighborhood that close to downtown is going to have that. I don't see why you need a backyard with all of the beautiful parks in SF.
Long term, I think you'll grow old of being so close to everything. Unless you plan on constantly draining your bank account to the shops around you, I don't see the point long term. The only thing nice about living in a dense neighborhood is proximity to transit, grocery stores, and maybe parks and local facilities. There is no allure to me to be next to coffee shops, cafes, or clothing stores as I know I'd never frequent any of them anyway!
Again, logically, buying is way more expensive than your rent in this case and I don't think you're getting a whole lot more for the money. That said, I agree with the "YOLO" mentality of another poster to an extent. It doesn't make sense to be unhappy with a housing situation with that kind of income. Considering you both could retire yesterday and live anywhere in 99% of America, why skimp on something so essential? However, if you're not going to buy EXACTLY what you want in a house, which could be well over $2M, then there's no point in making the switch. Trading trade offs for trade offs doesn't make much sense.
So to answer your question, of course, yes, you can afford it. But as you can see there are many intrinsic values at play too when deciding where to live.