My brother moved from DC to Baltimore and he regretted it. He spent about four years in Baltimore - he was robbed twice, followed a bunch of times, had his car broken into twice, etc. Baltimore is a dying, violent city.
Baltimore is not a dying city. The population has recently started increasing (slightly) after years of decline. It certainly has its problems and its poverty, but it also has lots of young people moving in, lots of people on all sides working to fix things, and lots of industries. It's no Detroit. It will never be Detroit, thanks to the proximity to the federal government and all the jobs that brings.
Heck, my favorite area of the city now sells $8 coffee and $10 ice cream. (I exaggerate, but only barely). Hampden is becoming downright bourgeois, Harbor East is blowing up, North Station is starting to happen, plus we've got tons of colleges and artists and museums and interesting people here. It's a great city for landlording or flipping real estate, if that's your thing.
(Also: The Inner Harbor is basically the lamest area of the city. It always blows me away when people talk about how nice it is. I mean the ships are OK I guess but other than that it's all chains. Explore some other areas!)
That said, yeah, safety is a concern. I've actually never lived in the city, but many of my friends have lived there for years. Some have been mugged. None have ever been seriously hurt. I do think you have to have the right constitution to not get worn down by the constant threat of crime -- but I also think in the suburbs you only have the illusion of safety, anyway, at least in terms of home invasions. Shit happens everywhere. I've spent a lot of time walking around the city, including the desolate neighborhoods around Johns Hopkins Hospital. Only rarely have I felt unsafe, and nothing bad has ever happened to me.
Or you can just live in the suburbs and commute in. I love the suburb where I live, thanks to its walkability and proximity to trails. It's probably a lot pricier than western Pennsylvania, though.