I think I am in a very similar situation to SilveradoBojangles and generally agree with her post (and so would my wife). I think there are definitely things you can do to try and feel safer in your neighborhood before deciding to move.
Find out how dangerous your neighborhood truly is compared to neighborhoods that are considered nicer. As others have mentioned, property crime may very well be higher in those "nicer" neighborhoods. Regarding violent crime, keep in mind that most rapes and murders are not committed by strangers. The odds of being a "random" victim of a violent crime are very small, and can be further minimized by taking some precautions. The thing about humans is that we suck at comparing relative risks. You are far, far more likely to be in a serious car crash which kills or maims you than to be murdered or raped by a random stranger, yet you are asking whether you should move to a nicer neighborhood, not whether you should stop getting into cars. As a woman, you are far, far more likely to be killed by your significant other than by a random stranger on the street. I don't know how you commute, but if it's by bike or car, it is entirely possible that the effect of keeping your expenses low by staying in your current neighborhood, which allows you to save more, which allows you to FIRE earlier, which allows you to stop commuting by bike or car sooner, which lowers your lifetime risk of dying on said commute is far greater than the added danger, if any, of living in your current neighborhood versus a "nicer" neighborhood.
At the same time, there are things you can do to feel more comfortable in your neighborhood. Get to know your neighbors. Bake cookies for them if you need an excuse to talk to them. Set up an informal neighborhood watch. Go on walks around the neighborhood during the day and say hello to people (with your husband or a dog, if you feel uncomfortable on your own). Go to community meetings. Get involved. Even in some of the "worst" neighborhoods in America, you will find that the vast majority of people are not that different from you. They want to make a living, raise their families, and work to ensure that their kids will have it easier than they did. How long have you lived in your neighborhood? Time can have a huge effect. I was nervous when I first moved to our neighborhood, but the more time I spend here, the more people I get to know, the safer I feel. And the more I love living here.
Which brings me to a touchy subject, and if you live in a predominantly white hood, ignore it. Examine your own prejudices. It's ok, everyone has them. Does the color of your neighbors' skin affect how safe you feel? Are there situations in your neighborhood that make you nervous (noises that sound like gunshots, groups of young men hanging out, etc. etc.) , that wouldn't make you as nervous if you lived in a "nice" neighborhood and they happened there? If someone sketches you out, ask yourself whether you would feel that way if they were white. Don't suppress or ignore your prejudices, but confront them and challenge them.
But yeah, if none of this works, then move. You live in a LCOL area and it sounds like you can afford it.