First of all, I find it amusing that you call a town with a population of close to 100,000 a "small-town", but you currently live in NYC, so it probably seems small to you.
Anyway, I don't know anything about this town in Texas, but I have some familiarity with small, conservative towns. You'll likely be surprised about how open-minded some of the "conservative" people are. Remember, you are married to a doctor and you will be a stay at home mom to three kids. You're not breaking any molds, or any rules, and you will likely fit in with your neighbors. That might be a positive or a negative for you. In a town that size, there will be other professionals, and there will be other liberals. And there will be other people with tattoos. There might even be homosexuals. Gasp!
Unless you are staunchly atheist, vegan, and your tattoos are huge and seriously grotesque, you shouldn't have many problems. In a town the size you are considering, there probably won't be too much religion. There won't be just one church where everyone goes, and they notice that you're the only ones in town that aren't there. There will also likely be some diversity, but if you're white that doesn't really matter.
Do you see family regularly now? It sounds like this move will allow you to afford a few trips a year to visit family and still pay off your loans before other jobs would allow you to do the same.
I think for you, one of the big reasons not to move is the "small-town" aspect. Have you visited the town? If so, did you like it? If not, you should before you move. Take a trip there for a long weekend, or a week, just to get the vibe. Is the town walkable, bikeable, etc? Have you or your husband lived in a "small town" before?