Move.
Not because the old school is overrun with 1540 SATs. Because your kids don't have a peer group where you live now, and because the current school district has no options that will allow them to continue to grow and develop and challenge themselves.
I would divorce the choice of moving from the decision to move back to your own town. IMO, your current situation is not great for your kids. Sure, now, they're fine, and with parents like you, they'd probably get through no problem. But being the odd one out is hard -- particularly being the nerdy odd one out, as those kinds tend to get bullied (AMHIK). And kids who are smart need to be challenged, both to develop their own potential and to keep them from going off the rails. You know the old cartoon saying, "if only he'd used his powers for good"? Yeah. Smart kids are going to find a way to use their smarts, and everyone's a lot happier if that smart kid is given some way to put those smarts to good use.
And it's particularly good when that productive way of using smarts comes from someone other than the parents by the time the kid hits adolescence, because that's the time when parents suddenly become stupid and parental ideas are summarily rejected as stupid and out of touch. It can come from a teacher, from a coach, from a youth group/club leader, or a hundred of other ways; you never know what and who is going to click with your kid until it does. Which goes back to why smart teens typically do best in an environment surrounded by people and options that are good and healthy, vs. negative and destructive. There are no guarantees anywhere; too much pressure can be just as harmful as too little. But when you surround yourself and your kids with their tribe, and give them opportunities to grow, you are giving them their best chance to use their not-inconsiderable powers for good.
IMO, the 1540 SAT thing is infinitely less important than how you've described the people there. Those are your kids' people. The school itself is almost immaterial at that point, as long as it has at least some academic choices that will challenge your kids -- doesn't need to be the best school in the state, doesn't need to have only brilliant kids, just needs to have a path that the smart kids can follow to develop themselves and their interests. IOW, it doesn't need to be a great school, just a good enough one; it's the people in it and around it who matter most.