If you want to push your tolerance even lower, 1) lift weights, and 2) spend more time outside so it feels warmer inside. ;)
This is our approach: we have a home gym outside, and do a good lifting session to bump our "internal" thermostat. We are limited though by having a geriatric dog who we love very much, and a complaining Brother who... well, we don't want him to freeze, anyway ;)
We keep it at 64 usually (DH and I both have weird schedules and are home most of the time), and turn it to 66 when Brother is home all day (rarely happens). The past few days Dog has been recovering from surgery, though, so we've had it up at 68 since she was shivery and sad.
If we remember, we will sometimes turn it down to 62 at night, but since Brother turns his space heater on high then, it tends to defeat any savings- the gas furnace is much cheaper than the $$ electric space heater, even for 1 room.
How many of the people who are fine with indoor temperatures that follow outdoor temperatures bike to work, or regularly do outdoor activities all year long? How many of the people who think that children will die at 60 degrees F spend the entire winter in climate controlled buildings?
DH and I work indoors, but for DH that is from home most days, and for me, it's only 2 days per week. But we walk to groceries and work out outdoors, so even 64 is toasty inside by comparison.
It's funny about people thinking the cold is so brutal for children and all that. Do you kids not play any sports outdoors, or hike or camp, or go outside at recess? That's just a baffling concept to me. We all spent more time outside than in as children. Even if it was frozen outside, or (much more common) raining, Mom would just hand us the appropriate jacket and we would still go outside to play.