When it comes down to it, you can think of your house as a black box. Heat escapes, and you have to replace it. You pay only to replace the heat that escapes. (If that doesn't make sense, consider that if your house was perfectly sealed and insulated your heating bill would be $0).
So then, it turns out that the rate at which heat leaves is proportional to the difference between your house and the outside. If your house is warmer, heat leaves faster, and you have to make more heat to replace it than if your house was cooler.
All the other effects (length of time taken to heat the radiator back up, etc) are red herrings.
Short answer, the higher your thermostat, the higher the bill, on an ongoing basis.