it doesn't have to cost you "way more"
Make sure to find a good reliable contractor that gives you detailed plan what exactly is included with the house.
what floors, kitchen cabinets, appliances?
Ask detailed questions, such as what is not included in the specs? What extra expenses should I expect? Ask to talk to other people who built with the company and how much more they ended up paying.
If you are not going too crazy about upgrading every single feature you should not be paying much more than the offer you get.
For example if the builder budgets flooring at a rate for an average carpet, but you want all wood floors that would cost at least X2, you need to take that into account.
It's a good idea to spend very long time on planing the house and writing down exactly what you want to have inside and out, and then start building only when you feel ready. You will minimize the chances of having surprises while you build, or wanting to make changes that would cost much more than what you had planned.
