Solar hot water is a good way to go, if you live in a sunny enough area. It even works in the winter if you get the right type (vacuum tubes, I think).
I did some calculations for ground source with a heat pump, and it never made sense over natural gas heat with current pricing in my area. Natural gas is so cheap, and electricity is relatively expensive, and the ground source didn't improve the COP enough to bring the electricity consumption very low. It might have been slightly cheaper to operate than gas, but it would never pay back the cost of getting the system installed (thousands for the heat pump/heat exchanger parts, and tens of thousands up to 100,000 to get ground source piping installed). If you live in an area with true geothermal, where you can extract heat from the ground without a heat pump, that's awesome and you should take advantage of it. But as far as I know, most places require a heat pump so you don't get free energy.
If you live in a place with cold winters, where it's cold all day, insulation is probably more important than thermal mass. Thermal mass is great for places like Colorado where it might get up to 60 in the day and the sun can heat the thermal mass, but then it gets cold at night. The key is that the sun heats it. If you have to heat the mass yourself, you might as well skip the mass and just heat the interior space. Thermal mass is good for things that provide free heat some of the time, like solar hot water. Using solar photovoltaic panels to heat thermal mass electrically is likely not a good use of resources. Solar panels are expensive and not very efficient, converting on the order of 15% of the sun's energy to electricity. If you are using them to replace an expensive energy source like electricity, it can make sense, but using them for heat is not usually worth it. You could use solar to run a ground source heat pump, but then you have the problem of a super expensive system with a long payback period.
I knew a family that lives off grid, and they have solar hot water for hot water and heat, with free wood to supplement that when it's colder. They use a tank of water in the basement for thermal mass (it looked like a hot tub, but I don't know if you could swim in it). Their electricity comes from 2 sun tracking PV arrays, and they have a huge bank of batteries (most of a room) which could power them through a few days of cloudy weather. They also have a diesel generator for times in the winter when it's cloudy for a long time, and they said they use it once or twice a year.
If you are living on-grid, you can use the grid as your battery for solar power, saving lots of money.
I've heard stories about super insulated houses that get enough heat from the occupants and regular electricity use that no other heat is required. If it got too cold, they would just play with the dog for a while or turn on a 60w light bulb for an hour.