Possibly it doesn't make rational sense, but that's the way I feel. My uncle was a geologist, and as a kid had all sorts of neat things like geodes, fossils, and even a meteorite he let us see and play with. Gemstones are created in earth, in a particular time and place. For some gemstones they have to have very unusual environment and chemicals co-occur in order to be created. I suppose you could make a fabricated fossil that looked identical to a real fossil. But the real fossil, or meteorite, or gemstone holds more fascination to me. Synthetic gemstones, whether diamonds, corundum, and silicon carbide, using petroleum energy, in controlled environments may actually be more useful (used from everything from space station windows and drill bits) than the naturally occurring varieties, which are basically (beautiful) "accidents" of nature.
I'm also an environmentalist, and so I also know my enjoyment of these things is contradictory with other values I have, as others have said, tons of earth has to be moved, poor working conditions to find these things.
I know I have a problem (lol) I just try to minimize the damage.