Either way, it does keep up for the most part and doesn't depreciate or require maintenance like other real assets. The downside being that it's a non-productive asset and the premiums for physical can be a bit high.
I have a few ounces of gold and several hundred silver dollars in physical form. I find them to be a pain as they take up room, you worry about loss, and they just sit there. I collected coins 30 years ago and have been slowly selling off them. Sold 15 ounces of gold at $1,722 only to watch it go up over the next month. If I was going to have more than 1% of my networth in metals, then the cost to secure, insure and protect the asset would be cost prohibitive. At this point I am taking on risk by storing them at home vs. paying for secure storage and insuring for loss. With transaction costs on the buy and sell side, gold/silver in physical form is not a very efficient investment. Again, maybe it is good if the world comes to end but really food stores, farming skills, hunting skills, and gun and ammo are better investments for the black swan events.
I think you are discounting the cost of owning gold if you were adequately securing your investment. I think many just self insure and don't fully understand that they are taking on risk. If it is less than 1% of your Stache then the risk is reasonable. If you are doing a PP with 25%, then I think you are very exposed to have it in physical form without insurance.
YMMV