I think there's just too many inefficiencies with specialization: taxes, transportation, recordkeeping, liability, regulations, transaction fees, and probably others I haven't thought of. People nowadays treat these as just "costs of doing business" but they don't exist if you insource. That sets a high threshold for an outsourced service provider to exceed if they are to deliver better value than you can make yourself. As an example, I can make alcohol for myself much cheaper than anyone else can make it for me, mainly because I don't have to pay alcohol and liquor taxes when I make my own. I can cook my own food cheaper than anyone else can cook food for me because I'm not subject to food safety regulations, sales taxes, payroll taxes, minimum wage, commuting costs of employees, and other things that restaurants are subject to.
It would be interesting to look at the above inefficiencies and see if these costs have been increasing or decreasing over the past ~200 years, and if they have become more pervasive (for instance, the national income tax in the US shifted the tax burden from a few industries to everyone, how did that change things?)