Just a few thoughts.
N=1 is always a good idea. Your body and its history are unique.
From a biologist's viewpoint:
Alkaloids* are plant responses to insects, they are toxins Unfortunately we like them. Caffeine is a good example. We are just a lot bigger than insects, so the dose is lower. Nicotine is also a plant alkaloid. In concentrated form it is also toxic, and is easily absorbed through the skin. Back when it was a commercial insecticide there were always deaths and near-deaths from spills**.
Plants don't like predators eating their seeds. If their main seed predator is a mammal (like us, cattle, horses, etc.) then they are more likely to contain chemicals that are not good for mammals. If the seeds are more likely to be eaten by birds, then the chemical defenses are more likely to be aimed at birds. Same for insects as main seed predator. Not guaranteed safe for mammals, odds are just a bit better.
Still on seeds - fruits are a plant's methods to move their seeds to a new home, with fertilizer. So ripe fruits where the seeds are protected are enticing not damaging. This has 2 implications - unripe fruits have seeds that are not ready to be released to the world yet, so the plant will make those unripe fruits unappealing - nasty tasting, poisonous, etc. We get around some of that with cooking. Second, the seeds inside those yummy fruits have some sort of protection so they don't get digested when the fruit is eaten - they are too small or too large to be crushed by teeth, they taste bad, they are toxic. After all, almond flavouring is partly cyanide, so almond seeds are edible but peach and apple seeds are poisonous, just a difference in the amount of poison. All of this to say, ripe fruit is better than unripe fruit and seeds are all potentially iffy to dangerous. Of course people can develop sensitivity to fruits and seeds, look at strawberries.
And even fruits that we think are OK may not be. Grapefruit interacts with some meds - I read recently that it inhibits the action of the cytochrome P45 enzyme in the liver, the enzyme that does a lot of metabolic detoxification. So the cP45 enzyme doesn't do what it should do to a medication, a lot of medications are inactive as they are taken and are activated by the liver.
I would say that plants invented chemical warfare, except they didn't, bacteria did (think botulism as just one of many examples), followed by plants and fungi (aflotoxins in grains and peanuts are a major health issue). Prey work hard at not being eaten, so anything we want to eat has protections against us, unless it is a very domesticated food that we have bred the protections out of. Like canola oil, mostly. Of course one of the big jobs of our liver is to detoxify all these protections.
Dairy - the original casein molecule gene had a mutation relatively recently. A2 casein is the original, A1, is the mutation. A lot of people who react to dairy, when it is not a lactose intolerance, are sensitive to A1. So casein products and regular milk can all trigger them, while they may be fine with A2 milk. Or if they are sensitive to all caseins (and/or whey) then they lose dairy. For some people they are fine with ghee and not butter because of the milk solids.
*illustrative story - agronomists were looking for a coffee species that was naturally low in caffeine They found one but it was high in other alkaloids and the coffee tasted horrible, alkaloids will do that.
** When I was a grad student I met an older chemist whose lab partner, back when they were grad students (so the 1920s or 30s), had spilled concentrated nicotine extract on himself. Went into convulsions and almost died. Fortunately it wasn't quite enough to kill him before his liver detoxified it, and he was OK.
