I think @shinn497 has some excellent points throughout this entire thread re behavioral economics on both a micro and macro scale. I'm glad to see some of the heavy-weights of this forum join in the discussion.
The thing with using behavioral economics is it breaks down on a micro-scale when n=1. An individuals (or single economic unit) behavior can easily be understood or modified. This is why the whole mortgage leverage/CC use/ease of transaction/use of "lazy" index portfolios, etc debate turns into personal choice on a micro-level. Using them IS more efficient, IF used correctly and the behavior of the individual can be predicted or modified to ensure efficient use.
The impacts of the availability of these tools on the macroeconomic situation is another thing entirely. Many on this forum have actually argued (and I agree with them) some of these tools may have been a contributing factor to the elevated CAPE median we have seen since their use has been incorporated into the economy. So to say they have no macro impact is probably not correct either. To argue there may be some extra value in companies excluded from lazy indices (due to market cap, or whatever) may be valid. To argue the increased use of CC/liberal issuance of mortgage debt has caused the average household to spend more and service more debt is also a legitimate assumption.
Bottom line, OP has some really insightful thoughts on behavioral economic from a macro-level. However, instead of basing personal choices on heard behaviors, to avoid herd mistakes, one should use this insight to increase personal gains on the micro level. IOW, understand how these tools influence group behavior and use this to your personal advantage.
correct OP keeps pointing out that everyone is jumping off a bridge so if he drives on that bridge he'll have to jump off too. once we understand poor behaviors we can adjust ours using data to govern our decision instead of emotion.
which you would think a data scientist would be interested in and more inclined to do.
Going to the bridge Analogy, I would start with a prior.
Lets say I highly believe that that you jump off the bridge if you go on it (there is no conditional for going on the bridge here, assumine it is marginal in the prior). Lets make this prior 70% or something.
I'd model it using bayes rule with some likelhood. The likelihood has all of the factors I suspect could involve you jumping off the bridge. The car you drive, the speed with which you jump, etc, etc, These go into some vector that is conditioned on the outcome of you jumping off (the anti outcome is you don't jump off)
Now, as a good bayesian, I can assume a prior, but I don't assume a posterior inference until I have some data. So lets say I observe some examples, model it, etc. etc.
What I would end up is a conditional on each factor. E.g, given each factor, what is the probability distribution of you jumping off. Lets assume everything is gaussian so it would be some value and uncertainty.
As a Data Scientist, this model doesn't guarantee if I make it across the bridge. I can isolate some factors, but what if my data set is too small, and my posterior has too great of uncertainty. What if I come to the conclusion that there is some bias in the dataset I am inferencing of. What if I look at other data sources and have contradicting results. etc. etc.
Notice, I would not naively calculating the probabilities and making a decision tree and saying at the end I have a perfect understanding of what will get me to live or not. I'm not assuming perfect confidence
Having the data alone, doesn't necessarily mean you can make a good model. And, even if you make a good model, that model might give you a result you disagree with. Modeling is hard, especially when talking about human behaviour.
Going back to the finance world. There are a LOT of variables here that I don't understand and I can't properly infer. This is doubly true when looking at others' situations. I know what people claim, what they project, and what they expect. And I even know a couple of results and that is good. But we are talking about financial expectations that go over decades. Given what I don't know, I can't look others'situations and extrapolate to myself with perfect, or even very good confidence. But this is more due to my own prior than anything.
So yeah. I'm all for the data. But having data doesn't mean you can make a perfect decision. So, in the face of this, I have to go back on my feelings. I mean I know they are irrational, but that is ok. If they cause me to err on the side of caution, I don't think I can do wrong.