I too grew up poor and have never completely banished my inner bag lady fears. My overall approach to achieving happiness, freedom, etc. is to plan for the negative events but hope for the best (or even the average).
I am friends with an older couple who retired in their 50s right before the Great Recession/housing market collapse. So they experienced the full effect of sequencing risk when they retired. I found their experience to be both inspiring and cautionary. They owned a house in a HCOL area with significant equity. They had retirement assets that they did not plan to access until later. I believe one of them had a small pension from an earlier job. They bought a small cabin in a ski area for their retirement home with the plan to sell their main home, make a modest addition to the cabin and live off the remaining house proceeds until they could access the retirement accounts.
Well, they retired and moved but the bottom fell out of the housing market before they sold their former primary house. When they realized they could not sell it without wiping out all the equity they had counted on, they rented it out so it was at least cash-flow neutral. Their retirement accounts dropped nearly in half so they did not want to make withdrawals at a low. They got part-time seasonal minimum wage jobs at ski resorts (which also gave them free ski passes). They took SS at 62 instead of waiting until later as they had planned. They did not do any travel beyond local camping trips. They did not do the addition to their retirement home. They lived like this for several years until the housing and stock market recovered. At that point they sold the old home, did the addition, started to travel again, etc.
When my friend told me this I found it inspirational because we had been friends through all those years and all I saw was 2 happy people skiing, biking, hiking, etc. while living in a beautiful area. Neither ever complained about the things they weren't doing. Also, they ultimately did have the total retirement experience they envisioned. They just had to wait a bit and they did the waiting while retired, rather than while working at high stress jobs.
I also found it cautionary because their ability to get through the recession was based on having the ability to meet their basic needs even after their "stash" had been essentially cut in half. They did have a relatively low cost place to live. They were both healthy and so could work at jobs that included a physical component. In some way (I think through a former employer) they had access to low-cost health care. Their main source of enjoyment (which I think is as important as shelter, food, etc.) is outdoor activities that are free given that they already owned any needed equipment. They had the mental flexibility to change their plans based on the external financial reality.
Of course it is possible to dream of combinations of terrible events that no one could get through. But it is possible to plan for some, especially events that are not that uncommon. If you plan so that you will be "OK" if one person gets sick, if the economy tanks, if a natural disaster causes a loss, etc. then you will be more than fine most of the time when things are just cruising along normally.