I don't know what's best for you but I hope these notes will help you figure out.
I didn't get to FIRE until I was in my mid '50s. That's when I started my PhD. It happened because I wanted to do "work," that is, spend a solid chunk of my time and energy, doing something I think is socially useful. In my own case, that meant relevant in light of climate change. I realized my strongest skills are the kinds of things one does in school. I went to see people who do research that interests me, by good fortune right here in my town. Bingo, great fit, better than volunteering. They said the easiest way for me to become part of a research community is to get a PhD. Mostly what that amounted to is doing exactly the kind of research I want to do anyway. So here I am.
My social life still is largely with people I knew before school. It's even richer now, though. At school I get to be around people of all ages, or at least all ages 18 and up. I like that. I didn't even realize it at the time, but at my old paid jobs I spent most of my time with people at similar career stages as me, which meant more or less my age. My 23 year-old office mate is a dear buddy I'd probably never have got to know well otherwise, for example.
I've gone back to school a couple times already. This time the first year back was crazy busy. I was learning so much that I want to know and I knew it was temporary so I didn't mind. After that, being FIRE has given me a huge degree of liberty and I can't recommend it highly enough. I can do research on the topics I choose rather than topics someone has got funded. I spend my time on research rather than on chasing funding. Being a (paid) teaching assistant was really fun until I realized it was eating up my research time, so after a couple semesters of that, I quit that too. I get to work hard some weeks because I want to and take lots of time off when that's what I want. I'm not in a rush to finish in order to get done while I'm funded so I don't have that pressure - though I'm moving along at a normal degree-completion pace despite not stressing over it. I don't worry about future employment, which most of my fellow students must.
Being older than average is a big asset for grad school. You may have more perspective. You may have clearer goals. You may already know how to write a long document, or speak in public, or organize a months-long project, or have some other skills that younger students sometimes struggle with. You may have already know how to live a social life that's pretty satisfying. You may not be working 3 jobs to pay tuition. Etc. Oh, about the last part, tuition can be pricey. Depending on your field, you may find, as I did, that a stipend for grad school, tuition plus some cash, just sort of finds you. The salary part goes in the ol' savings account / investments since I don't actually need it.
And yes, this was a big switch for me - hard sciences, mostly physics. I focused on history and sociology way back in undergrad.
Chase your dream!