I don't really talk about our finances much, but there was a FIRE group at work that used to get together for lunch and chat sometimes. I was pretty open with my friends there that I wanted to FIRE -- but I'm also a lot older than most of my former coworkers, so that helped them not question how I managed to save and invest. I think in their eyes, I'm of the generation that is supposed to do things like that.
If it comes up, I generally just say something like, "We've been working hard for 30+ years and saving our pennies, so...."
And if that doesn't work, I point them to the JL Collins stock series and to the MMM site, and I say, "Let me know if you have any questions!"
They never have any questions. :) And generally they let it drop then, because if they don't, I'll just ask if they've read through the material I sent them. Works like a charm.