I think your response to his comment (getting upset about it and staying upset about it for hours) is much more interesting than the comment itself. Whenever I find myself getting disproportionately upset about something that someone else did or said, it's always because there's something deeper going on than just the surface event that set me off. Have you taken some time to introspect about why this person's comment bothers you so much? Is it because you care about his opinion and want his respect and approval? Or maybe you subconsciously feel like he's a little bit right and you SHOULD be working? Or some other explanation? I think if you can get to the bottom of why you're reacting so strongly to what he's said, it will end up being useful for your own personal growth. Not to mention your sanity and your happiness.
+1 I would like the OP to respond, since I see a lot of me here.
Our society bond is tied to our profession. It is almost always the first thing we know about someone. First question we ask anyone we meet.... "What do you do?". If we think it might be someone without a job, we might ask.. "Do you work outside the home?", which is really the same question. I still struggle with this one. I know there is a thread that discusses it, but it is still unresolved for me. I started working at my Grandfathers business when I was 10. He worked until he was 86, died at 87. My father is still there at 77. I can not deny that there is a constant pull to work at something.
Even when talking with your FIRE peers(like on this forum), you see that we still feel the need to define what we DO (I am an investment manager for myself, or I volunteer, or I make mittens for puppies, or whatever). When I FIRED, I was so burned out I didn't want to do ANYTHING. So I allowed myself to do just that, nothing. For over a year. I still do not have an answer to the question "What do you do?". I am now approaching the point where I want to do something, but I must admit that part of me wants to do something just to have an answer to the question.
Not having to work confuses people. When I FIRED, I let people know I was "taking a year off", even though I knew I never had to work again. There were people that could not even comprehend a 1 year hiatus from work, let alone no need to work for 50+ years. I feel a lot of guilt for my situation. Yes, we saved more than we spent for almost 25 years and have "earned" our position. But we were also given unbelievable opportunities that we capitalized on. We did not at all do it "on our own". Without the help, we would still be working, though still planning to FIRE earlier than most.
Is it possible that your reaction was a combination of dealing with your neighbors jealousy as well as your own guilt for being so far ahead of others as it related to fiscal independence? Do you also feel a need to have an answer to "What do you do?"