As mustachians we have all felt the pressure to continue working when we would rather not. We've all dealt with incredulity from friends, disappointment from relatives and the general cultural thinking that only a working person is a useful person. Indeed, it seems clear from the multitude of threads addressing this issue, that dealing with wider society while living a mustachian/RE/FI life is the second biggest issue facing the community (the first, obviously, is "How do I become FI to begin with?"). The common advice on this problem - "Stay quiet and smile" - is very good as a social tactic but, does nothing to explicate or ease the "outsider" feelings that often accompany choosing FI as a goal.
If you feel these pressures or, are simply interested in how
America became obsessed with labor as virtue, I highly recommend reading Gordon Wood's The Radicalism of the American Revolution
This volume won the Pulitzer and for good reason as it comprehensively details the changes to American social society which occurred between ~1776 and 1816. Primarily, these changes included the dissolution of paternalistic monarchical ties, the rejection of leisured (aka RE) aristocracy and classical republicanism, the embrace of private interest in public policy and, most crucially, the elevation of labor from a condition of poverty to a condition of respectability.
This book won't help you deflect you co-worker's snide comments or build your stache any faster however, it will give you a great historical perspective on the early development of America and how those attitudes have trickled down into today's world.