I've worked in environments like this, where if you aren't in the office after hours you're automatically pegged as someone who doesn't work as hard. I even fell into this trap early in my career, working insane hours and weekends, just to feel like I was pulling my weight and keeping up with my peers, all of which was encouraged by managers.
At some point though I moved to a new job, where the culture focused more on how do you structure your time so you aren't working all hours. The mindset was totally different: If you feel like you have to work all hours, then maybe we need to examine the work and figure out how it can be delegated better, or if there are more efficient ways to do it. Sure, I still pull the occasional late night or weekend, but its night and day vs previous jobs I've had.
Now, when i see people sending me emails in the middle of the night and its not for something that is urgent, its almost a sign of dysfunction to me. I'm no longer thinking "wow, hard worker"!!. I'm thinking "wow, that person/organization must really not have their shit together".