This stuff is hard, no doubt. When people say everything changes after having children, it's no joke.
I went through similar feelings. I had my own business when my kiddo was born last year, and my work was flexible enough that I could do a lot of work at home. DH, though, had no real concept of how incredibly difficult it was to: a) care for the tiny human 24-7; b) do actual work that would produce income; and c) do random household chores throughout the day. All at the same time. However, when kiddo was about 3 months old, I had to work out of town for a week. DH was a SAHP for a week, and the first thing he said when I got back was, "I completely understand why there are times when you go 3 days without a shower and the house is a mess." Gave him a whole new appreciation for the difficulty of the situation.
However, my emotional gymnastics of whether or not I wanted to stay at home changed many times, due in part to what ended up being severe postpartum depression, which didn't actually being to clearly manifest until about four months after giving birth. It all ended with me shutting down my business, being unemployed for a while, and now currently being gainfully and happily employed full-time. Kiddo is in daycare part of the week, watched by the beloved grandmas for the other part. DH and I carpool to work, so no one is ever left to fend for themselves with what I am sure you know can be quite a strong-willed timesuck that is a child. But we love those little gremlins all the same.
You are not a bad mom if you want to work full-time, or part-time, or stay at home, or go to the gym, or have a drink, or interact with adults. If you do the things that make you feel like a human, you will be more human, and your babies will benefit as a direct result. If your cup is empty, you can't fill the cups of others.
Finally, I might get some flack for this, but listen compassionately to what your husband has to say. Just like you had ideas of what your family was going to look like and feel like, so does he. Yeah, they don't have to go through all the same stuff we do, for sure, but they are part of the team, too. So as you are having mom guilt, it's quite likely he may be having husband guilt. My husband felt worthless at times because he couldn't fix me or our situation and all he wanted to do was make sure everyone was happy all the time. Impossible. Consider going to therapy, consider marriage counseling, consider all the help. Big, huge, internet hugs to you. It's hard, but it will get better - go with your gut and communicate with your man.