I left work in March to be a stay at home...uh...person. I don't have kids.
The issue here is that going from a difficult job that you aren't enjoying to being a full time caregiver is not a break, it's a shift in responsibilities.
I'm personally taking real time off. I go for long walks, read a lot, cook complicated recipes, and generally do whatever the hell I feel like doing. Okay, that's a bit of a lie, I'm also doing an intensive graduate degree to keep myself occupied, but honestly, it's marginally more effort than the avid reading I was already doing, and is more like being part of a really organized book club.
If you need a break, then take a break, but make sure that it's actually a real break, as in, keep the nanny.
Quitting to become a full time caregiver isn't giving up work, it's giving up your particular job to get paid *exactly* what it would cost to pay your nanny.