I feel your pain.
DH and I each get "nights out" every week. He gets 2, I get 1, which makes sense (in my book) because he is the SAHP. He spends his hanging out with friends (they meet up in a particular bar, drink beer, and watch sports, scandalously stereotypical) and particularly in these colder/darker months (I often do stuff outside when the weather/daylight are better), I sometimes meet friends (last week I went to a movie!) but I also do scandalously extravagant things like stay late at work (I enjoy what I do, and having a quiet office and proverbially clean desk), go to quiet spot and eat a simple (bought or packed) supper while reading, and/or (brace yourself) go grocery shopping unfettered by other humans and time constraints. Ooh lala! Neither of us is "due back" at any particular time on these evenings, which I think is part of what makes them so relaxing (among the big downsides to parenting a young kid for me is the endless need to stick to a schedule for handoffs, etc.).
As for the actual travel stuff, would visiting grandparents (aunts, uncles, etc.) be an option? Someone you could leave the kids with and go out to dinner on your own (at least)? We did very, very little kid-free travelling during the first 3 years of DS's life and -- yeah -- having a kid along pretty much makes it "not a vacation" in a very important way (not that there aren't fun aspects of vacationing with kids, but the "getting away from it all" aspect is -- gone. We did, like @Useitup..., spell each other somewhat (take separate trips) and that helped, too.
Ours is, well, just one kid (so a smaller N) and older now (kindergarten), and it's getting better. Plus we're not dealing with everything you are, what with DH being out of the workforce. So hang in there, but also do what you can to stay sane while you're waiting for the light at the end of the tunnel.