I think what your SO is referring to is that you will get to keep more of the rent if the duplex is paid off, because the mortgage payment on the duplex is bigger. So he's thinking he'd rather not have the duplex payment to worry about every month than the single house payment. That actually (kind of) makes sense, but he's not thinking of the bigger picture, which is it's one pot of money, and you want to pay off higher interest first unless there are tax advantages.
Basically, it comes down to the fact that he wants to pay off more debt and have fewer financial obligations before you cut back your work hours. That may result in paying more interest in the long-term, but as it would keep you working longer, it's probably a financial wash or perhaps even net positive (because you're bringing in more income by working). That's not an unreasonable stance, in my opinion, but something the two of you will need to negotiate.
I know exactly what I'm trying to say, but I'm afraid I'm not being very clear. :-)