If I'm understanding you correctly, you want to refinance your mortgage and use the refinanced loan to pay off the personal loan, resulting in one mortgage loan in the amount of $23,000 at 2.64% for five years. Did I get that right?
Assuming I got that right, it seems like its a good idea in theory to do the refinance, pay the minimum on the resulting mortgage, and max out your investments. It's probably a safe bet that your investment returns would exceed the 2.64% interest you pay on the loan. But... the biggest question mark I have is in the cost of the refinance. Closing costs, including appraisal fees, recording fees, closing agent fees, attorney's fees, etc., as well as the early termination fee could be more than you would save in interest.
You need to find out what the closing costs would be and run some numbers to see if the interest savings are worth the costs. You might be better off paying the minimum on the 4% mortgage, throwing everything at the 8% personal loan to pay it off ASAP (a guaranteed 8% return is probably more than you can count on in your investment account), and then max your TFSA when the personal loan is paid off. But we can't really help you figure out what would be best without more information.