Cash out your whole life policy as soon as you get term insurance in place. Life insurance is meant to replace your income if you die. If your wife works and you don't have any kids then you probably don't have a huge need. I have 5 kids and am the sole breadwinner in my household so I carry about a million in term life insurance between a few policies (work, military, private). At 32 and in good health I was able to get a 500k policy with a 20-year level term for $249 per year. So that's $4.98 per year per $10,000 of insurance vs. $61.20 for your whole life policy. In 20 years I should have enough assets in place to be self-insured, plus the need to replace my income will be less with most of the kids through college and the house paid off. You will owe taxes on any gains but if you add up all the premiums paid in and deduct it from your cash value you'll find that it won't be much. That's also an indication of how low the returns were. Assuming your policy is 30 years old the total premiums paid were $4,590 and that's only grown to $9,400 over that time. That equates to a return of about 4.5%. Had it been invested in an index fund earning say 10% those annual premiums would have grown to about $25k (a bit less if you account for what $25k worth of term life insurance would have cost per year over that time).
If you cash out the whole life policy and the taxable stocks you could payoff virtually all of your debt immediately. That would save about $1,122 per year in interest, not to mention freeing up probably a few hundred dollars per month in cash flow. I became debt free a few months ago and it feels very nice watching my bank account grow each month instead of seeing payments on various debts deducted.