No, I wouldn't apply a blanket rule of "pay off non-deductible debt first." Instead, pay off debts starting with the highest interest rate first, taking into account any tax deductions that may apply. For example, if you're in the 25% tax bracket and you have student loans with fully deductible 5% interest, your effective interest rate is actually 3.75% (75% of 5%). Compare that effective rate to your other loans to see which ones you should prioritize.
Also there's some rate below which it makes sense to invest in the market instead of paying off the debt faster than required. Where that line is depends on your personal risk tolerance and aversion to debt; 5% is a common rule of thumb. Pick a number that makes sense for you, aggressively pay off debt higher than that rate, and then switch to increasing your savings when you only have loans with a lower rate remaining.