Worst case scenario is you pay extra to the mortgage, lose your job, and don't have sufficient savings. Then you are well and truly screwed, because you CANNOT get your $$ back out of the house to live on during unemployment.
So, the solution is to take the extra $$, put it towards savings/investing instead of toward the mortgage, then if there's job loss you will have lots of money to get you though any rough patch re: employment.
The only time paying the mortgage makes ANY sense at all (from a risk standpoint) is if you have enough cash on hand to pay it off completely. And even then, you're better off investing the $$ as you'll end up getting rich much faster by investing than by paying off the mortgage. See the link in my signature to see exactly how much. When I ran my numbers, it was a breathtaking difference.
Of course, I'm risk averse in some ways, so my plan is to hit FI using all extra money as investments until I hit $1.5m in the bank. At that point I should have only $250k on the mortgage, so I'll pay off the mortgage with $250 of my $1.5m and I'll have $1.25m left. I know from a pure numbers standpoint that EVEN THEN it's not optimal to pay off the mortgage, but it'll help me sleep better at night as I have job instability that seems to pop up at the most inconvenient times.