The most important thing is that both your assets and income are tied to the same company, and diversifying away from that is a good idea. Since you have prior year capital losses, selling won't impact the tax you pay next year. Sounds like you should diversify soon, based on what you've said.
Main drawback is that you can use $3,000 / year of capital loss against your income. So the loss you use up is coming from some distant year's income. But the larger the loss relative to $3,000 the less that matters.