I personally don't like mirrors at all. Too inaccurate, faster and more reliable to use your ears, supplemented with just turning your head. There is no headrest or door pillar on a bike, so looking back is easier than in a car. Besides, its not the cars behind you you need to worry about, its the cross traffic and oncoming cars.
I'm curious, have you tried one?
I have. 3 different types and sizes of handlebar mounted ones as well as one head mounted one, on various bikes over the years. I've always given them another chance when I come across one, and always end up taking them off. One thing was that you either have a very large "objects are closer than they appear" factor, and/or a very limited field of vision. Usually both.
But the larger issue is I don't really see the point.
You can't tell the difference between someone who is going to give you 1 ft of clearance and someone who is just barely going to nick you from 20ft away. The difference is just too small to work out such precise trajectories. So all it can really do is make you worry about every passing car, when getting hit from behind is extremely rare. I guess knowing a car is coming up behind you could warn you not to suddenly swerve left, but I find its just as easy to not ever suddenly swerve left. If you assume cars are there at all times, and ride accordingly, what difference does it make if there is actually one there at any given moment? The only time I think it would be useful is when you are going to change lanes to make a left turn (or to go straight if there is a dedicated right turn lane) but that comes up infrequently enough to just look behind you.
All that said, I don't think there is anything
unsafe about having a mirror, or any reason to avoid it - as long as you recognize the limited field of vision and "objects are closer" factor and never rely on it to ensure there
isn't a car if you are going to move left. I personally found I came to rely on them when I had them, and didn't always turn my head to double check, and that
was dangerous.