If it's in bad shape I would take it out. Have you gotten any quotes on the cost of restoration? Or the cost of a metalbestos install?
I like the more modern look of a woodstove with a pipe exiting vertically through the ceiling. No mantle, nothing on the wall, just the code required hearth (floor protection). Like a Jotul catalog ;-)
It can be single wall from the stove to the ceiling and then it's double wall up and out through the roof.
The portion of the piping that is above the roof is also easier to maintain and to flash properly. Brick chimneys are prone to leak at the roof. The roof flashing is embedded into the mortar and over time it breaks down.
I believe it's easier to clean a metalbestos chimney as well. I'm no chimney sweep, but I go up on the roof every year, remove the weather cap and clean it with a six inch round brush with an extension pole. Easy.
I've taken down a few chimneys, it wasn't difficult. Contain the dust, use fans to pull the air/dust to an open window if possible, use plastic to wall in the work area (slightly long 1x2 or equivalent can be bent into place to hold the plastic sheeting to the ceiling).
If it's not a major architectural element and you don't like it then just get rid of it. They're really only for looks these days.
One element in their favor is their value as a "thermal mass" that holds heat, but occasional use wouldn't really trigger that.
One other thing to consider; you can also install a metalbestos chimney in another location if that works better, you're not locked in to the brick chimney location (although the nearly ready made chase is tough too overlook).