I would absolutely remove it if I didn't want or need it. I think asphalt just continues to look worse the older it gets as more cracks develop and weeds start growing up in those cracks. I would absolutely not DIY it for a couple reasons. The biggest reason is disposal of all that asphalt and how you get it from where it is on your property to there. By the time you pay fees, trash out personal vehicles, take up space in a landfill somewhere, etc., it would probably be cheaper to just hire someone with a skid steer and a dump trailer to spend an hour doing it for you. What I would do DIY is just being your own contractor. So after you arrange to have it removed and hauled off, you can arrange to hire someone else to bring back some top soil, spread it out and then seed it over early this spring.