+1 for Mystic Seaport! I don't know if the aquarium is still there (I moved out of Connecticut 13 years ago), but it's lovely.
Also, Newport, Rhode Island is lovely, and just a few hours away.
New Haven isn't a super nice city, and sure, there's crime, but it's no worse than NYC honestly (but there IS a lot fewer things to do in New Haven than NYC). In NYC, I love the Bronx Zoo, the Metropolitan Museum of Art, and the Museum of Natural History.