Fireplaces do suck a lot of heat out. The only time you are gaining heat is when the fire is rip roaring. When flames are low as it's building up and while it's smoldering out with damper open you are sucking heat out of the house. Good glass doors that seal up tightly will help a bunch, but most still have a certain amount of air leakage around the perimeter, where doors meet, etc.
A good old open fireplace is sure a pleasure to sit around. Just be mindful and close the doors when it's going out so it's not sucking out too much heat as it extinguishes.