The wires to/from the light may not have been a hot and a neutral as you might expect, but what is referred to as a "switch leg" meaning they probably were both hot when the switch was on. This is how it would be wired if the main power came into the light box first, and the wires to the switch only switched the hot (black) wire that powers the light.
You'll need to run a separate neutral to the fan, and then use the same switched hot that the switch leg controls for the light as the hot for the fan. Use a tester to determine which wire to the switch is the "line" and which is the "load", meaning which is hot and which is hot when the switch is on.
Not as easy as you'd hoped.