You can get humidity from a variety of sources.
As has been mentioned, concrete is pourous. Without a vapor barrier, your basement will absorb some humidity from the ground around it (particularly if there has been heavy rain and the soil is saturated). This will often cause small leaks and puddles in the basement and should be fixable by grading all walkways/patios properly, moving the drainage from the roof further away from the building, patching any big cracks that are obviously letting water in, and digging up/fixing blocked french drains around the home.
An unfinished basement (or improperly finished basement) will usually be humid in the summer. You get warm air flowing into the basement from the upstairs. Warm air carries more moisture. Basements are naturally cooler places, so the warm air hits cool things (walls/floor) and will condense. This causes mold/mildew and your musty smell. Unfortunately it's not always easy to see where the mold and mildew is building up . . . but your nose is a better indicator of problems than your eyes. Finishing the basement properly, or constantly running a dehumidifier during summer months are the best ways to fix this problem.