Did you use sanded grout? Did you mix with water plus an acrylic additive? If you scrub at it gently, is it coming out from the grooves, such that they are wearing down a lot?
You shouldn't see more than a small amount of material on your sponge, and only if there was a bit of haze on the surface of the tiles.
Regarding sealant, it will help to keep the grout clean. Hopefully though, you did a waterproof barrier below the tile, like Kerdi or RedGard?
I'd suggest posting that photo as well as some close-ups on the John Bridge tile forum. You should specify what type of grout you used, what additives if any, and the instructions you followed. They will need to know all that.
http://www.johnbridge.com/vbulletin/forumdisplay.php?f=1
