Quit being lazy and put the three season tires back on. It's not even a full beer job. :p
The "bad things" that can happen slightly depend on the tire type and area.
If you're running studded snow tires, the rubber is fairly hard, and your problems will come from the fact that running studded snow tires is illegal in the summer pretty much everywhere, there's not a huge amount of rubber on the road (since the open tread pattern to grip snow isn't great on bare pavement), and the studs don't grip asphalt very well. You'll be down on grip, be wearing the road and tires pretty significantly, and it's loud (which, being paired with being illegal, isn't a great option - studded snow tires in the summer really stand out acoustically).
If you're running studless tires (Blizzaks or similar), they'll wear very quickly (as in, you can wear through the magic rubber layer in a few thousand miles, if that), and have the grip of a soft sponge in the summer. They're pretty much hazardous to drive on if you have to make any sort of emergency evasion on dry pavement.
Put the all seasons back on.