Yep, the hole is WAY too big. As other mentioned, if you run into the screw issue again, and it's a plastic box, just switch to a #8 x 1" sheet metal screw. If you decide to install another old work box, like this one, they need to be cut extremely neatly and tight to function. The way to do it is to use the box as a template. Place the box on the drywall, with the opening on the wall. Trace the sides, and the top and bottom of the box, NOT the flange. Now flip the box around and take a good look at it. If you traced everything, the cutout area for the screws and plastic "flippers" is drawn on the wall backwards, be sure to correct this before you cut.
If you removed a box that was attached to a stud, there is another way to fix this mess. Remove the box you installed. Now take a blue, single gang nail on style box, and use a hacksaw to cut the nail flanges off. Drill two 1/4" holes in the side of the box, about an inch in from the front of the opening. Hold the box flush to the drywall surface and use two drywall screws, through the side of the box, to secure it to the stud. Using this technique and a "midway" cover plate, which is oddly enough midway between a standard sized plate and a jumbo, will hide the oversized hole.