I vote for around age 11-12. At that point, after-school care options are nonexistent, kids are starting to get more seriously into particular after-school activities that often require parental participation, and the opportunities for the kid to go off-track get much more serious. Basically, kids are treated as if they're old enough to be on their own, but often are not actually prepared to handle that degree of freedom.
I came to this conclusion with my DD, who was the kid I definitely worried about going off the rails. But this past year with my now-15-yr-old DS, I'm discovering I really enjoy having him home at the same time. Not all the time, of course. ;-) And I never have worried about him in that manner, and I still don't. But he's a quiet kid who is at the age where kids often start with withdraw into video games and such (and that risk is magnified by the lack of any other options to hang out with friends or do activities). And so just physically being there, just the two of us, gives him the chance to poke his head in and connect, even if it's only 2 minutes at a time. And I can tell he enjoys it, because he pops in between every. single. class. to tell me everything he just did. . . . ;-)
Also, IMO, the daycare ages were much easier to work through -- I had no idea how good I had it until we got into the school system with it's multitude of holidays, in-service days, half-days, and events that always had me scrambling. (Plus I am one of those people who enjoys older kids much, much more; the little ones are adorable, of course, but not great conversationalists. ;-))