I lived in Toronto for 6 months back in 2001/02 so things have probably changed a lot, but I second the islands suggestion. It feels like a world away from the city, especially mid-week on the furthest side to the city. Niagara Falls is also just a Greyhound hop away and it's easy enough to escape the crowds who only head for the nearest viewpoint to the visitor centre. If you have time there are plenty of provincial parks and nature walks within a day trip of the city, many accessible by public transport. I really wish I'd spent more time exploring them. The main harbourside waterfront around the CN Tower & stadium is fairly disappointing, I rarely bothered going there except for the odd hockey game.
If you like natural history and art then the ROM (Royal Ontario Museum) is great, and I think it was free one night a week. The art gallery on Dundas St West was good too.
High Park is nice enough and there are often events on there, but to be honest I wouldn't make a special trip for the park alone. If you enjoy specialty shops there are plenty around Chinatown & Kensington Market (oh, how I miss the cheese shop at Kensington!).
It's quite quiet and leafy around the university for a nice wander, and there are often free public events there. I moved to Vancouver shortly afterwards and was really struck by how different the lifestyles were. Toronto was far more community/event oriented whereas Vancouver was outdoor destination-based, so it's worth doing some research online before you go to find out if there are any festivals or events going on.