Casa Loma can be fun to explore - it's visually impressive and does have some cool features, like an underground tunnel to the stables and a lovely conservatory. That said, I can't vouch for how engaging it would be for a three year old. There's certainly a lot going on, but in my experience it's not interactive other than moving around in it.
You'll be somewhat close to OCAD University (formerly Ontario College of Art and Design) which might be worth walking by if there's a nice day, if just to see the Will Alsop designed addition.
If you're willing to take a bit of public transit from downtown, the Ontario Science Centre is great - really interactive and wonderful for kids.
Happily, vegetarian is plentiful. Ryerson University is right there, and there are lots of nearby options.