Yep, you have to calculate (in total) what you paid for them and subtract that from what you sold them for and declare that as capital gains on your taxes. I didn't know this either the first time I did it--and I thought TD would automatically put that info on a form for me (like they do dividends), but I was wrong. I had to go back through all my purchases of the fund to figure out average cost basis. Now I carefully note down "Book Value", "Market Value", and number of shares on the day the sell transaction happens in my non-registered account.
It's not a "stupid" move though--espcially if you were moving cash to the TFSA to fill up last years' $10K room. Long term, it's better to have things growing there than outside it.
We can't avoid capital gains forever either--sooner or later you do have to pay the piper if you're making money. :) Yes, pay your taxes! (And luckily--(you didn't hear this from me but...) the CRA is apparently pretty forgiving when it comes to small time investors if we have difficluty calculating capital gains and our total is off by a few bucks. While you should make sure it's as accurate as possible, it's probably not going to incur an audit.)