I'm not looking for sympathy. My main reason for posting here is simple. I'm sure most of you did not jump into index fund immediately. Most of you probable went through a phase in which you dabbled in stock trading or individual stock picking. I'm more curious to know how most of you managed to give up that urge and started indexing only.
You can be successful in individual stock investing, but almost nobody has the key ingredients to do so:
1) Education: not even formal education, but understanding the fundamentals of business, economics, and stock markets so that you can be an informed participant. Understand how to read income statements, balance sheets, and statements of cash flow. Read about global macroeconomics and trade--understand and ponder the potential of another 2 billion middle-class market participants coming up in China and India. Understand the SEC's Edgar system, and subscribe or regularly check other filings, like debt offerings, major holders, and industry analyst events.
2) Effort: Religiously read 10Q's and 10K's, not just of companies you own or are studying, but also their competitors, customers, and suppliers. Keep abreast of business news. Devote time regularly to research, and rarely to buying or selling. (a dozen transactions in a year is a very busy year for me)
3) Discipline: Stick with the method, even if it isn't "working" this quarter or year; there are factors outside of your control. Take rote analyses and other people's recommendations with a huge grain of salt: grist for your own work, but never something to act on. Make you you thoroughly understand a situation before you act; the liquidity in the market entices you to "act now! limited time offer!" but it never is. As Warren Buffett says, there are no called strikes in the stock market.
If someone had a sure-fire key to knowing the market, and could guess right just twice in a row, they'd be the world's first trillionaire already. People have theses, and they can work more or less, but nobody can time the market perfectly. (again, once lucky, but never twice in a row)
Oh, by the way, Warren Buffett and his ilk are doing this; that's the "pack" you have to keep up with.
If this all sounds like too much, then just index and get on with your life.
The closest analogy I can give to successful stockpicking is fishing: a whole lot of time that seems nonproductive, mixed with occasional payoffs. If you don't enjoy the process: getting up at the crack of dawn, tieing flies or collecting favorite lures, weather that is cold / rainy / sweltering, days with nothing to show for it, cleaning the darn things, then you aren't cut out to be a fisherman. Everyone just wants to catch fish, and let someone else worry about the rest.
And even with all this said, my portfolio is about 50/50 indexed. It's where I park my money when the fish aren't biting. It's also my ballast--my hedge against my own limitations. While I am confident in my investing, and I hope it is the icing on the cake, I won't let it cost me everything.
Many people, recreational fishermen, might have just 10% of their assets in a "Mad Money" account. This is, essentially, a hobby: it may end up costing money versus indexing, but it keeps the person interested in investing. But in the end, I agree with a lot of comments here. You are using the language of addiction: "urge", "beating", etc. I think any kind of active investing may be quite dangerous.
Benjamin Graham, Warren Buffett's teacher and mentor, never talked about beating the market. (although he did, for long runs) rather he talked about achieving "satisfactory returns." If you can't find a point where your investments are satisfactory--where they are achieving your plan, and you are at peace with that--then you will probably be better off with them in someone else's hands. Investing isn't anything in itself. (speaking on a personal basis; of course, capitalism has a purpose) It is merely the means--a means, as there is more than one way to get there--to fund your life.