The March 2011 Scientific American includes "Financial Flimflam" by Michael Shermer, subtitled "why economic experts' predictions fail". But it is not so much about economic predictions as about the theory that index funds are a better investment than managed funds. He claims the market average return in 2010 was 2.5% higher than the average of the ten largest managed funds.
Index funds, of course, have their own perils. A lot of people bailed out of their index funds some time in 2008 or early in 2009, as a sort of stop-loss measure. Then they missed the market runnups in later 2009 and in 2010. Thus gutting their retirements.
Since I do my own research and make my own investment decisions, and also get paid to do specific research and analysis by a fund manager, I think it is fair to wonder if I might be better off just buying into an index fund like so many other people and institutions.
Lately I have had an extraordinary rate of return because my portfolio contains only 15 stocks, and those include 3 with extraordinary returns of late: TTM, the Printed Circuit Board manufacturer; Dot Hill, a storage company, and Dendreon, the maker of Provenge for prostate cancer. In the past I have had other stocks hitting extraordinary returns, but because I have tried a number of risky, turn-around, small cap situations, I have also lost all the money I invested in 3 stocks over the past decade. There have also been times when my stocks lagged the market. Partly this is because since I typically play Nasdaq 100, or smaller, stocks, so they typically go down more in down markets, but up more in up markets.
I agree that if you are paying someone to manage your portfolio, they need to beat the market enough to pay their management fees and then some. Otherwise you could do better by creating your own index fund. It is not hard, you could for example buy equal amounts of the Dow 30 and the Nasdaq 100 or S&P 500 or any other known set of stocks.
Still, there are people like Warren Buffet who had very long runs of better than average returns. That is not just luck.
Large managed funds have trouble beating the market partly because they are the market. Their individual stock positions are so large that their creating or leaving a position, or even trimming, moves stock prices. They also tend to be in a relatively large number of stocks, which again dilutes their performance back towards the market average.
To beat the market it really helps to play on a small enough scale that your buys and sells don't affect the stock price substantially.
It also helps to see the curve. Here I mean seeing more than the statistics we all tend to rely on, revenue growth and earnings growth and margins and cash. You need to see future value where others are missing it. You also need to be very serious about weighing risk. Truth be told, I thought Dendreon was riskier than Anesiva, but Anesiva went down with my investment, while Dendreon took me up by a factor of ten. If both had sunk, I would be writing a far gloomier story today. Interestingly, in addition to stocks with high risk levels, I always keep stocks with good potential for returns and relatively lower risk. Some of those stocks have performed badly for me, but none went out of business.
I basically doubled my money in the stock market during a period of time when the market went basically no where. I balanced my risks and made my mistakes, and have been better at avoiding mistakes lately. In retrospect, or course, I could have put every penny I had into Dendreon, but that did not look wise at the time. Backtesting is interesting, but you have to go with what you know in reality, at the time of investment.
On the other hand I do an enormous amount of research, considering the size of my portfolio. I don't just invest in biotechnology stocks; I study biology text books and journal articles. I don't just invest in computer technology, I do my best to keep up with the breadth of its developments. Also, I try to think like an owner of the companies I invest in. I like companies that invest wisely in the future and run a tight ship.
Frankly, unless you enjoy doing research, you are better off in an index fund. But if you are willing to do your homework (and it really is a lot of homework), you have a good chance of beating an index fund if you are an astute individual investor.
You should check out the Scientific American article because it has some other information, especially about investor psychology. I love this bit: "Being deeply knowledgeable on one subject narrows focus and increases confidence but also blurs the value of dissenting views and transforms data collection into belief confirmation."
Above all, a smart investor looks most closely at the data that challenges current beliefs.
If you aren't familiar with it, see the Random Walk Hypothesis at Wikipedia.