Super Crunchersis just a great introduction to the world of data-mining. Author Ian Ayres uses a ton of case studies to illustrate the need for, processes and benefits of Super Crunching (the analyzing of “large datasets to discover empirical correlation between seemingly unrelated things”). He presents his arguments in a similar manner as The Long Tail and it is just as easy to read.

One of the main concepts is that we too often rely on “experts” to predict the likelihood of future events. Where in all reality they tend to have too much emotional involvement to see a situation objectively, whereas computer and algorithms have no vested interest in the outcomes they predict. This fact paired with the ability of super-computers to account for literally millions of individual factors can produce insight that actually gives a confidence ratings on how accurate the prediction will be.

Another theme that I found particularly interesting is the effect Super Crunching has had on the value of data. Ian points out that “information is not only easier to capture now in digital form, but it is also virtually costless to copy…these breakthroughs in database technology that have facilitated the commoditization of information. Digital data now has market value and it is coalescing into huge data ware houses”. This added emphasis on data collection means that that there is now a high demand for those who can sort it into meaningful relationships. “People are looking past the M.D.s, who merely disseminate information, and toward the Ph.Ds, who create the database to discover information“.

Ian is close friend of Freakonomics author Steven Levitt and Super Crunchers regularly references Levitt’s work. I found that reading Super Crunchers gave me deeper insight into the Levitt’s book. You can get both their opinions on a wide range of topics on the Freakonomics blog.