Ayres, Ian. Super Crunchers: Why thinking-by-numbers is the new way to be smart. New York: Bantan, 2007. 260pp.

You know how I'm always complaining about business-y buzz/hype books & articles? How they're 1/3 repetition, 1/3 hype and 1/3 real ideas?

Like I commented to Michael not too long ago: "I find these tendencies very true of a lot of cases where I look to the business literature to understand something important about the way our culture is changing."

The book under consideration in this review, Super Crunchers by Ian Ayres, is a business book. It says something important about the way our culture is changing. On the other hand, it is also very profoundly a popular science book about the mathematical and statistical analysis of large datasets. Yes, indeed -- this is a popular math book about data mining. And it is a very good to boot. Thankfully, not so much plagued by the repetition and hype of many of the pure business books. I suspect it may have originally been aimed at a popular science audience as much as a business audience, accounting for a slightly different emphasis.

So, what is super crunching? (p.10)

It is statistical analysis that impacts real-world decisions. Super Crunching decisions usually bring together some combination of size, speed and scale. the sizes of the datasets are really big -- both in the number of observations and in the number of variables...And the scale of the impact is sometimes truly huge. this isn't a bunch of egghead academics cranking out provocative journal articles. Supter Crunching is done by or for decision makers who are looking for a better way to do things.

In other words, data mining. To it's credit the book doesn't really talk about the hows and whys of the actual mathematical analysis; it mostly concentrates on the applications and implications of these powerful tools. The core theme of the book is how do you make decisions in the data mining (I've decided to to not bother with Ayres's cutsie term and just say data mining) world: evidence or intuition? Evidence wins every time.

Some interesting points to consider: the rise of data mining tools is in large part to the drastic decrease in storage costs the last number of years, far more than any increase in processing power. On the other hand, the use of neural network technology has also contributed to better and better techniques.

The book basically goes through a bunch of applications areas and shows how each are affected by data mining -- basically showing that the evidence provided by statistical evidence beats out human intuition every time. It's an interesting examination of the nature of expertise: what does it really mean to be a human expert when math wizards can transform large data sets into much more accurate predictions about human behaviour. What's left for us to do? Of course, the human role is to decide what data to collect, what questions to ask in the analysis and how to apply the results.

Ayres looks at recommendation systems (like Amazon), data mining application in the entertainment industry (yes, scripts and box office data are data mined, resulting in, apparently, Will Farrell), economics and government policy and evidence-based medicine (perhaps the best chapter).

To his credit, Ayres doesn't duck the hard questions all this brings up. He deals with privacy concerns, the dangers of over-reliance on programmed creativity and other interesting areas. It's a powerful technology, and while balance is needed in some respects, understanding is a far preferable reaction to change.

Instead of a Luddite rejection of this powerful new technology, it is better to become a knowledgeable participant in the revolution. Instead of sticking your head in the sands of innumeracy, I recommend filling your head with the basic tools of Super Crunching. (p.191)

A good reaction to any new technology. And I like the way he ties it in with the general innumeracy of our times, especially the media and chattering classes. A tool can be used for many purposes. Let's all be

Passionate about the need to inculcate a basic understanding of statistics in the general public. "We have to get students to learn this stuff...We have to get over this phobia and we have to get over this view that somehow statistics is illiberal. There is this crazy view out there that statistics are right-wing"...One can crunch numbers and still have a passionate and caring soul. You can still be creative. You just have to be willing to put your creativity and your passions to the test to see if they really work. (p. 215)

I recommend this book without reservation. Any library that collects math or popular math books would find it a terrific addition to their collection. Business libraries would also find it appropriate. Collections that are looking at the way technology is changing our culture would find that Super Crunchers belongs alongside books like Wikinomics or Everything is Miscellaneous.