The Paradigm of Prediction

A system that is over-reliant on prediction through leverage,
(like the banking system before the recent crisis), hence fragile
to unforeseen “black swan” events, will eventually break into pieces.
–Nassim Taleb

I don’t read many business books anymore. For years I devoured them voraciously. But something happened to me as I began to realize that the paradigms I was reading about had much more to do with faith and beliefs than real science.

Oh yes, much of it was dressed up in pseudo-scientific language to give it a patina of authority, but when I dug deeper, what emerged was a subjective POV promulgated and defended with evangelical zeal instead of scientific detachment. I am certainly not against faith-based literature. In fact, I have spent a great deal of time with the Torah, Gospels and Quran and found them all quite uplifting.

Thomas Kuhn, who coined the term “paradigm shift”, defined a paradigm as a set of beliefs or views that members of a community share. Kuhn discovered over the years that paradigm stakeholders maintain their POV in a most irrational way, because faith-based worldviews – be they secular or religious – by design lie outside the realm of reason. And I say this as a person steeped in his own faith.

Aye, aye! and I’ll chase him round Good Hope, and round the Horn,
and round the Norway Maelstrom, and round perdition’s flames
before I give him up.
–Moby Dick

Nowhere is this stranglehold of irrational exuberance more evident than in the curious love affair the digerati have with prediction and predictive-modeling memes. Billions have been invested in pursuit of this obsession, and yet prediction remains an elusive modern day “white whale”.

What the zealots won’t allow themselves to see is how spectacularly this paradigm has failed. It’s as if they are all suffering from some collective delusion waiting for the Rapture (which by the way has been postponed to October 21st this year). How spectacular has the failure been? Two words; “Arab Spring.” This refers to the wholly unpredicted political uprisings and revolts that sprang up virtually simultaneously in Arab nations around the world recently, despite terabytes of information and the most sophisticated predictive models on the planet crunching the data.

Credit card companies armed with overwhelming amounts of data on you and me have still managed to create 30% toxic debt that they COULD NOT PREDICT. Massive financial crises in housing, health care, and the stock market have caught us completely flat-footed despite overwhelming attention to and investment in predictive methodologies.

The collapse of catalog-direct marketers who COULD NOT PREDICT variables or events that led to their demise should give us pause in our pursuit of the white whale, but that would require replacing irrationality with reason, and if there is one thing we can predict with certainty, it is that human behavior is not rational.

The Fukushima nuclear disaster could have been prevented if they had built a wall high enough to withstand a 24-foot tsunami. Instead, their predictive model provided for a 20-foot wave. The huge insurance company, AIG, spent millions of dollars to predict risk. Yet as we know, in 2008, they COULD NOT PREDICT the financial collapse that bankrupted them. So much for predictions.

There is no shortage of spectacular failures that can be traced to faulty predictions. And yet, instead of learning from our mistakes, instead of seeking refuge by embracing uncertainty, we react instead by doubling-down on our irrational persistence, seemingly impervious to the hazards of trying to predict human behavior. Nowhere is this more evident than in the online world where the promise of behavioral targeting has given way to utter chaos; resulting in a patchwork quilt of flawed premises with no cohesive design.

And even though the predicted day of Rapture has come and gone, followers of the delusional pastor are busy retooling their worldview to fit a new date. The digerati, too, are busy patching their quilt in a vain attempt to shroud themselves from perdition’s flames.

About Jaffer Ali

Jaffer Ali is the CEO of e-commerce company, He is a serial entrepreneur who has touched every part of the online ecosystem since 1998. Jaffer is known as a contrarian thought leader who has contributed over 250 essays to industry trades covering every aspect of online media and marketing topics. A collection of his essays can be downloaded for free: Chasing String In The Digital Era.



PreviousTomorrow Will Be Televised: Google TV's Second (AKA Last) Chance NextShelly Palmer Radio Report - July 1, 2011

Get Briefed Every Day!

Subscribe to my daily newsletter featuring current events and the top stories in technology, media, and marketing.