Managing the knowledge era

Information overload or filter failure?

(This the first in a series of articles on issues that impact competitiveness.)

Let us first examine whether we are indeed experiencing an information explosion

John F. Gantz, a project director at the advisory firm IDC, released a report a little more than a year ago entitled, "The Diverse and Exploding Digital Universe". In it he calculated that the digital world in 2006 contained 281 exabytes of information.

An exabyte is a measure of computer memory, and is one of the most senior members of the extended byte family. This family stretches from the youngsters kilobyte, megabyte and gigabyte (which we have all met on our laptops) through to grandma exabyte and granddad zetabyte (which we have not).

Now, it is not all that important to know exactly how hefty Nana exabyte is, but just to give an idea...

The largest library in the world is the Library of Congress in Washington, DC. It houses more than 30 million books; more than 58 million manuscripts; one million bound issues of world newspapers spanning the past three centuries; over 6 000 comic book titles (for some, a particularly valuable addition to the library); and the world's largest collection of films, maps, sheet music and sound recordings.

In a thesis written by David Gough of the US Air Force, he determined that if the holdings of the Library of Congress were to be digitised, it would use up 17 to 20 terabytes of computer memory. That's terabytes, not exabytes. What this means is that the entire content of the Library of Congress could be stored in digital format on a single exabyte of memory, not once, not twice, but at least 50 000 times over. And back in 2006, there were already 281 such exabytes of information in existence...

But back to Gantz. Of particular significance is that he projected that the 281 exabytes of information produced between the time Adam walked the earth and 2006 were set to increase tenfold over the following five years. In practical terms, this means that all the information ever produced by mankind since creation will double over the next 18 months, and then double again in ever-decreasing time increments.

The online search engine Google also keeps statistics on the growth in online content, as does the University of California at Berkley, which runs an ongoing project to determine how much information is 'out there'. Both these institutions may differ somewhat from Gantz on how much information mankind has produced to date, but in broad terms they concur with Gantz as regards the cliff-face in information growth with which we are currently confronted.

So it is true: We are undoubtedly amid an information explosion caused by the convergence of technology, a growing world population and increased global literacy. These are all factors that cannot, and will not, be reversed in the foreseeable future, and which will exacerbate the problem for some time to come.

But what is this problem exactly?

On the surface it appears to be that oft-repeated one of "information overload", a term first coined by Alvin Toffler more than 20 years ago. In short, information overload means that there is just too much information around and too little time in which to identify, digest and make use of the relevant bits.

This results in all kinds of ailments such as anxiety, stress, headaches, carpel tunnel and so on. In December 2008, the US research company Basex published the results of a survey which shows that the inability of workers to deal with the problem of information overload cost the US economy between $900 billion and $1 trillion in lower productivity and throttled innovation in 2008 alone. (Perhaps each e-mail should carry a compulsory warning: "Reading this could lead to heart failure.")

In an article published in CIO Enterprise Magazine as long ago as 1998, LM Fuld describes a cartoon of a man soaked to the bone, standing by the seashore with an enormous wave about to break over him. He looks up resignedly and says, "Oh, good, more information!"

This cartoon aptly illustrates the dilemma experienced by leaders in all spheres of life (business, politics, science, etc.) in the face of the modern-day deluge of information: Rather than empowering them to make better decisions, the glut of data and information often has the opposite effect - that of decision-making paralysis.

Larry Bowden, vice president of IBM, succinctly describes the heart of the problem: "Information overload slows me down because I am interacting with information that is irrelevant and out of context and does not allow me to get to the end point and have the impact I want."

But let us rewind a bit. When Johannes Gutenberg invented moveable-type printing in the mid-1400s, his invention lowered the cost of printing to such an extent that by the 1500s, more book titles were being published annually than could be read by any person in a lifetime.

Information overload is therefore not a new phenomenon; it is a problem of ancient lineage - the amount of available information already surpassed our ability to absorb such information half a millennium ago. Though every Sunday issue of The New York Times now contains more information than a 17th century individual would have been able to gather in a lifetime, the information explosion of today did not create the fire hydrant of information from which we wish to sip sustenance; it simply twisted the tap open even further.

The true problem

The true problem for present-day managers does not lie in the fact that the amount of available information has only recently outgrown their ability to make good use of it. (As they say in the information technology industry, 'too much' information is not a bug, it is a feature.) The reason for our inability to cope with the flow of information must lie somewhere else.

It is in searching for an answer to what the real debilitating factor is that currently impinges on our ability to handle the flow of information that Clay Shirky, a professor at New York University, makes a compelling case. He argues convincingly that "filter failure" lies at the root of the problem. His thinking goes something like this:

Before the advent of the Internet, information was distributed mainly in printed format. Printing information in hard copy required upfront expenditure from printing companies which was only recouped if the book sold well. If not, the printer could go bust.

Given this economic reality, printers had to make a judgment as to the potential profitability of the content of manuscripts submitted to them long before firing up the presses. Hence printers and publishing houses acted as a filter situated at the very beginning of the information distribution cycle, separating the wheat from the chaff. This filter - and other comparable filters built into other spheres of normal social life - protected us from the worst excesses of inbound information.

The Internet changed all that, and in short order. The cost of producing, publishing and distributing information via the Internet and other electronic distribution channels has dropped to close to zero.

The earlier economic imperative for filtering content at the point of origin has been rendered useless by the ubiquitous Internet, and making a judgment as to the veracity and value of information has been shifted onto the end user.

But - and this is the main point Shirky makes - we have yet to construct effective end-user filters as a substitute for the now defunct point-of-origin filters.

According to Shirky, the real problem faced by business leaders, politicians, butchers, bakers and candlestick makers in this knowledge era is not information overload but filter failure.

If it is true - and there is no reason to believe otherwise - that intellectual capital has evolved into one of the cornerstones of production and profit, business managers will need to construct working end-user information filters in double-quick time.

[The next instalment in this series will suggest what business managers could do to begin constructing end-user filters aimed at limiting unwanted inbound information and enhancing intellectual capital.]

Sources:

1 http://www.emc.com/collateral/analyst-reports/diverse-exploding-digital-universe.pdf

2 Available through Storming Media. (http://www.stormingmedia.us)

3 http://www.basexblog.com/2008/12/19/information-overload-now-900-billion-what-is-your-organizations-exposure

4 "The Danger of Data Slam", CIO Enterprise Magazine, 15 September 1998.

5 http://www.basex.com/web/tbghome.nsf/pages/home

6 Gerry McGovern and Rob Norton, Content Critical, Financial Times Prentice Hall, 2001.

7 Clay Shirky, Presentation at the Web 2.0 Expo in New York, September 2008. (http://www.youtube.com/watch?v=LabqeJEOQyI)