Are We Ready For The ‘Big Data’ Revolution?

Popular Economics Weekly

Is the Information Age intimidating? Have you been struck with information overload? Help may be on the way. There is so much in the news about it, but little on how to manage so much information inundating us daily—not only via computers and emails, but our 24/7 news cycle. But research is beginning to show us how we can benefit from such increasingly available information.

Kinsey and Company consulting has taken the side of businesses in its McKinsey Quarterly report, Are You Ready for the Era of ‘Big Data’? “In 15 of the US economy’s 17 sectors, companies with more than 1,000 employees store, on average, over 235 terabytes of data—more data than is contained in the US Library of Congress”, said its report.

All of this new information is laden with implications for leaders and their enterprises, said the Kinsey report. “Emerging academic research suggests that companies that use data and business analytics to guide decision making are more productive and experience higher returns on equity than competitors that don’t. That’s consistent with research we’ve conducted showing that “networked organizations” can gain an edge by opening information conduits internally and by engaging customers and suppliers strategically through Web-based exchanges of information.”

And the big data revolution will result in much greater efficiency on how private households use information as well. Yale Economist Robert Shiller looks into the future in his book, The New Financial Order, Risk in the 21^st Century. In it Shiller describes six fundamental ideas for using modern information technology and advanced financial theory to temper basic risks that have been ignored by risk management institutions--risks to the value of our jobs and our homes, to the vitality of our communities, and to the very stability of national economies.

“Right now we are witnessing an explosion of new information systems, payments systems, electronic markets, online personal financial planners,” he says, “and other technologically induced economic innovations, and consequently much in our economy will be changed within just a few years. Almost all of our economy will be transformed within just a few decades.”

Informed by a comprehensive risk information database, this new financial order would include global markets for trading risks and exploiting myriad new financial opportunities, says Amazon’s description of The New Financial Order. “From inequality insurance to intergenerational social security. Just as developments in insuring risks to life, health, and catastrophe have given us a quality of life unimaginable a century ago, so Shiller's plan for securing crucial assets promises to substantially enrich our condition.”

As information becomes more readily accessible across sectors, says McKinsey, it can threaten companies that have relied on proprietary data as a competitive asset. The real-estate industry, for example, trades on privileged access to transaction data and tightly held knowledge of the bid and ask behavior of buyers, information owned by Brokers. Both require significant expense and effort to acquire. In recent years, however, online specialists in real-estate data and analytics have started to bypass agents, permitting buyers and sellers to exchange perspectives on the value of properties and creating parallel sources for real-estate data.

So the results will give us a better understanding of our own finances as well. But greater access to personal information that big data often demands will place a spotlight on another tension, between privacy and convenience, says McKinsey. Their research shows that consumers benefit greatly from data in lower prices, a better alignment of products with consumer needs, and lifestyle improvements that range from better health to more fluid social interactions. The tradeoff is less privacy, as companies collect more information on individual consumer’s behavior.

Professor Shiller is one of the economic trail blazers of the information revolution. Not only did he study U.S. stock market behavior over the last century in his book, “Irrational Exuberance” (Princeton U. Press, Princeton, N.J., 2000) that predicted the dot-com implosion (and the housing bubble, in its second edition). He and Wellesley Professor emeritus Karl Case have set up the S&P Case-Shiller Home Price Index that tracks the historical swing of home prices in 10 and 20 metropolitan markets, which has helped to establish a futures’ market for home prices, and in turn helps to make such market swings more predictable.

Establishing and disseminating the historical record should also help policy makers avoid endlessly repeating history’s mistakes, which is something easy for the general public to understand. I.e., historical study of the Great Depression is one reason Presidents Bush and Obama were able to inject enough stimulus into our economy to avoid another Great Depression. And it helps bring about our own greater awareness of the financial environment that affects us as consumers and investors.

So the information age is a two-edged sword. The greater access of consumers to information via the Internet means companies will collect more privately held information of consumers. But, conversely, as more information becomes public, less can be hidden by insiders such as financial traders, corporate executives, and the like. More transparent financial markets should also decrease the occurrence of busted asset bubbles—two of which we have experienced just since 2000 (i.e., dot-com and housing)—which were mainly based on irrational exuberance—i.e., ignorance of the underlying facts.

How to manage such information then becomes ever more one of individual choice, thereby placing more responsibility on the informed consumer. That is really the sword’s other edge. More than ever, we are living in a ‘buyer-beware’ world of individual decision-making, requiring ever higher levels of education.

Harlan Green © 2012