Stop Hyping Big Data and Start Paying Attention to ‘Long Data’

By Samuel Arbesman | January 31, 2013

Our species can’t seem to escape big data. We have more data inputs, storage, and computing resources than ever, so Homo sapiens naturally does what it has always done when given new tools: It goes even bigger, higher, and bolder.

We did it in buildings and now we’re doing it in data. Sure, big data is a powerful lens — some would even argue a liberating one — for looking at our world. Despite its limitations and requirements, crunching big numbers can help us learn a lot about ourselves.

But no matter how big that data is or what insights we glean from it, it is still just a snapshot: a moment in time. That’s why I think we need to stop getting stuck only on big data and start thinking about long data.

By “long” data, I mean datasets that have massive historical sweep — taking you from the dawn of civilization to the present day. The kinds of datasets you see in Michael Kremer’s “Population growth and technological change: one million BC to 1990,” which provides an economic model tied to the world’s population data for a million years; or in Tertius Chandler’s Four Thousand Years of Urban Growth, which contains an exhaustive dataset of city populations over millennia. These datasets can humble us and inspire wonder, but they also hold tremendous potential for learning about ourselves.

Because as beautiful as a snapshot is, how much richer is a moving picture, one that allows us to see how processes and interactions unfold over time?

We’re a species that evolves over ages — not just short hype cycles — so we can’t ignore datasets of long timescale. They offer us much more information than traditional datasets of big data that only span several years or even shorter time periods.

Read full post here. (Originally posted January 31, 2013)