The Gateway to Algorithmic and Automated Trading

The Year of Big Data: The challenges of putting trillions of odd-shaped pegs into neat little holes

Published in Automated Trader Magazine Issue 27 Q4 2012

The volume of digital content is forecast to surge to a staggering 2.7 zettabytes in 2012. How big is that? One of those is 10 to the 21st power. We're talking billions and billions of terabytes. What's more, at least 90 percent of that will be unstructured, made up of enormous numbers of files based on social media or web-enabled processes. How do companies turn this information soup into something palatable? The short answer is, in lots of ways. Adam Cox reports.

The Year of Big Data. The challenges of putting trillions of odd-shaped pegs into neat little holes

This is the year when "Big Data" becomes a must-have competency, according to International Data Corporation. Its forecast volume of 2.7 zettabytes for the world's digital content represents a 48 percent jump from the 2011 total. Most of that - social media files, text, images, videos, audio - is unstructured and will offer formidable challenges for business information (BI) specialists. But don't expect those firms to be daunted.

"As businesses seek to squeeze high-value insights from this data, IDC expects to see offerings that more closely integrate data and analytics technologies, such as in-memory databases and BI tools, move into the mainstream," IDC said in a report on this year's data outlook.

How do information specialists make sense of all this unstructured data and what are the key hurdles they face? In the realm of financial markets, much comes down to the techniques they use to process textual data.

BT Radianz, which focuses on telecoms for the financial markets, is seeing an increasing number of information providers coming onto the network focused on delivering unstructured data that is reshaped into structured data.

Michael Cooper

Michael Cooper

"I have a feel for it, and that feeling is that it's broadening and increasing," said Michael Cooper, BT Radianz's chief technology officer, referring to the use of unstructured data. "The base of what people are using is definitely broadening and I think that the frequency or volume of that data is increasing as well."

That presents one of the most immediate challenges to this trend: data capacity. "So one of the things you have to do with this type of data, you really have to use it intelligently," Cooper said. That means not only dealing with the scale of the volume, which he suggests is actually manageable, but also addressing its fluid nature, which is less so.

The remainder of this article is only available to Paid Subscribers

Click here to purchase a subscription to Automated Trader

  • Copyright © Automated Trader Ltd 2018 - Strategies | Compliance | Technology

click here to return to the top of the page