News Article Big data: what is big data?

Big data was mentioned a lot in 2013 but even with all this talk, there is confusion still about what it is. You only have to Google the term to find a myriad of competing articles describing the concept, some more coherently than others.

Big data refers to datasets that are so large and complex that they can’t be processed with conventional manipulation tools such as relational databases. Instead, they utilise an ever-growing collection of alternative storage and analysis techniques which can deliver results in a tolerable length of time.

The likes of eBay and Amazon deal with many tens of petabytes (1 PB = 1048576 GB) relating to customer behaviour, but the context of “Big” can depend upon the perspective of the organisation and the capability of their tools.

For smaller businesses, the prospect of reporting on several gigabytes of data using their existing methods may give them cause to reconsider their data storage and reporting strategy. For larger organisations, the threshold to review this aspect of their operation may be closer to many terabytes.

While the big data approach can allow an organisation simply to replicate their existing reports and run them against a larger pot of data, there is far greater potential than just that. The analysis of big data can provide insight that might not previously been detected by human interpretation, such as inferring relationships between hitherto unlinked variables or factors.

For businesses, drawing conclusions from past customer behaviour can, in turn, help them to predict the behaviour of customers in the future. This type of insight can provide better engagement through more finely tailored user experiences.

Perhaps unsurprisingly, there are more than just commercial applications for big data. Much of the data captured on modern research projects would be superfluous without big data methods - the Large Hadron Collider is one familiar example. Similarly, the data collected by governments and security services is almost certainly processed and analysed using the big data approach.

It’s clear that capturing and storing big data is not enough in itself. To be successful, you need to harness the information hidden within it. And that’s easier said than done.