“To infinity and beyond!” charges Buzz Lightyear, space ranger and hero of the hit movie Toy Story. The amount of data in our world has been exploding, well beyond the “cloud” – is “infinity” within reach? Creating, storing, managing, and analyzing these vast digital data sets – so call “big data” – is all the buzz in the information technology world of 2012. Data analytics research firm McKinsey Global Institute has touted big data as “the next frontier for innovation, competition and productivity.”

Big data is generally understood to mean data sets whose size are beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. The data is too big, moves too fast, or doesn’t fit the confines of typical database architectures. It requires exceptional technologies to efficiently process large quantities of data. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set.

Examples of big data include: input from chatter from social networks, web server logs, traffic flow sensors, satellite imagery, Internet search indexing, broadcast audio streams, military surveillance, weather and space data, banking transactions, medical records, MP3s of rock music, the content of web pages, scans of government documents, photography archives, GPS trails, telemetry from automobiles, financial market data, geophysical data from energy companies, prescription drug testing, the list goes on. Look at your own organization – it maintains a mountain of data in all of its datacenters, regional offices and on all of its user-facing systems (desktops, laptops and handheld devices).

How big is ‘Big?’
To understand how much data is now at our fingertips, you need to start with “bits” (remember them?), then move along the spectrum to “yottabytes” (which sounds like Yoda, the Jedi master in Star Wars, who wisely said “Size matters not.”)

Source: http://www.economist.com/node/15557421

To put this into context, consider that Wal-Mart handles more than 1 million customer transactions every hour which it in turn imports into databases estimated at more than 2.5 petabytes, which is the equivalent of 167 times the books in the Library of Congress. Facebook handles 40 billion photos from its user base. Decoding the human genome originally took 10 years to process when it can now be achieved in one week.
McKinsey found that in 2010, people around the world collectively stored more than 6 exabytes of new data on devices like PCs and notebook computers; each exabyte contains more than 4,000 times the information stored in the Library of Congress. Cisco predicts that the amount of traffic flowing over the internet will reach 667 exabytes annually by 2013. Every day, 2.5 quintillion bytes of data are created and 90% of the data in the world today was created within the past two years. Throughout 2012, those data sets and others can be expected to grow exponentially. The amount of data being generated globally increases by 40 percent a year, according to the McKinsey.

The Data-Rich Landscape
Big Data has emerged because we are living in a society which makes increasing use of data intensive technologies. According to The Economist, there are 4.6 billion mobile-phone subscriptions worldwide and there are between 1 billion and 2 billion people accessing the internet. Basically, there are more people interacting with data or information than ever before.

The world contains an unimaginably vast amount of digital information which is getting ever vaster, ever more rapidly. McKinsey believes that the transformative power of big data will amount to a fifth wave in the technology revolution, building on the first four: the mainframe era; the PC era; the Internet and Web 1.0 era; and, most recently, the mobile and Web 2.0 era.

The biggest challenge of the Yottabyte Age will be figuring out how to make sense of it all. As technologies emerge to help harness the volume of these massive data sets, more will be unveiled. Our ability to capture, warehouse, and understand these astronomical amounts of information – or big data – will no doubt change science, medicine, business, and technology, and virtually every aspect of our lives.