The Big Data Tsunami

Data Storage Growth Rate

According to EMC Storage with research and analysis by the International Data Corporation (IDC), the rate of data growth in the world is 40% a year resulting in a doubling of the total data stored every two years. They estimate that the total volume of data stored will reach 44 Zettabytes by 2020 – that is 44 followed by 21 zeroes.   Compare this with 0.8 Zettabytes in 2009.  That’s staggering growth!  This graph illustrates this visually.  Interestingly, the US stores a third of the world’s data according to the global economic news outlet Quartz.

You can probably guess the variety of factors contributing to this enormous growth.  According to the United Nations, global population growth is about 1% per year.  While population growth will contribute some to the growth of data, it is a small factor.  In 2010, only 1.7 Billion of the 6.7 Billion humans on this planet used the Internet, roughly a quarter of the population.  This is expected to grow to 5 Billion users (roughly two-thirds of the population) by 2020 according to Network World.  This penetration of the Internet globally will undoubtedly spur growth in global data.  In addition, the number of ways to be “connected” is increasing with the increase in wireless services and the advent of the “internet of things”.
​
Meanwhile what is to be done with all this data?  According to Gartner, data growth continues to be IT’s biggest challenge.  According to IDC, the amount of data generated outstripped storage for the first time in 2007.  This gap has continued to increase.  The amount of storage capacity available is increasing roughly linearly while data growth is increasing exponentially.  IDC estimates that, at the current rates of growth, there will be a 60% gap between available storage and storage demand.  Improvements in storage technology and innovation in storage media has reduced the price of storage exponentially.  According to SAS Business Analytics, the price per GB of disc dropped from $17/GB to $0.07/GB in 2012.  So is this the answer?  Continue to feed the beast, crank up production and meet the exploding demand?  Imagine the collateral impacts to data center construction capacity, power and air conditioning needs, etc.
​
While growth of data is a good thing from the perspective of global IT job growth, following the data growth curve blindly is not a good idea.  Here’s why.  According to IDC, only 22% of the data that is stored was considered “useful”.  D.J. Patil, Chief Data Scientist of the White House says, “Data is super messy, and data cleanup will always be literally 80 percent of the work”.   Think about the amount of storage that is wasted in storing dead or dormant data that has no business use.   Useful data is expected to grow to 37% of stored data by 2020.  That’s only a growth of 10% per year as compared with the 40% per year of overall data growth. So maybe we should be finding ways to continuously remove dead data and identify useful data.  What do you suggest, so we avoid making our data warehouses into data landfills.
 
The rate of data growth in the world is 40% a year resulting in a doubling of the total data stored every two years.   IDC states only 22% of the data is actually useful.  Join the discussion on how to prevent data warehouses from becoming data landfills.

---

Sai Balakrishna

​About the Author
Sai Balakrishna is the Managing Partner and Chief Executive Officer for ProcureVision, LLC. An expert in strategy development, business process optimization, program and portfolio management, Sai has over 25 years of management and leadership experience from Fortune 100 companies including American Airlines and Verizon. 
​