Big Data is the Commercial Supercomputing in the Age of Datafication
Mar 19, 2013 / By Alex Gorbachev
I’ve been reading a book by my good friend Jeff Needham “Disruptive Possibilities: How Big Data Changes Everything” and it cemented some thoughts that had been forming in my head for a while and gave me bunch of new insights. Jeff managed to pack incredible amount of information in a very concise form. I thoroughly recommend getting a copy for you when it’s published (I’ve got author copy from Hortonworks at StrataConf in Santa Clara few weeks ago).
Supercomputers, and later High Performance Computing (HPC), have been around for a while. However, they were only available to organizations and projects that were extremely well funded. Military defense and intelligence departments are a good example. Oil and gas exploration that required enormous amount of processing to predict with high degree of certainty where to drill is another good example. The reason they can afford to invest in supercomputers is because the alternative was to drill expensive wells with little chance to hit the oil reserves. Since drilling a single well costs millions of dollars, companies can get high return on investment into supercomputing.
Modern commercial supercomputing in the age of Datafication is what we today call Big Data. I think a better term for it would be Data Supercomputing but the industry has already spoken so Big Data it is. The architecture shifted from environments that required massively-parallel compute-intensive number crunching to massively-parallel data-volume-intensive processing. The performance of processors and storage media capacity has been growing much quicker than storage and network performance while data volumes growth trumped all of them and this was triggered the shift in supercomputing architecture.
Hadoop is the first modern commercial supercomputing platform — it’s here to stay and evolve.
One comment on “Big Data is the Commercial Supercomputing in the Age of Datafication”
Pingback: Latest data Industry news round up, Log Buffer #312