Do You Even Need Big Data?
It’s no surprise that the rise of big data coincides with the rapid adoption of cloud computing technologies. The flexibility of cloud computing allows resources to be deployed as needed, and as a result, firms avoid the tremendous expense of buying hardware capacity they'll need only occasionally.
But the first question any company needs to ask is if they will benefit from big data analysis. To do that, they need to begin by understanding the data sets available to them. For example, analysis of twenty years of stock closing prices would not likely require the power of big data systems. Given the relatively small size of this dataset, analysis can and probably should be performed using SQL or even simply Excel.
For historical analysis, high-powered data warehouses are incredible tools. Unlike databases, which are designed for simple storage and retrieval, data warehouses are optimized for analysis. In other words, data warehouses are built, from the ground up, to answer questions using stored data. As a result, they are much faster and efficient at performing complex analysis than a simple SQL database.
But large sets of unsorted and unordered data—such as financial transactions, production output records, or even weather data—do require big data analysis to bring order to the chaos and shed light on relationships, trends, and patterns made visible only by systematic and structured analysis.
Where to begin? To start, formulate a relatively simple hypothesis and use big data analysis to test it. The results of this analysis should reveal information that will lead to further, more complex questions.
Cloud computing has given businesses access to computing resources that only a few years ago would have been unavailable due to cost. This availability, along with the tremendous flexibility provided by cloud computing, has brought big data analysis to the masses. As technology continues to advance, the key for many businesses will be to determine whether they truly need big data—and if they do, whether a mix of cloud and big data is the right solution for their use cases.
