It’s a fact that technology is always evolving—rapidly. What’s new and hot today, may be old news and on its way to becoming obsolete tomorrow. Traditional data warehousing is no exception. We have been seeing that the old school data warehouse is on its way out and a new data platform approach is taking its place. We’re also seeing the rise of several trends and have a few thoughts on why they're happening and how we think the industry will respond over the next several years.
Why are data warehouses becoming obsolete?
There are several factors causing data warehouses to meet their boundaries. A few of them include:
- An increase in demand to acquire data when it’s needed. Traditional warehouses were designed to consume flat file structures and data from other relational systems. Today, that’s no longer the case.
- The variety of data sources has increased dramatically and this has pushed data warehouses to the edge of their capacity, in terms of how fast data sources can be acquired. Some of the legacy warehouses can't keep up with this real-time demand.
- The type of users working with legacy warehouses has changed. Data scientists are one of the biggest and newest groups and they have very specific requirements for the data they use. They need way more processing power than a traditional warehouse may be able to give. They also need the ability to use the tool sets they're most comfortable with, which, in many cases, warehouse databases don't support. One of the top requests we hear from data scientists is they want access to raw data.
- More automated systems consume data for analytics and existing warehouses are just not well-suited to serve this type of workload.
- Maintaining traditional data warehouses is a capital expenditure. And maintaining a modern data warehouse on the cloud is an operational spend, which many organizations find an attractive alternative.
- Lost business opportunities
- Unhappy users
- Performance problems
- Rising costs
- Shadow IT
- Limited visibility into KPIs
