Big Data – Is this the end of the road for traditional Data warehouse?

Data Warehousing is close to my heart as I’ve spent a significant part of my career building, modelling or architecting large and complex data warehouses and analytics systems. If I am not building one, I am certainly using one. Finding the data sources, cleaning and loading them into structured star schemas and deriving intelligence has been great fun, besides solving some unique problems along the way.

The data that we handled (a terabyte a month) then in comparison to the data that we are handling now (a terabyte a day) just tells the story in terms of data explosion. 12 TB of tweets and about 500 million call records every day are just a few things that I keep reading from time to time.

The more I look back, however, the more I feel that the Data warehouse approach was (and still is) too expensive, too complicated and too slow to get the data to where it’s needed. The thought of waiting for the data to get into a structured database to even begin acting on it is like waiting for the vegetables to be carved up intricately before it’s served. A certain amount of latency was understood by the business as necessary in the case of a Data warehouse.

And the worst part is that some data (e.g. call recordings or unstructured email messages) that don’t lend themselves to traditional relational databases and cannot be dealt with in traditional ways are simply brushed aside or in some cases expensive technologies are used to mine them separately. A certain amount of constraint (e.g. space, structured data etc.) is again understood as necessary to operate a Data warehouse.  Some would argue these things can easily be fixed (like throwing more hardware). Well, I agree, if you have deep pockets.

Data warehousing, and in particular relational databases, for the last two decades has been seen as the only solution for pulling together disparate data sources to make business sense, until of course Google and Facebook came along and disrupted things a bit with Big Data technologies. They have led the way for a new form of storing and searching data that don’t use any databases, runs on cheap commodity hardware and the best part is: it doesn’t cost half as much and is not constrained by size, volume, where the data lives or whether the data is structured or unstructured.

Well then, is this the end of the road for big Nellie? Well, with some reservation, we think it is. We say this with much excitement, of course, as it will give way to new technologies and will force the traditional database vendors to innovate (to be fair they already are!). We don’t think the Data warehouses or relational databases would be out any time soon and there will always be a place for it standing shoulder to shoulder other Big Data technologies.  In future, however, data warehousing or relational databases may not be the default choice or the only choice available to data crunchers.

You Might Also Like
Battle tested technology.
Use it just the way you want it.

Whether you are just starting out or you are miles ahead and want to optimise your customer experience, you can use our technology just the way you imagine it. In multiple ways for multiple use cases.

Native Mobile SDKs

Offline Android and iOS components for identity capture. Works without internet connection. Quick integration into your native Apps. Tested in over 1000+ mobile devices.

View SDK Documentation video kyc
Cloud APIs

Restful APIs that can be integrated instantly without worrying about infrastructure or auto scaling. Our battle tested AWS environment is ISO 27001:2013 certified and monitored 24x7.

View API Documentation video kyc

Use our technology deployed as Docker containers in your own servers. In this set up there are no external calls outside your servers giving you total control over your data.

Contact Sales video kyc
Cloud Dashboard (no-code)

Get started instantly and begin your identity verification projects. The dashboard provides you with everything you need to onboard your customers as per prevailing regulations.

Book a demo video kyc

Trusted technology platform.

Trust is hard to earn. We certainly do not earn them through paid advertising. Instead, we earn your trust by providing a high-quality product and reliable service that you can count on. Every single day.

Award winning technology
Patent pending technologies matured over ten years with proven accuracy, quality and scale.
Support that truly supports
Whatever it takes, we are here to help you succeed with our tools and services.
Secure enterprise platform
Use our cloud platform to get started now. Or deploy this within your own premises.
Pricing that makes sense
Pay per transaction with discounts as you scale. Or annual subscription with unlimited usage.

Trusted by 150+ customers worldwide

Book a free demo

Built for flexibility, compliance and reliability to serve multiple industry segments.

Gig Economy