There is a lot of buzz around Big Data and the NOSQL movement these days and rightly so. The issues with data have essentially been two-fold: find cost effective ways to store ever increasing amounts of data and information, and find ways to mine this information to extract meaningful Business Intelligence.
This problem has been compounded by the emergence of web 2.0 technologies whose legion of loyal fans who can number into the millions generate copious amounts of data every minute, and by the time you realize it you have gigabytes and terabytes of data in one single day. Obviously, this calls for very radical departures from the current state of the art for data storage and mining technologies.
While traditional IT houses not of the web 2.0 stripe may not face this sort of real estate issues when it comes to data storage, mining that data for meaningful intelligence is still a work in progress and a major headache no matter what the size of your Data Warehouse. So while you may not want to be on the bleeding edge and opt for a grid based MPP solution for your ever increasing storage needs, you will certainly want to take a serious look at the emerging Algorithm and Heuristics driven data mining techniques led by Map/Reduce.
Map/Reduce may yet be your killer app that can be the panacea for all your Business Intelligence ailments. This is very serious stuff. If Google has bet its house on it and has made this the foundation for their search technology, then you better believe that this is very strong medicine.
Using traditional relational database technology to cater to your Big Data data warehousing (DW) needs is now quite well known. It is not easy performing operations between databases, especially if they span networks. Try performing a join between two database instances and you will know what I am talking about. To solve these issues, there are custom solutions from vendors like Teradata and Netezza. The barrier for entry is still quite high in adopting these systems, however, both in terms of license fees and setup and maintenance costs.
There is an alternative. We are now in the era of framework-based DW, DIY DW and DW in the Cloud. The current set of tools and technologies that have emerged have helped democratize this domain which was for long the exclusive preserve of a few select vendors. The revolution was led by grid-based implementations adopted by the leading players like Google (Bigtable), Facebook (Cassandra) and Yahoo (Hadoop).
Hadoop has emerged as one of the most popular Map/Reduce based open source frameworks for Big Data and several Information majors have adopted this technology. Beware that this is a framework and may need significant amounts of customization and programming to get it to do what you want. If Hadoop is not your cup of tea, then there are similar implementations like AsterData and GreenPlum which work on the same concepts but can get you up and running very quickly with their own abstractions libraries like SQL-MR and intelligent dashboards for easy configuration and maintenance. Another very appealing feature of these offerings is their ability to be hosted in a Cloud so all your advanced analytic needs can be performed off premises.
Speaking in a broad sense, there are three general flavors to choose from when it comes to Big Data solutions:
* Custom build BigData frameworks like Teradata and VLDB implementations from Oracle that are proprietary frameworks designed to deal with large datasets. These frameworks are still very relational in orientation and are not designed to work with unstructured data sets. * Data Warehouse Appliances like Oracle’s Exadata. This introduces the concept of DW-in-a-box where the entire framework needed for a typical DW implementation (the Hardware, Software Framework in terms of data store and Advanced Analytical tools) are all vertically integrated and provided by the same vendor as a packaged solution. * Open Source NoSQL-oriented Big Data Frameworks such as Hadoop and Cassandra. These frameworks implement advanced analytical and mining algorithms such as Map/Reduce and are designed to be installed on commodity hardware for an MPP architecture with huge Master/Slave clusters. They are very good at dealing with vast amounts of unstructured, text-oriented information. * Commercial Big Data Frameworks like AsterData and GreenPlum, which follow the same paradigm of MPP infrastructures but have implemented their own add-ons such as SQL-MR and other optimizations for faster analytics.
