New Trend Pushing Further Decommodization of Business Intelligence

For several months I've noticed a new trend in Business Intelligence that I was finding hard to concisely articulate. Some research on Google led me to the right buzzwords - "Information Post-Discovery". Information Post-Discovery is the most exciting shift at the forefront of the amalgamation of Online Analytical Processing (OLAP) and Enterprise Search.

"Pre-discovery" of information typically relates to the design phase of a BI project that might include extensive modelling of data structures, the design of data extract definitions and ETL data contracts. These design processes are employed by the architect to solely understand the scope of data to be stored in the OLAP system and ensure that a justifiable outcome is delivered. As most seasoned BI professionals are aware, the Planning stage in any BI project (in which the above design artefacts are delivered) is the most time consuming.

Enterprise Search products such as Microsoft FAST Search, Attivio (led by Ali Riaz former director of FAST Search) and Endeca offer a new storage structure that leverages inverted indexes a.k.a. n-gram indexes at the basic level. These products are designed to identify and present user-explorable relationships in structured and unstructured data that can be seamlessly fed to the engine via connectors. Information is therefore "post-discovered". The "Navigator" mechanism in FAST ESP is one such example where users are driven to explore relationships in information that they did not know existed.

While I do not believe there will be an immediate, major shift to post-discovery systems, these systems certainly present a very compelling proposition to I.T. decision makers who struggle to enable transparency across enterprise information silos.