The Future Data Stack
How to bring data to the core of business
Data landscapes change constantly over time, but technology can bridge the gap between the two extremes. In business, we are confronted with recurring trends that do similar things with the technologies that are relevant at the time.
Fig. 1: Data usage moves back and forth between centralization and decentralization over time. (credits to BARC, 2024)
Often decisions for a data stack lies between two sides:
Saving costs vs. making revenue
Freedom for independent business units vs. central control and oversight
Being efficient vs. being effective
But the meaning of data ist different today, so technology is. Data is still often a support factor doing reporting, having some dashboard with KPIs to steer the business. While we have interaktive data tools today and even augmented analytics, most companies are still on a relative low data maturity level.
Fig. 2: Data is increasingly becoming the core of value creation
But we see the will to change things. After years of digitalization, data is getting more important for the business. While having data as core of the business model can be seen as the highest degree of data usage, being data-driven is the ideal of our time. Many enterprises want to make this step.
Many data stacks are still build for efficiency. They follow technical requirements and are build layered to support a central team delivering the single source of truth.
Fig. 3: Technology as an enabler - the business domains are taking center stage
The idea of a central data platform is still great. But we have to enable decentralized data teams at the same time bridging the gap to todays needs to support them the best way. This means decentralization where necessary, supporting the business, lifting data quality. At the same time we centralize overarching functions and needs like transparency and data security, delivering new features like AI fast for everyone. The currently very popular data product approach turns out to be an enabler if done right.
If we see the data platform as a product too, we can build modern cases and enable the business in adopting innovations fast bringing data even more to the core. This means the data platform has to deliver fast and be able to evolve.
Fig. 4: Data & AI architecture for new use cases in retail marketing
In the customer case in Fig. 4, we have seen a fast adoption of new technologies like AI in the retail area, which was not able in the legacy data warehouse before. With the monolithic data warehouse they were not been able to deliver new functions fast what leads than to a fragmented and uncoordinated data stack with local systems and services. A modern data platform has to deliver features in an integrated way, but enabling integrated ecosystems to optimize to the companies needs at the same time. In general modern data platforms like Databricks and Snowflake are build to deliver that coming from the Modern Data Stack history. But also more integrated stacks like Microsoft Fabric or SAP Business Data Cloud can be a great fit for many companies integrating well into existing ecosystems.
What do you think about the changing trends between centralization and decentralization? Can modern data platforms bridge the gap?
This blog can also be found in German language on the INFOMOTION website.