Transformation to a Data Performing Organization

When enterprises face new challenges needing a strong support from data, analytics and AI or stuck in a long grown system landscape and data organization, often changing in small iterations and steps won’t help. Then it could be the time to craft a data strategy and start a Data Transformation Journey.

Data Transformation Journey
A Data Transformation Journey is a structured, organization-wide process that aligns people, organization, data architecture, and culture with data-driven goals. It integrates change management to foster acceptance and capability, and redefines organizational structures to enable agile, cross-functional collaboration around data. This journey ensures that data becomes a central asset in strategic decision-making and operational efficiency.

In the following I put together different data transformation experiences to reflect on the different perspectives of such a journey.

Challenges Of Modern Data Organisations

I recently discussed a clearly grown data architecture (20-25 years) with a customer with two major data platforms, several ETL frameworks incl. self-service data prep and multiple frontend tools from BI to data science and data apps, used by strong business domains. From an architectural pattern perspective we had a mixture of data warehouse, data lakehouse and data fabric.

Some of the challenges:

• Different platforms are optimized for different tasks - e. g. while SAP systems can handle business data and business context very well, what is sometimes hard to rebuild, a platform like Databricks have some challenges with smaller data.

• New and successor technologies from the vendors like SAP Business Data Cloud are on the watchlist, but concepts changed and maturity isn't ready in every area.

• It is hard to get a consistent data security concept in a rather distributed landscape over multiple tools and data silos.

• Not every domain is on the same maturity level to leverage technology on their own.

• Through different specialized operational systems, overarching terms and concepts are often not harmonized.

• Self-Service and domain-ownership is increasing and changing collaboration modells between IT and business.

• First ideas of data products are discussed but currently in a very initial state.

Sometimes you can do everything right but end up with a bunch of challenges. Here, no one will sell you the perfect and ready solution. Often you can just make a plan and clear your priorities (aka crafting a data strategy) and realize it step by step. Without a clear north star, complex landscapes are getting expensive and inefficient over time.

Consider The Data Teams

In larger data transformations, where a lot of things are moving and team structures, working modes, data tools and so on are changing, it can become hard, to get people where they should be.

In a current situation I experience data teams doing hard to shift to the new world:

• Falling back to working in the new team structure - on old infrastructure topics rather than moving things to the new platform

• Focus on deconstructing or also rebuilding the legacy platform - rather than building the new one

• Prioritizing short term operational requests - over long term strategic skill development

• Calling for projects - rather than building and operating the new data platform

It is not the plan and the understanding about the goals. Changing in general is hard. We expect data people to be open for the new tech and throw themselves on the latest innovation. But this is often wrong. Change and transformation needs to be seen as a continuous and long term process which needs leadership and guardrails on the one hand, and individual responsibility and intrinsic motivation on the other hand.

Create A Single Point of Truth

When it comes to a data transformation, many streams need to work together like:

• Data Platform – architecting and building the right platform for what you need, while typically handling the legacy platform

• Data Operating Model – transforming the data organization the way, needed for delivering the performance on data you aim for

• Data Product – building the projects with data, which realize the highest value for the organization (often today based on a data product-oriented approach)

• Change Management – communicate and support transformation of culture and behavior in the context of the streams above (critical but often neclected)

While many parallel activities need a loose coupling to go forward, we need a connecting point of how everything glues together. Goals, principles, processes, organization, meaning of the technological concepts in these contexts.

Each of the streams can include many activities. It is important to have a central document or point (of truth) for everyone, where the living truth during such a transformation is documented and everyone can come back to. “Living” because you can never bring such a document from outside. It always need to reflect possible new concepts to existing culture and context.

Priorities and Realities

Sometimes things get complicated and you can think your stakeholder is your best friend but he or she also has to justify somewhere, too. Even if you think you did your best job, build the best data platform or AI solution, the priority is not the same for everyone.

• Communication – It is essential to talk with people, understand their needs and deliver on that. E. g. for the Head of Data and the CIO the needs can be very different. To communicate through just one side is very common for different reasons (availability, hierarchy, …), but is an anti-pattern at the same time. Following the way to write things down, communicate what you want to communicate and than do it this way is helpful.

• Clarify expectations – As an external we are basically responsible for the project we deliver. Our stakeholders can have a different perspective. Especially if you do a large strategic data transformation it is maybe not just about the external project you have a order about, like building a new DataPlatform. This is maybe just a part of the whole. Sometimes the big picture including multiple external and internal activities is the focus, like the replacement of legacy systems.

• Being honest – Looking back, it is easy to say we did wrong decisions. This can happen if you priorize the wrong stakeholder or as a external elevating the relationship above the results. You should always make clear what it means to not make things, even if it result in personal circumstances. There will always be conflicts of interest. As external you can sometimes just care about that it is documented and consequences are clear. Maintain a project decision record or something similar.

• Expect nothing – A transformation is characterized by changing behaviors and breaking up existing structures. People tend to forget and fall back to established behaviors all the time. You can explain a concept like DataProduct several times, but should never expect, that people understand it the same way you do. Especially if they don’t live it yet, or are not effected at all. Highlight the context as often as needed during a data transformation journey.

Cost and Value

The cost of a data transformation depends on many aspects.

• Is it a technical move or an organizational?

• On what stage of maturity is the organization already?

• Do you change a central approach or a decentral or move from one to another?

• and many more…

To estimate such costs can be a though act. Do you go bottom-up with estimations and interviews? Do you go top-down by creating a model and work e. g. with t-shirt sizes and complexity levels?

People want to know what they get and what they have to pay for. If not in the beginning, so earlier or later.

Conclusion

A Data Transformation Journey can be complex, expensive and being a long-term endeavour. Above I just mentioned some of the possible relevant aspects. Do not underestimate what is needed to reduce the risk of a premature termination of the journey. As a result you will have a lot of costs, but not really a change or in the worst case, a change for the worse.

---

This article is based on different parts of my recent weekly LinkedIn articles.