Data Platform Market Transparency

Or The Art to Stay Up-to-Date While Permanent Change

As a Data & AI Strategy Consultant I don’t focus on a single tool or vendor. In todays world where nearly daily many vendors throw out a wealth of new components, features and capabilities. How to stay up-to-date?

While we are moving to an AI-augmented consulting mode, AI can make it easy to collect, structure, visualize and update information.

In my case I had several tries but currelntly I work on a Data Platform Viewer, helping me to master this challenge. Let me introduce, how it works.

The Vendors Which Matters Most

Which vendors matters may be individual. For me I see no limit. Currently I extend my base of 82 vendors in different areas with Open Source projects relevant and significant in the Data & AI world.

Fig. 1: 82 Vendors on my list and regularely updated from what happens in the market.

My typical update run ist 10 vendors and for every vendor there is an information about the last run. Therefore I can be sure, search focused in the right timeframe and shouldn’t miss much news.

Fig. 2: Config file for steering the next update run

Currently I run it manually every few days to hold the effort and costs limited. It runs via Claude Code and Sonnet 4.6 at the moment.

The Knowledge Base

Every concept may it a component, feature, capability but also the general concepts behind is covered here. Currently there are > 800 short articles here:

Fig. 3: Snapshot of some articles as a base of data platform knowledge

Every article is a single markdown file including German and Englisch description. The idea is to keep it short for fast understanding. As every article is connected with further concepts, Furthermore the format makes it easier to reuse the knowledge in other context and having a cleaner separation of content and visualization.

Fig. 4: Every Knowledge Base article connect to others and to sources

Furthermore every article allows to jump directly into the knowledge graph and highlights the concept in its context.

The Knowledge Graph

There are several ideas behinde buildingthe knowledge graph:

1. It shows knowledge connected to other concepts. I can travel visually through the concept room and get a big picture while having the posiblity to dive in on demand.

2. When creating articles I use the 1st and 2nd level connections for grounding to enhance quality of the result.

3. I read several articles from Jessica Talisman, MLS about Semantic, Ontology, Knowledge Graph and want to try out concepts like Controlled Vocabulary, Taxonomy, Ontology and Knowledge Graph.I’m still a long way from fully realizing the potential of these concepts here.

Fig. 5: Building a knowledge graph to better understand the big picture

I can filter based on specified categories to focus more on certain aspects, jump into the knowledge base article or see if there are research articles about.

The Taxonomy

I made it simple with the taxonomy and just build a hierarchy of typical categories in the context of Data & AI Platforms. I can filter the result area with every branch or leaf to see the relevant concepts beneath this category.

Fig. 6: Viewing a taxonomy perspective

Additionally I use the taxonomy as enrichment within the article itself where I can always jump into the category for similar concepts or as in a vendor component overview to have context for the term itself.

The Changelog

IAs I work iteratively and in an learning and exploration mode to find out what is possible, I found it helpul, to see the changes from the daily run. This is just a small but incredible helpful function.

Fig. 7: The Changlog make aware what really happens in the market

As my updates are typically based on what is new in the last few days, the change log gives a perspective on what is happening in the market.

The Articles

Now all these functions shown are bites of knowledge connected in several ways. Bus sometimes I need a deeper connected understanding. Therefore I use differnt storytelling modes to create articles based on the knowledge graph connections to have a clear grounding. Based on the graph knowledge and checking additional web knowledge an article is build, giving a wider perspective of a certain topic.

Fig. 8: Articles are based on the knowledge and connections

Furthermore as I have some further independent knowledge bases I sometimes mix those to include certain perspectives.

Conclusion

To make it short after some time of building it now really helps me to explore new concepts, understand current marketing terms from the vendors, easily see similar concepts from other vendors and help me to understand new and innovative concepts in the context of what happens in the market.

Just one example what is possible leveraging AI. I think the approach has some similarities with Karpathy’s LLM Wiki. I can directly use my knowledge base in Obsidian but I wanted to implement my individual knowledge and semantic concepts to have the best fit.

---

Do you have similar approaches or apps where you use AI e. g. for managing all the knowledge you have tho handle?