Ashby's Law for Data Strategy
Variety absorbs variety
Maybe you know the following quote:
All models are wrong, but some are useful.
George E. P. Box
In the following, I will take a look at “Ashby’s Law”. For sure it was created in a totaly different context (cybernetics). But such laws are empirical rules, not valid in every case but in most cases a good rule of thumb. So the following is just a thought on whether considering Ashby’s Law might be helpful in the context of data strategy and data management, or not.
Picture: Created by ChatGPT
If you are not familiar with Ashby’s Law or more complete “Ashby's Law of Requisite Variety“ it is as follows:
When the variety or complexity of the environment exceeds the capacity of a system (natural or artificial) the environment will dominate and ultimately destroy that system.
W. Ross Ashby
There was a update of his law by Boisot and McKelvey → "law of requisite complexity", that holds that, in order to be efficaciously adaptive, the internal complexity of a system must match the external complexity it confronts.
If we think of that for Data, what could that mean? I give two examples, open to discuss:
Today’s Roles in Data & Analytics - Data Experts (Data Engineers, Data Scientists) need a complex environment to work effectively (e.g. code-based, broad range of functions, high maturity) - The upcoming Citizen Data or Self-Service BI roles tend to need a simple interface (low-code/no-code) to work effectively. Think one step further, what does the LLM1 trend (e. g. CoPilots, AI assistents) means for these roles and is Ashby’s Law still valid? We know, LLM’s are incredible complex but they are able to deliver a simple interface in form of a ChatBot to adapt to different roles and needs. What are your thoughts on this?
The Right Data Platform - Complex organisations (large enterprises) or enterprises with complex requirements (start-ups, tech-driven companies) require a complex or decentralized data platform or data architecture (e.g. Data Mesh, Data Fabric) in order to be able to implement their requirements effectively, while for simple requirements (standard/ad-hoc reporting), a more simple and centralized data architecture (e. g. Data Warehouse) is required. Is the current Data Lakehouses trend a bridge to build for a broad range of requirements, supporting the evolution and maturity of an enterprise?
Current developments like LLM’s and Data Lakehouses show a move towards adaptable approaches for Data & Analytics, as they better supports scalability, different collaboration models and the evolution of the tech stack in a more flexible way than before.
What experience do you have with Ashby’s Law? Is it valid for Data & Analytics?
LLM - Large Language Model
Hi Peter, I just read this post (only 8 months late) and I am intrigued by it. Let me offer an opinion.
Data strategy and data management itself is based on a model of the organisation, an information processing model. I think this matches quite well with Ashby’s perspective. The implication of Ashby’s Law for data management is felt very much with all the “non standard” data solutions that exist in an enterprise. Personal Excel files, mailing lists, social media accounts, SaaS apps used by marketing, etc. There are often attempts to integrate these into a data platform but, in my experience, the variety of non-standard data solutions usually exceeds the capability of the “standard” data platform.
One way to look at this is through the lens of Ole Olesen-Bagneux’s Meta Grid. His Meta Grid is an Ashby-like variety of metadata systems.
Why do we do this in organisations? Precisely because of the variety that exists in our environment.
How might that impact data strategy? Well one way to view this is through the lens of “architectus oryzus” (an old Martin Fowler article, c.2012) where he talk about the (software) architect as needing to coordinate, link, collaborate across domains and systems, not acting as a centralised “architectus reloadus” (a play on the architect role in The Matrix Reloaded) controlling the centralised IT architecture.
This approach appeals to me. I try to emulate it in my work as a data architect.
It is also why I am such a strong fan of Zhamak’s data mesh idea, which has implications beyond analytical data in my view.