What Is A Data Mesh Organizational Architecture?
Data Mesh is an increasingly popular concept among data platform specialists. Technological innovations and the popularization of Big Data in companies lead to new paradigms for data decentralization and consumption. In this sense, the Data Mesh organizational approach can help corporations looking to organize data teams.
What Is a Data Mesh Architecture
Data Mesh is a technical and organizational architecture approach aimed at the decentralization and large-scale management of an organization’s analytical data.
Why is Data Mesh Being Adopted
Blanca Mayayo is the Product Owner of Sidra Data Platform at Plain Concepts, has previously worked as an engineer and product leader in companies such as Adidas, Nestlé or Telefónica. In her opinion, there are several trends that are leading companies to take an interest in a new way of managing data:
- Companies want to differentiate themselves and provide value thanks to the data they possess.
- They want all areas of the company to take advantage of it, in an effective and efficient way.
- At the same time, data governance and data sovereignty aspects are becoming increasingly relevant.
Problems that Data Mesh can solve
Data Mesh allows facing several problems that companies have about data management, such as:
- Lack of clear ownership or responsibility for the data. For example, in centralized data warehouses or data lakes, technical managers do not have the specialized business knowledge to take advantage of and optimize the data.
- Lack of data metrics translates into distrust of the data to draw conclusions or make decisions.
- Difficulty in bringing engineering expertise to the rest of the organization. As a single team manages the centralized platform, this can lead to bottlenecks or friction between teams.
If these problems persist in the medium and long term, the situation leads to low use of data and difficulty in innovating or adding value.
Data Mesh Principles
The Data Mesh is built around four principles:
- Domain oriented property
- Data as a product
- Self-service data platform or infrastructure
- Federated government
The third and fourth principles are more technological approaches.
Domain-oriented property
A ‘domain’ is a department, section, area… of the company. In the principle of domain-oriented ownership in Data Mesh, the responsibility for data would go beyond the centralized data platform team, to bring this duty to those teams where it is generated (for example, the commercial area where customer information is ‘born’) and that could extract a broad and quality value from it.
Data as a product
The principle of data as a product in Data Mesh means conceiving data as a consumable product in the business.
These data as products have input and output ports:
- Input ports: Data-producing sources.
- Output ports: In charge of exposing the data so that other parts of the company or end users can consume it.
And not only this: the products have to be easy to use, with metrics and metadata. Moreover, they are offered in packages that include not only data and metadata, but also the code and infrastructure with which they have been produced.
DATSIS Principles
Within the data grid, these products are governed by DATSIS principles:
- Discoverable. The product has to be easily found through some tool, such as a data catalog.
- Addressable. To access it, some kind of generic or global guidelines must be followed.
- Trustworthy. To be trusted, the product must have quality and service standards.
- Secure. Effective granular access policies to this data must be defined.
- Interoperable. Ideally, products should follow open standards and multiple interfaces can be used to search and find the data.
- Self-describing. The package must include the enunciation of the input and output ports, as well as a product schematic and updated documentation.
Product Thinking
As in some work methodologies, applying the Data Mesh philosophy also helps to move forward to obtain a good final product or service.
In Mayayo’s opinion, perhaps the most important thing is to prioritize according to the delivery of business value, but there are other ideas to take into account:
- Multidirectional feedback (with the customer, with other departments…).
Continuous delivery
Continuous measurement of successes
To achieve all this, the ideal is to have functional and autonomous teams to deliver the products, as well as to focus on continuous improvement and innovation.
Self-service data platform or infrastructure
With a self-service data platform or infrastructure, the idea is to reduce the technological hurdles with which to produce and consume data, in turn creating and consuming products.
Federal Government
Federated governance in Data Mesh means that decision-making is as close as possible to each domain while maintaining centralized control. In this governance, responsibility is exercised by a multidisciplinary team, which includes not only data specialists but also those responsible for cybersecurity, legal or other departments.
This federated governance ensures this centralized control through standardization and policy interoperability. By being standardized, different teams are allowed to work autonomously.
However, some centralized governance policies will still be necessary. Mayayo suggests that there should at least be a set of critical data governance policies: standard nomenclatures for data paths, guidance for data modeling, documentation format…
Data Mesh Considerations
Data Mesh is not just technology
Data Mesh also means managing skills and knowledge to develop a product, learn from mistakes or set KPIs and OKRs to check progress.
Data Mesh is not for everyone
If there is already efficient collaboration between departments in the company, and this is effective and beneficial to the business, it is not necessary to implement an entire Data Mesh transformation.
There is no single Data Mesh
Data Mesh seeks to decentralize data ownership. We are likely to find different mesh formats. Thus, there may be different degrees of centralization and decentralization for both raw and transformed data.
What to take into account when technically implementing Data Mesh
When technically implementing the Data Mesh philosophy, it must be considered that these meshes not only distribute data in isolation but that data may be needed to integrate with other applications and tools of the company. Therefore, the Data Mesh infrastructure can also be considered as an integration platform, with different mechanisms such as an API.
In addition to that, as in any modern application design, it is necessary to take into account other cross-cutting capabilities, such as:
- Security
- Elasticity
- Access control
- Automated updates
- Monitoring
- Etc.
The self-service infrastructure can be seen as a toolbox with different utilities, such as catalog and data classification services, or infrastructure automation. In addition, scalable storage and computing is ideal.
Sidra Data Platform and Data Mesh
Sidra Data Platform offers an architecture design compatible with the Data Mesh concept. It is a set of Plain Concepts native tools, automated and adaptable to each scenario, that ingests, catalogs, manages, and integrates data in Azure, as well as accelerates the creation of value from data in the company.
Sidra supports Data Mesh’s self-service data infrastructure or platform principle. If companies decide to apply this philosophy in their day-to-day business, the software helps them achieve their goals of having various departments disseminate and manage data.
Using Sidra and its accelerators, it is possible to integrate data from different systems and build use cases by creating and deploying Client Applications. These modular applications distribute ownership of the data, while simplifying its production and consumption across domains.
Want to know more about Data Mesh?
In the next talk, Blanca Mayayo explains what Data Mesh is and how it is linked to Sidra Data Platform:
And if you want to know Sidra Data Platform in-depth, our team will answer all your questions:
