DATA APART TOGETHER
A federated, actively collaborating “mesh” of data producers and data consumers – owned and governed by the business domains themselves – brings data as close as possible to where it is picked up and used, a hallmark of a true Technology Business
Poor data people in the center. Caught between a rock and a hard place. So many different sources, uses, and perspectives of data – all seemingly changing overnight. Why not fully embrace diversity, and create a much more federated business take data on? Emerging concepts such as “Data Mesh” move the ownership of data to the business domain themselves, where data is best activated. It stimulates these domains to manage data as a first-class product and share them through lively internal and external marketplaces. And all of this enabled by open, standard, enterprise-scale platform services. It really is the best of both worlds.
Yashowardhan Sowale Expert in Residence
WHAT
- As an alternative to centralized data management, federative approaches emerge that stimulate business domains to truly “own” and manage their data and actively collaborate with internal and external partners.
- Data Mesh is a leading approach – building on the notion of loosely coupled mesh networks – shifting the ownership of data to the business domains that typically have the best subject matter expertise closest to the sources of data.
- Data Mesh also propagates domain management of data as a first-class product, not only providing quality and timeliness, but also making their data products available to internal and external consumers through a compelling, self-service user experience.
- To enable federative data management, a rock-solid platform of open standards and services is needed to make data findable, accessible, interoperable, and reusable – it’s a non-negotiable prerequisite for the data autonomy of domains.
- Platform services include data catalogs, marketplaces, metadata management, graph navigation, data lineage – as well as microservices, APIs, automated data pipelines and stream processing, and data virtualization for easy access.
USE
- Online fashion retailer Zalando implemented Data Mesh principles to guarantee accessibility and availability of data at scale to the business domains – actively pushing managing data as a product by these domains.
- Using a cloud-distributed Data Mesh, ABN AMRO bank gained flexibility in managing the needs of cross-functional teams at scale while eliminating silos – acknowledging that business domains access the same data sets for different purposes.
- With a Data Mesh Platform, Netflix Studio decreased the lead time for diverse studio teams to create a new data pipeline while new support features were offered such as end-to-end schema evolution, a self-serve UI, and secure data access.
- DPG Media took an agile approach to Data Mesh through the notion of data products, domain data stores, and domain-based governance. The company can now extend its data-powered culture across the entire organization.
IMPACT
- Access and ownership of data as close as possible to the business brings increased responsiveness, responsibility, and agility, without giving up on enterprise-scale governance – it derives more value from data, closer to the business.
- Self-service and data pipeline automation increase the productivity and cost-effectiveness while decreasing time to market of creating data solutions. It also resolves the scarcity of highly skilled (central) data experts.
- Ease of finding data products within the organization enables producers and users of data products to collaborate more effectively, resulting in better business outcomes with data.
- Embracing not only ownership – and data product management – by business domains is a crucial step towards creating an organization-wide data-powered culture.
TECH
- Data virtualization and federation: Tibco, Denodo, RedHat, Oracle, SAS, Actifico, Atscale, Data Virtuality, DataMeer
- Data sharing: Microsoft Synapse, Informatica, SnowFlake, Databricks, Baffle, Cloudera, Vendia, Delta Share, Azure Data Share, Datavant, Demyst
- Data collaboration/Data Mesh enablers: DataPlex, Atlan, Cinchy, K2View, IBM Data Fabric, Informatica Intelligent Data Management Cloud, Infosum, Alteryx, Snowflake, Box, Omnisient, Duality, StarDog, Harbr, TIBCO, Gigaspaces, Starbust, Dataiku, Nexla
- Major cloud providers: AWS, Microsoft, Google, IBM, Oracle, SAP