The Current State of MDM

MDM and Data Quality have had a long-standing relationship. The first MDM systems in the early 2000s all had data quality functions built in. Many then made external calls to Data Quality tools for certain functions, like address and name standardization and validation. Over the last 20+ years, the relationship has grown deeper and the functionality much, much more impressive. Today, most modern MDM tools have extensive data quality capabilities. They have the ability to profile source data as it is loaded to MDM. Beautiful visualizations and easy-to-use interfaces allow data stewards to understand and drill into a variety of MDM Data Quality issues. Missing values. Frequently appearing values. Disallowable values. Disallowable patterns across values. Duplicate records sorted by match scores. And so on. It’s all very impressive. But has it translated to better MDM Data Quality?

The answer appears to be a muted “somewhat”. High likelihood matching records have been consolidated. Bad data has been removed. Data is stored in standardized formats. However, most organizations have very long work queues for their data stewards. Completed work is dwarfed by incomplete tasks. And the tasks keep on growing because data is created ‘dirty’. The greatest DQ success in MDM has been data standardization, with near 100% completion. Why? Because that task (a) can be fully automated and (b) isn’t controversial. Nearly every other DQ task isn’t automated and does require a lot of debate. Let’s take the example of a customer record who has an invalid SSN of 999 999 999. This was obviously a data entry error. How does a data steward resolve it? Well, they’d have to look at data records in MDM, and ultimately in source systems, to track down whether there is a valid SSN in another source system. They might find the SSN in a different text field. It’s a 9 digit number, but is it an SSN? More investigation is required. During that time, probably 5 more records were created in a CRM system with bogus SSNs. The problem just gets bigger. Most companies have 4-5 data stewards assigned to MDM. In an average size deployment of 1,000,000 customer records, it is likely that 20% are flagged as potential duplicates, and 30% will have at least 1 data quality issue that requires manual resolution. That’s 500,000 tasks. 100,000/steward. That’s a HUGE amount of work. 

MDM Data Quality requires hyper-centralization. MDM DQ functionality is very centered on MDM. Data must be loaded into MDM in order to be profiled. Data quality issues are fixed in MDM, and then the cleaned data is exported to other systems to be used. So in order to benefit from better data quality, organizations must commit fully to putting MDM at the center of their data architecture. The fact is that MDM isn’t at the center of every data pipeline. Data is profiled, cataloged, and cleansed and loaded into cloud warehouses all the time. And the data quality rules are often redundant with those in MDM. It’s a duplicated effort. 

While MDM and Data Quality have had a long relationship, it seems like the right time to update and renew their vows. MDM Data Quality has been centered in MDM. The trend of Data Engineering & Pipelines suggest a lot of Master Data Quality needs to take place outside of MDM. Perhaps those rules will evolve to be based more around metadata and occur before data is loaded to MDM. Profiling data and fixing issues before data is loaded to MDM ought to become a new best practice. The trend of Data Observability is driving an overall, enterprise-wide view of data health. Data Quality, including MMD Data Quality, is a key part of health. And it would seem that MDM Data Quality needs to evolve to be observed, and managed, as part of an overall data ecosystem. Finally, MDM and DQ need to become more decentralized. Master Data Quality needs to be managed and improved across the data ecosystem, not just within MDM. That will lessen the need for extensive source and consuming system integration to benefit from MDM data quality. It will also improve the quality of data ingested into MDM. 

MDM Data Quality is an important aspect of data management modernization. Despite their long-standing and strong relationship, evolution is required. That evolution must not take place in a vacuum. Rather, the evolution of MDM Data Quality within your overall data ecosystem will be a critical success factor for current and future initiatives with business applications, analytics and AI.