– already seen this, already been there ?
Magazine: Business World 10/2006
Author: Michal Klaus, Adastra Corporation
Key words: Master Data Management, Data Quality
Any organization with a few systems and several hundreds of customers has been facing this nightmare for some time – portions and disparate copies of information on key data entities scattered in applications across the whole enterprise. Naturally, the idea of „extracting“ those key data elements and putting them in one place is not new. So WHY is it coming back, is it the same thing under different name ? Not exactly – few thing have changed, few prerequisites are now in place. More mature technologies, critical volume of good and bad projects and - last but not least - willingness to take the data issues onto the board level. Hopefully Master Data Management brings new energy and fresh air, combining best practices in data integration, data quality, application integration and business process reengineering to turn the old idea into new reality.
Is it really worth it ?
Huge, IT driven project without business case ? Are you crazy ? Perhaps when forced by SOX or Basel II, and even that only after it is accompanied by a reasonable business case.
Why would business leaders give „yes“ (and resources) to something that looks like risky redesign of the whole IT architecture ? Is it that valuable to have customer and product data managed centrally in one place ? Well, looks like this is exactly what business leaders do and they know why:
Marketing, Sales, Customer Care – critical business functions and departments, that only needed „its own“ portion of data yesterday. Today they need as complete and trustworthy data as possible, and they need it with no delay. Especially in retail organizations. Otherwise the customer may get nervous and turn away to the competition.
Whether customer calls by phone, browses company web or enters the branch store, all contact points should treat this customer with exactly the same level of service, knowing her preferences, having the same, complete information on her history with the company, etc. This business requirement is in the core of all those unified customer care, multi-channel architecture projects in banking. Similarly, telecommunication operators are solving the tricky real-time integration of CRM, self-care systems through billing down to network infrastructure. The simple business requirement underneath? To allow the customer easily order and activate lets say mobile internet connection and start using it immediately.
Risk, Fraud – once again departments with a real business need for integrated data, moreover integrated data enriched by complex risk scores, customer value indicators and reproved credit limits. And they need it operational, on-line - when approving credit, monitor suspicious transactions or in the service-activation workflow. More often than ever the whole process runs without human interference, leaving no space for erroneous decision based on poor quality data.
Regulatory an legal requirements such as SOX, Basel II and others also require a lot of integrated and really provable data.
MDM structure and contents (as we understand it)
All project and initiatives mentioned above build on the presumption that the data they need is:
- available no matter where in the organization is their origin
- of reasonable quality, i.e. reflecting the reality,
- reliable and trustworthy,
- always available with almost instant response times,
- up to date at any moment
Let’s add few, reasonable requirements, such as:
- provided data should by independent of changes on the side of their providers
- any new or changed requirement on data should be implemented in a few days
From the business perspective these are all logical, easily understood and relatively simple requirements. From IT perspective, with many systems, complex relationhisp and constantly changing environment, the problem to be solved is really a complex one. MDM concept and principles evolved as a new attempt to answer the business requirements while respecting the complex IT nature
There is not a single, commonly accepted understanding of what exactly is MDM . Based on experience from project we implement for our customers., Adastra view of MDM space can be described by following matrix:
MDM: Master Data Management
CDI: Customer Data Integration
PIM: Product Information Management
Does it really exist ?
MDM is definitely “hot” in terms of marketing hype. The real world, however, looks little bit different – there are only a few projects covering all aspects of MDM as described above. At the same time, almost all large IT projects in last 2-3 years include some kind of customer or data integration.
Customer data is alone of the key company assets– and most decision makers are beginning to recognize that. Thus a project including customer data consolidation easily gets attention of top management sponsor and necessary resources. In addition, customer data focused MDM initiative brings nice added value by converting more entitites than just customer itself.
Well designed and implemented CDI architecture can serve as a master database for customer and partner entities, as well as sales agent, employee, etc. In generally, when designing CDI solution, IT architect should plan for all entities of the type “person” and “organization”.
Thanks to several pioneers, who were brave enough to test different approaches to CDI, we can now built on their experience, because customer structures are fairly similar for any business vertical. So far, majority of live, functioning CDI solutions are custom-build, combining best of breed technologies, best practices and internal and external know-how.
Several ERP (and CRM) vendors are announcing “packaged” CDI solutions, but their maturity and usability is very limited.
Product oriented MDM (also called Product Information Management) is a different story, considering very dissimilar structures of product data in various verticals. Telecommunications company products have very little in common with financial services products. And think of the complicated, multi-level hierarchies in manufacturing industries. This diversity means that finding similar project to learn from is much more difficult than in CDI. Once again, there are “ready made” PIM solutions being introduced. In this case they might be more functional, thanks to vendors specialization usually on one or few similar industries.
When it comes to centralized management of other key entities and reference tables, organizations typically deal with it in a more practical, less sophisticated ways. Master database covers more, less complex entities with trivial unification and rules and simple data governance processes. In case CDI or PIM solution is already in place, it is obviously beneficial to leverage existing infrastructure and know-how, at least on the organizational and conceptual level.
Lessons learned- critical success factors
Flexible data model
Master database data model has to be comprehensive - covering all relevant entities, attributes and relationships. It needs to be designed for general purposes, some of which are not know at the time when the model is built. To make it even more difficult, the designer should be able to foresee possible new business – originated developments and changes such as new products, customer segments etc. In this way, data modeling for master data management is somewhat similar (but only similar !) to data warehousing. (see exhibit 1 for more about MDM and DWH) .
Model flexibility and extensibility are some of the reasons why companies find it very difficult (if not impossible} to use standard ERP/CRM systems as master databases. Until the vendors improve their data models significantly, dedicated, independent master database will remain best practice.
5.2. Data quality and master record identification
Data cleansing and unification (i.e. identification and/or creation of the single master record) is usually underestimated, even in organization with extensive data integration experience. Typically in the analysis and design phase, project team does not spend enough time and energy on exact definition of cleansing and unification rules. As a result, project delivers “functionally correct” solution with the contents (integrated data) that business is not able to use.
Communication barriers between IT and business departments might be one of the reasons, sometimes even identifying the right person with the given domain knowledge is almost impossible.
In order to initiate communication and active participation of all sides, it is useful to perform a quick data profiling and sample, one time integration of key entities. External consultants or data quality tool vendor can be helpful in this phase, provided they posses localized and well developed methodology.
With data quality reports and samples of deduplicated records, you will be able to:
- look at the real quality of your data from the business perspective,
- get a preview of unification complexity
- start defining business rules for the data cleansing and unification in a very natural yet comprehensive way
- set up expected data quality and data availability KPIs
Right integration infrastructure
If you want to have a successful master data project, it is critical that you integration infrastructure delivers. High number and frequency of requests, deadly requirements for response times, not easily estimated load characteristics - these are all factors that make selection and design of the integration part very tricky. It is very wise to use different integration tactics for different needs – ELT, EAI, EII, even point-to-point interfaces are all valid. Trying to use only one type of integration as proposed by some technology vendors can lead to fatal performance and scalability problems.
Data Governance program
Now we have overcome all analytical, technical and organizational obstacles. We have a functional master database, integrating and providing high quality data to all places they are needed with the required response times. And it is in this point when the new challenge starts - we have to “defend” the status quo, constantly renew it, prevent deterioration by new projects, extend it to a new systems, educate new managers in the LOBs about the importance of data quality.
At the beginning of the project we needed to identify and attract business users to define integration and unification business rules. Now we need to include this and couple of similar functions into orgchart, together with business sponsor strong enough to enforce MDM ideas down the whole organization which will be resistant by nature.
Some of the key roles and competencies defined by Data Governance program are:
- Data owner – produces data, responsible for the data quality on the input
- Domain data steward– responsible for compliance with given DQ metrics in his domain, keeps business related know-how on data in his domain
- System data steward – responsible for technical metadata and consistency of the data in a given system
Business process owners, IT architecture and IT development should recognize data stewards as a “data authority”. Ideally, data steward should be authorized to propose and enforce change to any business process or system when it is needed to reach or keep given data quality SLA. For example, Customer entity data steward can propose and enforce bonus schema adjustment for the call center operators so that it reflects not only number of calls but also quality of entered data.