The Data Quality Challenge

According to a study published by The Data Warehousing Institute (TDWI) entitled Taking Data Quality to the Enterprise through Data Governance, some issues are primarily technical in nature, such as the extra time required for reconciling data (85%) or delays in deploying new systems (52%). Other problems are closer to business issues, such as customer dissatisfaction (69%), compliance problems (39%) and revenue loss (35%). Poor-quality data can also cause problems with costs (67%) and credibility (77%).

Another recurring statement that we come across frequently is : Only less than 5% of our data is “bad”.

If you consider a fact table with 10 million rows, then about 500K rows are bad – which is a huge problem for analytics, operations or anything else you may want to do with the data!!

Even if you choose to ignore all the statistics and studies, it comes down to just a few question –

1. How do we get the information that we need and where do we get it from?
2. Can we trust this information?
3. What does it mean and how can we get this information in the format we need?

Data Quality Definition: Commonly known, data is deemed to be of high quality if it correctly represents the real-world construct to which they refer.

The implementation of Data Quality processes and procedures can dramatically affect the quality and therefore usability of the information in your Data Warehouse.

The Goal: The reason for building a Data Warehouse is to provide consolidating data from multiple sources, both internal and external, into an enterprise solution. This consolidation involves ensuring data consistency, integrity and quality. The data warehouse is the medium to provide enriched analytical data to empower better decision making. The goal is not to copy the same old bad information from multiple locations and provide little or no benefit to the enterprise.

We must consider two key factors to achieve this goal.

• Data Profiling identifies the problems. It provides snap-shots of a company‘s data quality and measures the evolution of data quality over time.

• Data Cleansing corrects, or ―cleanses incomplete or inconsistent data by crosschecking against other databases and reference data. It also enriches data by providing value-add information that actually improves the quality and usefulness of existing data.

I will post more on the two factors soon...