Assess your data quality

Data quality analysis / assessment

Data quality analysis is the process of assessing the quality of data using various metrics and techniques, aiming to identify all significant data quality issues, to estimate the impact on the business processes, and to implement corrective action. Whether it is about missing, inaccurate data, typos or format errors, inconsistencies, duplicates or whatever it may be, they have to be accurately identify and measure, to providing an understanding of the reason related to those issues.  The term Data quality assessment is also used for this process, usually when it is about a more formal and specific analysis, focusing on the gap vs specific rules and quality standards. Data quality improvement is planned to follow, involving implementing data management and governance strategies, such as data audits, data cleansing, data integration, and data reporting.

Is an assessment necessary?

In order to resolve a problem, it is necessary to accurately define it and understand its implications. When discussing about data quality issues, it is important to have a clear picture of different issues, not by simply describing the error, but also by  measuring the extend of it, identify the implications and understand the causes. Taking the example of duplicates, which can be a usual problem to CRMs and ERPs: It is not enough to say that we have some duplicate clients records. It makes much difference if those are 1% or 10% of the records, if they are limited to one specific category or if they are all over the client base.

A thorough analysis will be a solid ground, for any following project regarding data correction (cleaning, deduplication, format homogenization), data organizing and matching, as well as for any system reconfiguration to enforce or control quality rules. Deepening on the findings of the assessment, proper actions will be selected to resolve the problem. The gravity and the extend of quality issues may indicate how much resources have to be deployed in next steps. The nature and cause of quality issues will guide us to the right policies to avoid future quality deterioration. Other technical findings may reveal the best way to make corrections to the existing data, minimizing the risk for mistakes and service disruption.

A detailed assessment is also necessary for another reason. It needs to explore the technical aspects of any direct corrective actions directly to the data or to future configuration changes to prevent data quality problems. Every system has its own capabilities and restrictions in terms of data processing. Some systems may allow the execution of massive data corrections  directly in a system, in the production environment while others not. Some have advanced reporting capabilities while others not. Some systems have also very limited capabilities on exporting - merging data into the system, which can be a serious difficulty because in most cases data have to be exported, processed with a specific algorithm and re-imported to the system. Therefore. the assessment must also define what is technically possible, how it can be done and if are any implications or risks associated with it.

Executing a data quality analysis / assessment

 The whole process, its complexity and duration may vary depending on the following factors:

• The amount of data and complexity of data to be assessed
• The quality rules / requirements that have to be met to assure the desired level of quality.
• The technical details about how data are stored, accessed and processed as well as any technical limitations or risks
• The timeframe to complete the project
• Availably on internal resources to work on the project (in case of insourcing)
• Budget limitations (in case of outsourcing)

Despite the different approaches and the particularity of every case, most of such assessments  will include the following tasks:

1. Setting up the framework by understanding the quality standards required and how this affect the business processes
2. Understanding, the way of producing, processing and storing the specific data. (dataflows)
3. Make a preliminary analysis to confirm all basic hypothesis, evaluate the options about the steps ahead
4. Agreeing on the specific rules that have to be verified and the final outcome of the assessment
5. Make the core analysis, identifying all quality issues and quantifying them
6. Present results by identifying possible corrective actions

The depth and accuracy of the assessment will have an impact on corrective actions.