Maintaining data

Build for consistent data collection

Build data collection processes at the outset. Your data needs to be fit for purpose and reuse by other systems and agencies.

Automate your data quality processes where possible. This will ensure the quality of incoming data is consistently controlled and managed whenever it is received. For example:

• only allowing addresses to be entered in the same way. For example, Thomas Street, instead of Thomas St or Thomas Str
• use a common unique identifier when you're collecting data after the same person or thing.

If your data collection needs change, ensure you review and update your storage and security controls as necessary.

Implement data quality processes

Determine what processes you can implement in the design and build of your product or service to ensure

• the data you collect, use and manage is accurate
• it can be used for the purpose you collected it.

Talk to your data and security experts about any agency specific policies and guidance you can follow to make sure you maintain your data quality.

Data cleansing

This is the process of removing errors from your data to ensure it is correct. Use automated error checking methods to help speed up this process.

Conduct regular systems audits to demonstrate your data management controls are operating correctly and that you maintain the integrity of the stored system information.

Data validation testing

You can do data validation testing to check the accuracy and quality of the source data against your business requirements. Validate data before you use and process it and ideally automate it, so errors are corrected at source.

Regression testing

Regression testing can help you confirm that a recent program or code change has not impacted existing functionalities of your system.

Automate your regression test and data comparisons to make sure good data is maintained consistently. Follow your agency specific policies and guidance.

Describe your data

Metadata is information that helps people to find, understand, authenticate, trust, and use and manage information.

Follow your agency's specific policies when describing your data. If none exist, the NSW State Archives and Records has guidance on metadata requirements to ensure that records are authentic and reliable, have integrity, and are usable.

The data quality reporting tool

Data.NSW provides a data quality reporting tool to help you describe your data. This will allow users of your data know how the data can be used.