Overview of Data Quality Policy

Data quality is a measure of how healthy the data is inside a data source from the perspective of a consumer or a business.

In Torch, data quality assurance is done with the following policies Data Asset Quality and Reconciliation Rule.

Data Asset Quality Policy: Data Asset Quality Policy is a quality measure for a single asset. It checks the various types of properties like an asset's datatype, null values, and regex match.

Reconciliation Policy: Reconciliation Policy is a quality measure between two similar type of assets. This rule does a comparative analysis of a source asset and sink asset by matching various properties like profile match, equality match, and so on.

A policy can contain several different rules - all the rules are checked and the policy passes only if the result of all the rules are positive. You can run a policy manually or at predefined intervals. Once a policy is run, an execution is created in the system. The execution contains information about whether the policy has passed or failed during its execution period.

You can configure notifications by email, Slack, or any other webhook URL.
With the result of multiple executions over a period of time, you will get an understanding of the quality of the data. If an execution fails more than once, then you can find the exact location of where the failure had occurred in the pipeline. This makes it easier for you to rectify the issue in your system.

Viewing Policies

To view the list of asset policies, follow the procedure given below:

  1. Click View Asset Policies from the Actions panel in the Discover page. The Asset Policies page is displayed.

  2. Enter the name of policy in the search bar that is provided in the Policy panel.

  3. From the search results provided, click the vertical ellipsis icon for the required policy and click View. The View Policy panel is displayed at the right.

Editing Policies

To edit the policy, do the following:

  1. Click the vertical ellipsis icon.

  2. Click Edit from the drop-down list. The Edit Policy panel is displayed at the right.

Deleting Policies

To delete a policy, do the following:

  1. Click the vertical ellipsis icon.
  2. Click Delete from the drop-down list. A confirmation pop-up dialog box is displayed on your screen.
  3. Click Ok.

Execute Policies

To execute a policy, do the following:

  1. Click the vertical ellipsis icon.
  2. Click Execute Policy from the drop-down list. A confirmation pop-up dialog box is displayed on your screen.
  3. To stop the execution, click the Stop Execution button from the Execution Result pop-up wizard. Else, wait for the execution to finish running.