Скачать или смотреть BDB Pipeline 8.0 - Schema validator

Welcome to the BDB Platform! In this video, we'll introduce the new feature added to the data pipeline 8.0, which is Schema Validator Component.

In this video, we're going to talk about how to configure the Schema Validator component in the Ingestion pipeline.

 The Schema Validator component is used in the Ingestion pipeline to validate and enforce data schema rules.
 It allows users to create custom validation rules for their fields, including allowed data types, value ranges, and nullability.
 The component has two outputs: Schema Validated records and Bad Record Event.
 The component supports uploading and viewing schema files.

So, let's get started! First, access the Schema Validator component from the Ingestion Section and drag it to the Pipeline workflow canvas. Then, connect it with the required components.
Next, click on the Schema Validator component to access its configuration tabs. You'll see the Basic Information tab first, where you can select the invocation type from the given choices. Under the Meta Information tab, you can specify the name of the uploaded Schema file and choose to upload it. You can also view the uploaded schema file or remove it if necessary.

There are two mode choices provided under the Meta Information of the Schema Validator. The first is the Strict mode, which aims to prevent unexpected behaviors or silently ignored mistakes in user schemas. It does not change any validation results compared to the specification, but it makes some schemas invalid and throws an exception or logs a warning (with the strict: "log" option) if any restriction is violated and sends them to the bad record event.

The second choice is Allow Schema Drift, which allows slight changes in the data schema. Schema drift is the case where the used data sources often change metadata. Fields, columns, types, etc. can be added, removed, or changed on the fly.

Finally, any records that are rejected by the Schema Validator will automatically go to the Bad Records Event.

That's it! You now know how to configure the Schema Validator component in your Ingestion pipeline. Thanks for watching!