Versatile Data Kit 0.8

[antoniivanov]

Summary

Major features include:

New plugin: VDK Audit

This plugin provides the ability to audit and potentially limit user operations. These operations can be deep within the Python runtime or standard libraries, such as dynamic code compilation, module imports, or OS command invocations.

If we want to forbid some os.* operations we can do it like this:

export AUDIT_HOOK_ENABLED=true
export AUDIT_HOOK_FORBIDDEN_EVENTS_LIST='os.removexattr;os.rename;os.rmdir;os.scandir'
export AUDIT_HOOK_EXIT_ON_FORBIDDEN_EVENT=true
vdk run <job-name>

See more details in the vdk-audit plugin page

Any version of python in VDK Control Service

Deployed jobs by Control Service can now use any version of Python and not just 3.7 automatically.

Insert only impala load template

This template can be used to load raw data from Data Lake to target Table in Data Warehouse. In summary, it appends all records from the source table to the target table. Similar to all other SQL modeling templates there is also schema validation, table refresh and statistics are computed when necessary.

See more details in the template documentation page

Package versions

See installation instructions here.
The versions of VDK components released under VDK 0.7 are:

Main components

control-service 1.5.671965442
vdk-core==0.3.662978536

Plugins

vdk-ingest-http==0.2.670842377
vdk-impala==0.4.672320306

What's Changed

• control-service: CVE fix - upgrade commons-text by @tozka in control-service: CVE fix - upgrade commons-text #1255
• control-service: Dynamic python site-packages directory detection by @mivanov1988 in control-service: Dynamic python site-packages directory detection #1247
• control-service: fix cicd deployment by @tozka in control-service: fix cicd deployment #1226
• control-service: fix integration tests by @tozka in control-service: fix integration tests #1211
• control-service: fix race condition in test by @murphp15 in vdk-control-service: fix race condition in test #1227
• control-service: refactor job cancellation method due to 404 errors by @mrMoZ1 in control-service: refactor job cancellation method due to 404 errors #1114
• control-service: remove executables from secure job builder by @mivanov1988 in control-service: remove executables from secure job builder #1202
• control-plane: better error logging for transient error in tests by @murphp15 in vdk-pipeline-control-plane: better error logging for transient error in tests #1222
• control-service: improve docs and local runability of integration tests by @murphp15 in vdk-pipeline-control-service: improve docs and local runability of integration tests #1217
• control-service: upgrade java client k8s version by @murphp15 in vdk-pipeline-control-service: upgrade java client k8s version #1216
• vdk-core: errors occurred and the state (handled or not) context missing by @ivakoleva in vdk-core: errors occurred and the state (handled or not) context missing #1182
• vdk-core: errors occurred and the state (handled or not) context missing by @tozka in vdk-core: errors occurred and the state (handled or not) context missing #1212
• vdk-core: platform error no longer logged when skipping execution steps by @mrMoZ1 in vdk-core: platform error no longer logged when skipping execution steps #1223
• vdk-impala: Fix parsing while analysing profile for lineage information by @kostoww in vdk-impala: Fix parsing while analysing profile for lineage information #1206
• vdk-impala: Refactor query classifier for data lineage by @kostoww in vdk-impala: Refactor query classifier for data lineage #1239
• vdk-impala: improve explanation in readme by @tozka in vdk-impala: improve explanation in readme #1248
• vdk-impala: stop using errors.get_exception_message by @tozka in vdk-impala: stop using errors.get_exception_message #1224
• vdk-impala: update documentation with link by @tozka in vdk-impala: update documentation with link #1237
• vdk-ingest-http: Adopt simplejson in place of json by @doks5 in [vdk-plugins] vdk-ingest-http: Adopt simplejson in place of json #1229
• vdk-ingest-http: Move data conversion above size calc by @doks5 in [vdk-plugins] vdk-ingest-http: Move data conversion above size calc #1245
• vdk-ingest-http: fix default value for backoff factor, add retry test by @dakodakov in vdk-ingest-http: fix default value for backoff factor, add retry test #1218
• vdk-plugins: fix broken link by @tozka in vdk-plugins: fix broken link #1204
• vdk-plugins: introduced vdk-audit plugin by @mivanov1988 in vdk-plugins: introduced vdk-audit plugin #1221
• vdk-plugins: run tests on release of vdk-core by @tozka in vdk-plugins: run tests on release of vdk-core #1210
• vdk-plugins: set dind tempalte job for default build of plugins by @tozka in vdk-plugins: set dind tempalte job for default build of plugins #1225
• versatile-data-kit: required approving reviewers update by @ivakoleva in versatile-data-kit: required approving reviewers update #1220
• versatile-data-kit: update contributing.md by @tozka in vdk: update contributing.md #1214

New Contributors

• @BoBaeva made their first contribution in Insert only impala load template #1233

Full Changelog: v0.7...v0.8

---

This discussion was created from the release Versatile Data Kit 0.8.