v6.1

This will be our last ever release (blog) post - sniff sniff - actually it's very good news, because we're moving to a fully automated and PUBLIC release system!

Here you can find the notes (and assets) from the 6.1 release: https://git.datacontroller.io/dc/dc/releases/tag/v6.1.0.

In addition we've added the ability to fully delete formats from catalogs, reduced the volume of audit data, and zapped a couple of smaller bugs too.

Source Available

Transparency is very important to us, as a team, and also for you - as you are trusting our software inside one of your most strategic platforms. Whilst we have had a 'source available' policy for several years now, it has been on a private invite / request basis. With version 6.1 it is now possible for ANY customer of SAS to freely evaluate our software without having to trust our build, or even to talk to us - you can create a release yourself, directly from the source repository, available here: https://git.datacontroller.io/dc/dc. All the steps can be viewed in the project release.yaml.

The source is available on a dual licence (the same as our OEM-licenced grid system).

We will continue to publish the SAS code in doxygen form at https://code.datacontroller.io. If you would like the ability to raise issues, or would like to submit a pull request, do get in touch via [email protected] and we will gladly create a user account for you.

Format Deletion

Unlike data in regular tables, formats must be modified and reloaded to catalogs in their entirety. Our previous approach for deletions was to export the format, remove the offending rows, and reload the catalog.

This is problematic when every row is marked deleted, as there is nothing to reload. The fix was made in the underlying SASjs Core macro (mp_loadformat.sas) - now, when all format records are removed, proc format is invoked with the delete statement against the relevant formats in the relevant catalog.

Reduced Audit Data

Previously when loading an audit table, we always included the entire row - including values that have not changed.

This resulted in some very large audit tables, especially for tables with hundreds of columns!

To limit data volumes, audit data is now excluded when MOVE_TYPE="M" (modified record), IS_PK=0 (not a primary key column) and IS_DIFF=0 (no change to the value).

We will continue to post the full record where MOVE_TYPE in ("A","D") (added/deleted) so that the table state can be recovered from a backup of the table, or reverted back from a modified table.

Fixes

Bugs we've blasted:

• Frontend rejection of excel uploads with duplicates on the primary key
• Missing mf_existds() macro issue when refreshing Table Lineage (SAS 9 EBI) or regenerating the Table Catalog (all versions).