Automatic Data Encryption- Structure

MASTER KEY

The Master Key is simply a string or phrase that the database encrypts and encodes then stores in the product.info (UniData) or .uvconfig (UniVerse) binary files. In this state, it is protected from reading or from the physical theft of the file.

It is used in all encryption/decryption processes and to protect certain administrative functions (see ‘encman’ in the documentation, page 152). It is system-wide, meaning only one can exist per install of UniData or UniVerse.

When it is set or changed, the database MUST be restarted before taking any further ADE actions. This is very important. If this step is missed, it will look like the ADE system is working…until your next restart where all encrypted files will suddenly become inaccessible. Doing this will result in an error similar to “unable to read from encinfo” or “invalid keystore”.

The key is set/changed using confcmd -m (UniData) or uvregen -m (UniVerse).

It can be protected by one or two passwords. This is helpful if you wish to provide people with administrative rights to ADE without letting them know the actual key. You can also split responsibility between two people (or two teams of people) by giving them each one of the passwords, this means no one person has control of the ADE system or access to your encrypted data.

Knowing where the master key resides, and its importance tells us where to look if we hit a master key-related problem such as…

Total loss of access to all encrypted files including the keystore.
Error: “Data was not previously encrypted by U2”
Error: “Master key invalid”
Error: “Master key has not been setup”
Error: “Failed to load encinfo for file &KEYSTORE&”

Mostly these issues can be resolved by restoring the product.info (UniData) or .uvconfig (UniVerse) files from backup.

You can also take a ‘blank’ or ‘default’ copy of the file and reapply the master key (assuming you know what it is or can retrieve it from a safe storage place). Some later releases will have a product.info.ship or .uvconfig.default type file. Simply put this in place of the ‘live’ file and apply the master key (don’t forget to restart UDT/UV!). Note, you will also need to redo your license configuration and authorize the database as this data is also stored in these files.

Screen 2.1: UniData. If the key was damaged, these are the only steps. Provided the key you put back after these steps matches the original, the data will now be accessible. If you ‘forget’ your master key and need to set it to something new, you MUST decrypt all your data first. NOTE: After replacing product.info you will need to relicense the product with serial number, user count, expiry date etc.

Screen 2.2: UniVerse. If the key was damaged, these are the only steps. Provided the key you put back after these steps matches the original, the data will now be accessible. If you ‘forget’ your master key and need to set it to something new, you MUST decrypt all your data first. NOTE: After replacing .uvconfig you will need to relicense the product with serial number, user count, expiry date etc.

One last note about the master key. It belongs to and is used by the system, nobody but the system needs to know it for ADE to work. This doesn’t mean you can set it and forget it; you’ll need it to migrate your data, administer ADE and of course should something happen to the file it resides in, you’ll need to know it to put it back. But, for ADE to work, it just needs to be in its proper place. So, if you wake up one day and can’t remember the master key, don’t panic. You are now at risk, but if you decrypt all your files, your data is safe. You can then reset the master key (see above regarding .ship and .default files), delete and recreate your keystore and start again.

METADATA STORE

This is a hashed file stored in…

• UDTHOME/sys/_ENCINFO_
• UVHOME/&ENCINFO&

The file has the same characteristics as any hashed file such as a header, modulo, block size, groups and a dictionary. However, it cannot be accessed at all via conventional tools because it is encrypted. While its size should remain small, it is still a hashed file and therefore should be checked for proper sizing and included in regular backup processes.

The metadata store holds a record for every other encrypted file on the system, including the keystore. This record holds everything ADE related regarding the encrypted file: the file’s name, each encrypted field (including any encrypted indexes), the algorithm used and of course the key(s) used to encrypt it.

The @ID of each record in the metadata store is numerical. The encrypted file the record relates to has a hexadecimal representation of this @ID in its header. This is what we call a hard link, if this data is out of sync, the file will be inaccessible.

This record is referenced when a file is opened using the OPEN statement. The process checks if the file is encrypted, then takes the value in the header, converts it to decimal and opens the corresponding record in the metadata store. This same process is done when running the LIST.ENCRYPTION.FILE command against an encrypted file. The information you see is from this record.

If you copy an encrypted file at the operating system level, you will have two files pointing at the same metadata record. These files will work fine for a while but as soon as the encryption setup is changed, one file will be out of sync and no longer function correctly. This could be weeks or months after the file was copied. Do NOT copy encrypted files at the operating system level.

Screen 3.1: UniData’s ‘fileview’ utility does the work for you. The raw data is stored in hex. After converting to decimal, you’ll have the @ID in the _ENCINFO_ file.

Screen 3.2: UniVerse’s ‘filepeek’ tool shows you the raw data but it is a simple step to convert it yourself (you can of course use U2 Basic, TCL, or any number of conversion tools). Take the value from the header (hex) and convert it using whatever tool you like (dec), the result will be the @ID in the &ENCINFO& file.

 

KEYSTORE

The keystore resides in a hashed, encrypted file in…

• UDTHOME/sys/_KEYSTORE_
• UVHOME/&KEYSTORE&

These files are hashed files like any other, but they are encrypted with internal credentials so can’t be read or written to with conventional tools. The keystore holds a record for each key (not the master key) and each wallet on the system. Some of the data in each record is in clear text and some is encrypted.

As a regular hashed file, the keystore has the same characteristics as other database files. It has a header, groups, a modulo, block size and a dictionary. While it will likely remain reasonably small it should be checked for proper sizing and of course be made part of your backup.

The keystore is ‘linked’ to the master key. If you change the master key, the keystore is no longer accessible and neither are your keys – which means you cannot access your encrypted data.

The file is also ‘tagged’ to the system it was created on. This provides an additional layer of protection and stops someone from physically transferring the file to their own system. The tagging process happens at creation, but if you need to migrate it, your keystore ‘encman -retag’ will help tag it to a new system. See ‘encman’ in the documentation for more information; the retag process is protected by the master key.

There is what we call a soft link between the keystore and the encrypted files on the system. This means these encrypted files and fields are listed against each key’s record in the keystore file, but it is no more than text in a field. If this data gets out of sync, the key will still work. This can only happen very rarely, and usually only if the keystore or files are moved or accessed in an inappropriate way without the proper tools.

You can LIST and SORT the keystore, in fact the LIST.ENCRYPTION.KEY command is just a LIST command with predefined fields.

Screen 4.1: LIST.ENCRYPTION.KEY output from UniVerse. UniData is the same, but the keystore file is _KEYSTORE_ instead.

 

ENCRYPTION KEYS

These keys are the database entities we use to physically encrypt database files. Remember ‘Automatic Data Encryption’ is a ‘Data at Rest’ encryption style meaning the physical data on disk is encrypted. We need to perform a decryption operation if we want to read this data. ADE performs this for us automatically provided we have:

• configured it to do so
• privilege on the appropriate key
• the key activated in our session.

Encryption keys have the following characteristics…

• Key name
• Key password (optional)
• Owner/creator
• Time/date of last changeGrantees
• Encrypted Files and Fields

The key name is simply a string used to access the key in Basic and at ECL/TCL. This string is not the key passed to the encryption engine, although it is part of the encryption string that is.

Keys can be password protected. This is of course recommended for added security. However, this means the key will need to be positively activated by the application. If a key has no password, it is activated automatically when the session begins (provided the user is on the list of Grantees).

The owner/creator of the key is automatically granted privilege and is the only one able to grant/revoke privilege. It is recommended a role-based user be created to ‘own’ the keys.

Key passwords can expire. This is managed by the ‘encman -passpolicy’ command. Because of this, key passwords can of course be changed. This is out of scope of this article.

Grantees can be a user or group or PUBLIC (everyone).

Encrypted files and fields, as mentioned, is merely text in a field. There is no ‘hard link’ to the files themselves from this key record in the keystore,meaning that if this information gets out of sync, the keys will still work.

WALLETS

Encryption wallets, while they do have a record in the keystore file, are not physical encryption entities. Think of them more like a pointer to several keys. Thereis a way to activate multiple keys in one command. Wallets must have a password.

Wallets are quite powerful in terms of designing an ADE-aware application. You can group several keys inside a wallet for each, for example, department within the business.

A great feature of wallets is, once a key is added, you can use the wallet without ever referencing the key’s password. So, wallets are a great way to give your users access to keys without them ever knowing key passwords.

Wallets live alongside keys in the keystore. You also use the same commands to manage them for the most part. You can tell a wallet record from a key record in field 7 (TYPE) of the record in the keystore file – 1 for keys, 2 for wallets.

Screen 5.1: Wallets and keys are stored in the same file. They are distinguished by the TYPE field

 

ENCRYPTED FILES

These are your existing data files you have decided to encrypt. The encryption process will modify the 0xEC location of the file’s header. It will place two values…

• The encryption version
• The file’s ID in the metadata store

There are currently only two encryption versions for files…

• 0 – UniVerse 10.2
• 1 – UniData and UniVerse 10.3.0+

The difference between versions is the strength of the encryption. At version 1 we started to encode the encrypted data as well as encrypt it. So, version 1 files can’t be read on a version 0 database and visa versa. There is a tool in UniVerse to convert the files and the keystore when upgrading from 10.2.

Next to the encryption version is the file’s @ID in the metadata store. It will generally be a four to six-digit number. It will be converted to hexadecimal before being stored in the file’s header. So, to determine which record in _ENCINFO_/&ENCINFO& pertains to your data file, run the header value through a hex converter (hint: Windows Calculator can do this) to get the decimal value.

If this location is null or all 0’s the database will skip the encryption portion of its code and treat the file like an unencrypted file. And conversely, an unencrypted file with something in that location will force the database to treat it as encrypted. You’ll see errors like “Unable to find record in encinfo” or “file was not previously encrypted by U2”. This is almost always due to corruption of the file header.

Encrypted data generally takes up more space than clear-text data. So, encrypting a file will cause it to grow. Also, because your data is now physically larger, you may need to change your file sizing parameters to regain some performance. There are no special instructions for this, when doing your file sizing do it as you always would, just use the new, larger values for length/size.

You can encrypt individual fields on a file or the entire record. Generally speaking, ‘whole record’ encrypted files perform faster as there is only one ‘pass’ through the encryption engine. They tend to take up less additional space too (depending on the algorithm) due to the ‘padding’ some algorithms require to reach a multiple of a certain number (again, depending on the algorithm).

As of UniData 8.1 and UniVerse 11.1 you can encrypt @IDs and indexes. These are treated the same as any other encrypted field of a file, they just add a little extra complexity.

SUMMARY

ADE is not a licensed product, and you don’t ‘turn it on’. You set up a master key, create a keystore, create some encryption keys and then finally encrypt your files. It is only at this last step you are really ‘using’ ADE.

OpenSSL is an industry standard encryption tool used to perform the actual encryption/decryption steps for ADE. It is installed when installing UniData and UniVerse.

The master key (product.info/.uvconfig) is involved in all encryption operations and cannot be changed without decrypting the data. It can have one or two passwords which can be changed without decrypting the data.

The metadata store is a hashed file that contains a record for each encrypted file on the system. It requires backup and proper sizing.

The keystore (_KEYSTORE_/&KEYSTORE&) is a hashed file that contains keys and wallets. It is encrypted and has a record in the metadata store. It requires backup and proper sizing.

Encryption keys are stored in the keystore. They can be password protected. They must be granted to users/groups before being used by the userbase. They, along with an algorithm, are used to encrypt files.

Wallets are a pointer to multiple keys. They are mostly a quality-of-life feature allowing you to activate multiple keys with a single command. They can be used to ‘hide’ encryption key passwords from your users.

Encrypted files are your hashed files once they’re encrypted. They have a record in the metadata store which can be found by converting the value stored in 0xEC in the file’s header.

Screen 6.1: Overall Automatic Data Encryption Structural Diagram