YAML confg migration tool

Nowadays YAML configs are everywhere. And often such configs must be updated over time. In many cases, it is possible to simply override the entire config file, but what if not?

Here is the way for updating YAML configs preserving comments and all current values.

Preparing the tool

The tool could be used in different ways, but I'll show the most universal command line usage.

Download yaml-config-updater-cli-1.2.0-all.jar from maven central.

wget https://repo1.maven.org/maven2/ru/vyarus/yaml-config-updater-cli/1.2.0/yaml-config-updater-cli-1.2.0-all.jar

Java 8 or above must be installed in the target environment.

Adding missed properties

Suppose your config is:

section:
    # Important description
    # Value was manually updated (must be preserved)
    prop1: 12

In the new version, new property was added:

section:
    # Important description
    prop1:  # value must be put manually after installation

    # New description
    prop2: 10

Performing migration:

java -jar yaml-config-updater-cli-1.2.0-all.jar config.yml update.yml

Output report:

Updating configuration: /home/user/test/config.yml

Configuration: /home/user/test/config.yml (105 bytes, 5 lines)
Updated from source of 137 bytes, 6 lines
Resulted in 93 bytes, 7 lines

    Added from new file:
        section/prop2                            6  | prop2: 10

    Backup created: config.yml.20210911140958

As a result, the new property would be added to the current config:

section:
    # Important description
    prop1: 12

    # New description
    prop2: 10

Note that the prop1 comment was also changed, as comment from updating file may contain important information updates. Only in-line comments (after value) survive merging.

If you don't trust the report you can easily build diff with the created backup file (original config):

diff config.yml.20210911140958 config.yml
5a6,8
>     # New description
>     prop2: 10
>

The trust

Good question: why should you trust the tool?
The tool is very young and most likely there are still bugs and uncovered cases.

And still, I can guarantee that the resulted file would be correct:

• Internally, time-prooved snakeyaml parser used for all files validation. This way "comments" parser can be sure to always work with the correct YAML.
• But that's not all. Parsed trees from comments and snakeyaml parsers are compared to eagerly detect incorrect parsing (self-validation)
• Merge result is read by snakeyaml to make sure the file is correct.
• Merged file values are carefully checked: all old values must be preserved and all new values added (using snakyaml trees to operate exact values).

So you can be sure that the resulted config would be correct. If anything goes wrong, the configuration file would not be touched at all.

Removing values

Adding new properties is not the only case: some properties might be removed in the actual configuration.

By default, the tool would not remove properties, so the merged file would still contain them. In order to remove a property, you must explicitly specify it.

For example, suppose in config:

prop1: 1

sub:
    foo: 2
    bar: 3

prop1 and foo are no longer needed.

java -jar yaml-config-updater-cli-1.2.0-all.jar config.yml update.yml -d prop1 sub.foo

Updating configuration: /home/user/test/config.yml

Configuration: /home/user/test/config.yml (36 bytes, 6 lines)
Updated from source of 11 bytes, 2 lines
Resulted in 18 bytes, 4 lines

    Removed from old file:
        prop1                                    1  | prop1: 1
        sub/foo                                  4  | foo: 2

    Backup created: config.yml.20210911144139

This may be also used for replacing values: removed property would be replaced with a property from updating config with the default value.

You can remove entire sub-trees this way.

Config-template

Another common need is configuration "adoption" for the target environment. In this case, updating configuration become "a template" with placeholders to replace by environment-specific values:

prop1: #{one}
prop2: #{two}
prop3: #{three}

java -jar yaml-config-updater-cli-1.2.0-all.jar config.yml update.yml -e one=1 two=2

Also, this time, updating configuration does not exist yet (first installation case).

Updating configuration: /home/user/test/config.yml 

Not exising configuration: /home/user/test/config.yml 
Updated from source of 33 bytes, 3 lines
Resulted in 34 bytes, 3 lines

    Applied variables:
        two                       = 2
        one                       = 1

    New configuration copied as-is

The resulted config would be:

prop1: 1
prop2: 2
prop3: #{three}

Note that the third variable was not processed (as the value was not specified), but it is not a problem as any YAML parser would treat this placeholder as an in-line comment.

Environment variables might also be specified in a property file (and multiple files could be specified). By default, system environment variables are all provided (use -i option to see the list of all available variables during processing).

Links

This was just a brief introduction. More info could be found in github repository:

• Project root
• API module
• CLI tool
• Dropwizard module