Cloud-config in Terraform does not work, nothing seems to happen and it fails silently

[obi]

I'm currently provisioning a machine with Cloud Config in user_data (I've also tried user_data_base64). When I run my command in Terraform I have,

resource "aws_instance" "web" {
    user_data_base64 = filebase64("${path.module}/scripts/web-init-script.yml")

However, nothing happens. It silently fails. What's the problem? The contents of web-init.script.yml are,

❯ cat scripts/db-init-script.sh
#cloud-config
package_update: true
package_upgrade: true
fqdn: db.acme.com

prefer_fqdn_over_hostname: true

hostname: db
packages:

- podman

[rielyn]

Failure to Merge - YAML whitespace

This turned out to be because of whitespace. You can confirm this by sshing to the machine and running

sudo grep WARNING /var/log/cloud-init.log

Which should return this,

util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: "while scanning for the next token
util.py[WARNING]: Failed at merging in cloud config part from part-001

Basically what happens is this,

1. You upload the user_data to the providers instance's metadata.
2. The client you're provisioning launches cloud-init.
3. That downloads the user_data from the instance's metadata at http://169.254.169.254/
4. It sees the #cloud-config on line one of your user_data
5. You merge your own cloud_config with the downloaded user_data
6. The merge-algorithm doesn't normalize for whitespace, so the yaml is invalid.
7. The merge-algorithm returns a blank cloud-config.

You can verify this by trying to re-run cloud-init init which will produce something like this,

2022-05-16 04:50:04,351 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: "while scanning for the next token
found character '\t' that cannot start any token
  in "", line 11, column 1:
        - podman
    ^"
2022-05-16 04:50:04,351 - util.py[WARNING]: Failed at merging in cloud config part from part-001

Full Log

__init__.py[DEBUG]: {'MIME-Version': '1.0', 'Content-Type': 'text/cloud-config', 'Content-Disposition': 'attachment; filename="part-001"'}
__init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (text/cloud-config, part-001, 3) with frequency once-per-instance
util.py[DEBUG]: Attempting to load yaml from string of length 252 with allowed root types (,)
util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: "while scanning for the next token
found character '\t' that cannot start any token
  in "", line 11, column 1:
        - podman
    ^"
util.py[WARNING]: Failed at merging in cloud config part from part-001
util.py[DEBUG]: Failed at merging in cloud config part from part-001
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 140, in handle_part
    self._merge_part(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 116, in _merge_part
    (payload_yaml, my_mergers) = self._extract_mergers(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 95, in _extract_mergers
    mergers_yaml = mergers.dict_extract_mergers(payload_yaml)
  File "/usr/lib/python3/dist-packages/cloudinit/mergers/__init__.py", line 79, in dict_extract_mergers
    raw_mergers = config.pop('merge_how', None)
AttributeError: 'NoneType' object has no attribute 'pop'
__init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (__end__, None, 3) with frequency once-per-instance
util.py[DEBUG]: Writing to /var/lib/cloud/instances/i-0804054440b55ca17/cloud-config.txt - wb: [600] 26 bytes