What is a good strategy to prevent Ansible playbook runs against the wrong hosts?
It is too easy to run playbooks on the wrong hosts in Ansible
I know the best practice would be to use --limit to make sure you can not select the wrong host. I do not trust --limit to ensure Ansible runs playbooks only on the intended hosts.
Is it a crazy idea to use firewalld to disable communication to all the systems you do not want to update? Is there a more logical way to accomplish the same thing?
Q: "Disable communication to all the systems you do not want to update."
A: The only safe way is to physically isolate such https://docs.ansible.com/ansible/latest/tips_tricks/ansible_tips_tricks.html#separate-production-and-staging-inventory . A firewall might be an option. But, it's not feasible if you want to keep ssh or other https://docs.ansible.com/ansible/latest/inventory_guide/connection_details.html#connection-methods-and-details that Ansible might use.
On the configuration level, there are many options.
1) Special user
Create a special user for this purpose on the systems you want to update. For example user ansible_update. Depending on your https://docs.ansible.com/ansible/latest/tips_tricks/ansible_tips_tricks.html#separate-production-and-staging-inventory configure the https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-remote-user to ansible_update. This way you make sure no other systems can be connected by mistake.
In addition to this, you control who can connect to these systems by putting their public keys into the authorized_keys of the user ansible_update at the remote hosts.
To enforce this you have to:
- monitor users at the remote hosts and their authorized_keys
- scan your project (playbooks, roles, inventory, configuration) for remote users other than ansible_update
2) Bastion host
A better option is creating a https://en.wikipedia.org/wiki/Bastion_host and making it the only source of the configuration for the environment.
The most flexible option is configuring the remote hosts to https://docs.ansible.com/ansible/latest/cli/ansible-pull.html their updates on their own.
Conclusion: Create special users, create a bastion host, and make the remote hosts pull the updates from it.