[tower-deploy] Update known_hosts file when new machines are built

[maxkadel]

What maintenance needs to be done?

When a new server is built using a pre-existing FQDN name, we want to be able to deploy to it using Tower without manually updating the known_hosts file.

As an example of what the failure looks like, see this Ansible Tower deploy to bibdata-qa after the servers were re-built.

Level of urgency

• High
• Moderate -
• Low

Why is this maintenance needed?

This will be needed to make automatically provisioning new VMs sufficiently cheap to do it all the time, which will allow us to iterate on and improve our infrastructure much faster.

Acceptance criteria

• When a server is built with the same FQDN, I can deploy to it without ssh'ing into the tower deploy server and doing anything manual there

Implementation notes, if any

There is an Ansible known_hosts module, not sure yet if it matches our use case.

[kayiwa]

This is not an ansible problem (certainly not the process of ssh connection)

Rather this is the deploy user from the deploy box connecting to the remote VM to run capistrano.

[kayiwa]

Does capistrano have a ignore ssh

This can possibly also be flagged in capistrano

[acozine]

If we can get the deploy playbook to run in an EE on Tower itself, we will solve this problem, because the EE is always "fresh". So far I have not gotten the EE to work, though - see #4744.

[acozine]

Closed by #5215. That PR adds a task that updates the known_hosts file on the tower-deploy1 VM.