roachprod: implement --dry-run and --verbose

[srosenberg]

Debugging roachprod in a remote (user) environment can be challenging.
On several occassions, simply printing all executed (cloud) commands
was sufficient to reproduce and root cause an issue.

This change implements two mutually exclusive
modes, namely dry-run and verbose. The latter
merely logs every command before it's executed.
The former suppresses execution of any command
which can modify the (cloud) infrastructure;
i.e., dry-run executes read-only commands; it logs
every command, executed or not. Also note that
dry-run leaves temporary files like VM startup
scripts on disk, so that they can be readily examined.

Thus, dry-run is primarily for dumping all
commands without the risk of modifying anything;
verbose is primarily for live debugging.

Release note: None
Epic: none
Fixes: #99091

[blathers-crl]

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[srosenberg]

SELECT_PROBABILITY=0.25 runs,

• GCE

• AWS

[srosenberg]

I'll do a bit more smoke testing, but it's ready for review now. Apologies for a fairly large PR. Most of the changes are fairly shallow and hopefully quick to review; e.g., we need to thread the logger through every func. which may invoke a command. The rest is refactoring, by moving command invocation logic into its own package (see cli.go), and consolidating dry-run and verbose logic.

[golgeek]

This looks great! Both options will be very helpful, and the cli package is the cherry on the cake!

[DarrylWong]

Couple of commands that don't fully work (e.g. destroy errors out trying to unmarshal DNS records, create errors out setting up SSH), but the approach SGTM and I think it's fine since it's just an "experimental" feature.

[srosenberg]

Couple of commands that don't fully work (e.g. destroy errors out trying to unmarshal DNS records, create errors out setting up SSH), but the approach SGTM and I think it's fine since it's just an "experimental" feature.

Thanks, I'll have a look at those. Not everything is going to work since some commands may require changes (or side-effects), but methinks destroy should be doable.

[srosenberg]

Couple of commands that don't fully work (e.g. destroy errors out trying to unmarshal DNS records, create errors out setting up SSH), but the approach SGTM and I think it's fine since it's just an "experimental" feature.

Thanks, I'll have a look at those. Not everything is going to work since some commands may require changes (or side-effects), but methinks destroy should be doable.

So, create does print the commands which would have been executed; it fails upon trying to List the cluster; that's as far as --dry-run can go. As for destroy, it was missing cli.AlwaysExecute() in listSRVRecords; works now without an error.