palm new-machine && palm-clone

[ethanknox-palmetto]

Context
We are actually 2 steps away from palm being a complete instant-dev solution. think about it....

You start at Bluth's Bananastand co on a Monday. The onboarding box arrives with your brand new Mac/PC/Linux laptop, and a single post-it note on the box:

    Welcome to the team Pat! to get started (from the command line): 
    1. pip install palmcli 
    2. palm new-machine
    3. palm clone git@github:bluth-bananastand/banangrabber.git

    run `palm --help` if you have any questions.
    Please try to have jira issue NANA-123 done before lunch, it's a 1 point naming fix ;) Excited to have you aboard! 

Is your feature request related to a problem? Please describe.
There are still a few manual steps outside palm. this eliminates them.

Describe the solution you'd like

1. Open up my brand new Mac, PC or Linux laptop.
2. Power it on
3. open the CLI, run palm new-machine
4. Now the magic:
   • palm checks the OS, checks for docker, and installs the correct docker (or at least opens the browser dialog to do so)
   • palm prompts you to select a git repo provider (github & bitbucket are good intial options)
   • palm (generates if needed and) installs your ssh key into your gh/bb account

palm announces that you are ready to begin work! now to get started:

1. cd into the directory you want your repos in
2. use the palm clone command to add the repo palm clone [email protected]:your-company/some-repo-with-palm.git
   • palm clones the repo like normal git
   • palm looks for a .env file and parses it for expected environment variables (i.e. BANANA=${BANANA}
   • for each found environment variable, it checks to see if that is already exported into your path.
     • if it is, palm announces it ("found environment variable BANANA, skipping")
     • at the end of the cycle, it prompts you with a list of envars it needs ("repo banana_grabber requires the following environment variables: BANANA_USERNAME, BANANA_API_KEY, BANANA_HOST. proceed? (y/n):")
     • palm then sets each variable for the os (so .zshrc or .bashrc for unix, win sys envs for windows)
     • bonus we could make it super easy to set a custom message with the envars message in the palm file, like "if you do not have a BANANA_USERNAME yet, contact core systems at [email protected] and request one."

Describe alternatives you've considered
Manual documentation. which is fine. but not magic. This is amazing magic.

Additional context
This should be broken into implementation slices; each part has value on its own (ie palm new-machine and palm-clone)

Is there an existing feature request for this?

• I have searched the existing issues

[jakeberesford-palmetto]

Neat idea, I especially like the ideas around palm new machine. However, there are some things we need to consider:

• Running palm outside of an initialized git repo is not currently supported. Trying to do anything with Palm outside of a repo will raise an error due to checking for branch validation.
• .env files should usually not be comitted to VCS, our pattern of using interpolation works in some cases, but not all. We should be careful about baking this pattern into palm as it could lead users down a dangerous path (having .env files committed to source control). Without .env management, palm clone just feels like an unnecessary alias to git clone.
• without adding some new functionality, palm new-machine would be present in everyone's command list, always. Even though it's only useful on day 1. This feels a little messy to me.
• Since palm is no longer internal to Palmetto, we would need to make any messaging around requesting access/credentials configurable.

[ethanknox-palmetto]

Good points - some of them I was thinking about this weekend,

• Running palm outside of an initialized git repo is not currently supported. Trying to do anything with Palm outside of a repo will raise an error due to checking for branch validation.

^^ hmm.. maybe new-machine doesn't use the standard palm interface? similar to how django-admin startproject mysite is only used initially, and manage.py is the interface for every following command? that would also break us free of showing new-machine in the help because it would be a command for the attached mini-package palm-admin.

• .env files should usually not be comitted to VCS, our pattern of using interpolation works in some cases, but not all. We should be careful about baking this pattern into palm as it could lead users down a dangerous path (having .env files committed to source control). Without .env management, palm clone just feels like an unnecessary alias to git clone.

^^ was thinking about this. maybe we encourage a good pattern by compiling from .env.template. If there is already an .env (for projects where all the variables are stored as envars and it is used as a readable index) we check to see if they are set and if not prompt to set them. For projects where there is NOT an .env we check to make sure it is not in source control (updating .gitignore if needed) prompt the user for each envar, set them, and generate the .env file.

• without adding some new functionality, palm new-machine would be present in everyone's command list, always. Even though it's only useful on day 1. This feels a little messy to me.

^^ that palm-admin or whatever micro package could solve this maybe

• Since palm is no longer internal to Palmetto, we would need to make any messaging around requesting access/credentials configurable.

^^ Oh yeah, I was thinking this should maybe be in the .palm/ directory as a configurable yaml or markdown file. So as the lead dev on a project you would build that out, maybe as a key/value pair for each envar

BANANASTAND_STITCH_API_KEY: |
    Stitch API Keys are managed by IT Services in the Toronto office. 
    Contact them at [email protected] to request an API key, version 2 or higher. 

This literally becomes "tribal knowledge as code" for onboarding ;)

[jakeberesford-palmetto]

I think you and I were having similar thoughts about this over the weekend. I just did a spike of how I think this should work - take a look at spike/setup-machine - this solves My 1st and 3rd points

Essentially, I've added exception handling around running palm outside of a git repo. Then I'm using that exception to change the set of plugins we're loading commands from in that context - right now I just have a shell of setup_plugin to prove the functionality works.

Still not 100% sold on the .env thing. I get the value, and I think the point about .env.template is a good one, my concern is still that this is dangerous grounds from a security perspective. If we can come up with a solution that helps prevent credentials get committed, I'd get behind this. The idea of a /.palm/env_vars.yaml (or similar) is interesting, and could be a more useful replacement for .env.template

BANANASTAND_STITCH_API_KEY: 
    default: ${BANANASTAND_STITCH_API_KEY}
    message: |
        Stitch API Keys are managed by IT Services in the Toronto office. 
        Contact them at [email protected] to request an API key, version 2 or higher. 

Then, palm clone would read this, check for existing globally exported defaults, print the message if they aren't found, and create the .env (and make sure it's listed in .gitignore

[ethanknox-palmetto]

Still not 100% sold on the .env thing. I get the value, and I think the point about .env.template is a good one, my concern is still that this is dangerous grounds from a security perspective. If we can come up with a solution that helps prevent credentials get committed, I'd get behind this. The idea of a /.palm/env_vars.yaml (or similar) is interesting, and could be a more useful replacement for .env.template

OOOH what if palm yells at you any time the .env file has a non-interpreted value in it? so for example I have a file .env-template:

BANANA_HOST=https://bananacompany.redshift.aws.com
BANANA_USER=${BANANA_USER}
BANANA_PASSWORD=${BANANA_PASSWORD}

when you run palm clone [email protected]/banana/bananarepo.git it clones the repo, then you get this prompt series:

cloning repo banana/bananarepo...
cloned. 
preparing variables and secrets... 
**WARNING!** environment variable BANANA_HOST is a plaintext variable, 
this could is very dangerous! Do you want  to proceed? (y/n): 
>> y
setting plaintext envar BANANA_HOST...
set. 

please enter a value for BANANA_USER (type `help!` if you are unsure of this value):
>> help! 
BANANA_USER should be the username provided to you by the banana application team. 
If you need a username and do not have one, 
contact Tobias Fünke via email ([email protected]) to have one assigned.

please enter a value for BANANA_USER (type `help!` if you are unsure of this value):
>> job_bluth
setting environment variable BANANA_USER...
set! 

Congrats, bananarepo is ready to go! to get started, run palm --help from inside the repo. 

[ethanknox-palmetto]

re-thinking envs..
maybe we follow 2 paths. Ask the user if they have a provided env, if they do, drop it. if they do not, do the cycle above, and also ask if they should be set globally. this supports 2 patterns:

• I-already-have-an-env is perfect for temporary, single app or untrusted users. for a contractor, they are simply sent the env file they will use in an email. quick and simple, the limitation is that you can't share envs across projects but that is probably not an issue for this use case.
• help-me-build-an-env is perfect for onboarding permanent employees. it tells them where to go to get env values themselves, builds the env in a way that is shared across projects (setting system envars) so they can use the same creds across dozens of internal repos really easily. This requires more engagement with the business, which is actually ideal for this use case.