Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add reusable workflow scripts #307

Merged
merged 19 commits into from
Jun 2, 2021
Merged

Conversation

imheresamir
Copy link
Contributor

@imheresamir imheresamir commented May 18, 2021

What does this PR do?

This PR adds end-to-end workflow test scripts, ported over from conjurdemos/kubernetes-conjur-demo
The ported scripts were modified to:

  1. Install conjur-oss in a KinD cluster
  2. Rework cluster config to use our new helm charts kubernetes-cluster-config and app-namespace-prep
  3. Use bitnami/postgresql helm chart for the sample app backend

The app-deploy chart was modified to take enabled flags to select one or more subcharts. The app-summon-sidecar subchart is enabled using --set authn-k8s.enabled=true when installing the parent app-deploy chart.

The workflow can be tested by launching bin/test-workflow/start.

Commit 2 (947fb35) copies unchanged files from conjurdemos/kubernetes-conjur-demo (a4272e3)

Commit 3 (51a92d5) is where the script changes are.

What ticket does this PR close?

Resolves #239 and #301

Checklists

Change log

  • The CHANGELOG has been updated, or
  • This PR does not include user-facing changes and doesn't require a CHANGELOG update

Test coverage

  • This PR includes new unit and integration tests to go with the code changes, or
  • The changes in this PR do not require tests

Documentation

  • Docs (e.g. READMEs) were updated in this PR, and/or there is a follow-on issue to update docs, or
  • This PR does not require updating any documentation

Manual tests

If you are preparing for a release, have you run the following manual tests to verify existing functionality continues to function as expected?

@imheresamir imheresamir requested review from a team as code owners May 18, 2021 14:05
@imheresamir imheresamir self-assigned this May 18, 2021
@imheresamir imheresamir changed the title Add reusable workflow scripts 2 Add reusable workflow scripts May 18, 2021
@imheresamir imheresamir marked this pull request as draft May 18, 2021 14:57
@imheresamir imheresamir marked this pull request as ready for review May 18, 2021 15:23
@imheresamir imheresamir linked an issue May 18, 2021 that may be closed by this pull request
1 task
Copy link
Contributor

@diverdane diverdane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great stuff!!!
I'm still working my way through this, but here are the comments that I have now.

helm/kubernetes-cluster-prep/bin/get-conjur-cert.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/start Show resolved Hide resolved
bin/test-workflow/kubernetes/conjur-cli.yml Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
@imheresamir imheresamir force-pushed the add-reusable-workflow-scripts-2 branch from 21d09a9 to 6ef73bb Compare May 19, 2021 21:02
Copy link
Contributor

@diverdane diverdane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More comments. Still plowing through everything.

bin/test-workflow/0_prep_conjur_in_kind.sh Show resolved Hide resolved
bin/test-workflow/4_kubernetes_cluster_prep.sh Outdated Show resolved Hide resolved
bin/test-workflow/4_kubernetes_cluster_prep.sh Outdated Show resolved Hide resolved
bin/test-workflow/5_app_namespace_prep.sh Outdated Show resolved Hide resolved
bin/test-workflow/6_app_build_and_push_containers.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/8_app_verify_authentication.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/7_app_deploy.sh Outdated Show resolved Hide resolved
bin/test-workflow/8_app_verify_authentication.sh Outdated Show resolved Hide resolved
bin/test-workflow/8_app_verify_authentication.sh Outdated Show resolved Hide resolved
bin/test-workflow/8_app_verify_authentication.sh Outdated Show resolved Hide resolved
bin/test-workflow/etc/secretless.yml Outdated Show resolved Hide resolved
Copy link
Contributor

@doodlesbykumbi doodlesbykumbi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a lot of great improvements in the PR. I've left some comments about things that stood out, but I'm still thinking about ways in which we can eradicate the undesirable things that are inherited from the pre-existing scripts.

I think it's good for us to really take that step back and look at this workflow from a high-level, and ask ourselves how we would implement each section given a blank slate. In this case we really do have a blank slate.

A key criteria (for myself) in considering the scripts reusable is for them to be compossible in somewhat arbitrary ways. We are constrained on how granular that decomposition by the logical boundaries of the tasks. I imagine the reusable workflow scripts should allow me to provide input to a script that can carry out any logical task listed in the docs and provide output in a way that could potentially be piped into another script.

bin/test-workflow/utils.sh Outdated Show resolved Hide resolved
bin/test-workflow/set_env_vars.sh Outdated Show resolved Hide resolved
bin/test-workflow/start Outdated Show resolved Hide resolved
COPY --from=test-app-builder /usr/local/bin/summon /usr/local/bin/summon

#---copy secrets.yml into image---#
COPY tmp.$namespace.secrets.yml /etc/secrets.yml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be introduced as a configmap.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's discuss and create an issue

bin/test-workflow/stop Outdated Show resolved Hide resolved
bin/test-workflow/policy/users.yml Show resolved Hide resolved
bin/test-workflow/openshift/postgres.template.yml Outdated Show resolved Hide resolved
bin/test-workflow/kubernetes/mysql.template.yml Outdated Show resolved Hide resolved
@imheresamir imheresamir force-pushed the add-reusable-workflow-scripts-2 branch 3 times, most recently from 282ff99 to 001386c Compare May 27, 2021 02:08
@imheresamir imheresamir force-pushed the add-reusable-workflow-scripts-2 branch from 40a4bcc to 75690a5 Compare May 27, 2021 05:20
Copy link
Contributor

@doodlesbykumbi doodlesbykumbi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: The PR will require squashing the commits to merge since there's been a lot of back and forth and so the commit history isn't necessary made up of logical units.

Approved, with some follow up issues in mind, here's a rough list

  1. Rip out platform branching
  2. Revisit platform branching. Helm is great for this, openshift=true  can be used to toggle between openshift and k8s in the spirit of the bitnami postgres chart
  3. Update policy loading to match dap-wiki (simpler)
  4. Refactor scripts in a way that allows personas to execute steps independent of each other in a way that aligns with dap-wiki
  5. Refactor wholesale loading of policy, to allow for loading of policy and writing of variables local to scripts and using the identity of the relevant persona

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

There are reusable scripts for development environments and automated testing
3 participants