Make User Supply Docker Image(s) [major] (#7)

* Use Docker Container * fix build location * use `$GITHUB_ACTION_PATH` * use `$GITHUB_ACTION_PATH` - 2 * add missing Dockerfile commands * setup env * setup env - 2 * docker-in-docker * `shell: bash` * `docker rm $(docker ps -a -q) -f || true` * split `docker build` into own step * split `docker build` into own step - 2 * activate docker daemon in entrypoint.sh * env fix * env fix - 2 * entrypoint path * don't suppress docker-in-docker error * don't suppress docker-in-docker error - 2 * don't suppress docker-in-docker error - 3 * docker-in-docker - 7? * docker-in-docker - 8 * docker-in-docker - 9 * fix install * `semver_parser_tools_local.py` * decrease sleep after `dockerd` * (debug) * (debug - 2) * (debug - 3) * (debug - 4) * (debug - 5) * (debug - 6) * (debug - 7) * (debug - 8) * (debug - 9) * (debug - 10) * (debug - 11) * add container pulling/building logic * add container pulling/building logic - 2 * add `~~~`s * add `~~~`s - 2 * add `~~~`s - 3 * fix pull * fix pull - 2 * check if there *should* be an image (v# tag) * check if there *should* be an image (v# tag) - 2 * step names * (debug) * (debug - 2) * docker save * docker load * save disk space * `trap 'rm -rf saved-images/'` * get rid of `dockerfile_nametags` * dep from the user-supplied containers * dep from the user-supplied containers - 2 * refactor `gen-deps-from-user-docker-images.sh` * docker image rm $image && docker system prune --all --force # save disk space * pt 2 * remove sudo * parallelize * save more disk space * no Dockerfile * `/bin/bash` * add back `pip -r` * fix pwd's * `gen-deps-within-container.sh` * bash typo * (debug) * no relative paths! * no relative paths! - 2 * no relative paths! - 3 * no relative paths! - 4 * subititle fix * update subtitle * again, no relative paths! * add `use_directory` * add "-image-" to fname * echos * `podman images` * `podman pull docker-daemon://$DOCKER_IMAGE` * `podman pull docker-daemon:$DOCKER_IMAGE` * call them `dependencies-docker-*` * update README * compare counts of dockerfiles vs images * compare counts of dockerfiles vs images - 2 * verbage
WIPACrepo · Jul 26, 2024 · 1abd762 · 1abd762
1 parent 9db23c6
commit 1abd762
Show file tree

Hide file tree

Showing 11 changed files with 306 additions and 187 deletions.
diff --git a/.github/workflows/image-publish.yml b/.github/workflows/image-publish.yml
@@ -0,0 +1,48 @@
+name: docker release
+
+
+on:
+  push:
+    branches-ignore:
+      - '**'
+    tags:
+      - '**'
+  workflow_dispatch:
+
+
+env:
+  DOCKER_IMAGE_NAME: ghcr.io/wipacrepo/wipac-dev-py-dependencies-action
+
+
+jobs:
+
+  docker:
+    name: "Docker Image"
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Project
+        uses: actions/checkout@v4
+      - name: Docker meta
+        id: docker_meta
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            ${{ env.DOCKER_IMAGE_NAME }}
+          tags: |
+            type=ref,event=branch
+            type=semver,pattern={{major}}
+            type=semver,pattern={{major}}.{{minor}}
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Push Docker Image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: Dockerfile
+          push: true
+          tags: ${{ steps.docker_meta.outputs.tags }}
+          labels: ${{ steps.docker_meta.outputs.labels }}
diff --git a/README.md b/README.md
@@ -1,16 +1,25 @@
 # wipac-dev-py-dependencies-action
-GitHub Action Package for Automating Python-Package Dependency Management
 
+GitHub Action Package for Automating Python-Package Dependency Management
 
 ## Overview
-This GitHub Action creates 1+ `dependencies*.log`-type file(s) for documenting dependency versions (ex: `dependencies.log`, `dependencies-dev.log`, `dependencies-from-Dockerfile.log`, etc.). These files are similar to `requirements*.txt`-type files, with the distinct difference that `dependencies*.log`-type files are not intended to be the source of truth, but rather, a reflection of the tested environment(s). If dependency-version pinning is wanted, it should be done by the user in the `setup.cfg`/`pyproject.toml` file.
+
+This GitHub Action creates 1+ `dependencies*.log` file(s) for documenting dependency versions (ex: `dependencies.log`, `dependencies-dev.log`, `dependencies-docker-default.log`, etc.). These files are similar to `requirements*.txt`-type files, with the distinct difference that *`dependencies*.log`-type files are a reflection of the built environment(s).* If dependency-version pinning is wanted, it should be done by the user in the `setup.cfg`/`pyproject.toml` file.
 
 ### Details
-The root directory's `dependencies.log` is overwritten/updated (by way of `pip freeze` + [`pipdeptree`](https://pypi.org/project/pipdeptree/)) along with dedicated `dependencies-EXTRA.log` files for each package "extra".
 
-_However,_ if there is a `Dockerfile` present at the root of the target repo, a similar process occurs but _within_ a container built from the `Dockerfile`. This log file is named `dependencies-from-Dockerfile.log`. If there are other `Dockerfile`s present (ex: `Dockerfile-foo`), additional files are generated using the appropriate name (ex: `dependencies-from-Dockerfile-foo.log`).
+This action uses `pip freeze` + [`pipdeptree`](https://pypi.org/project/pipdeptree/) to generate the `dependencies*.log` file(s). All current `dependencies*.log` files are deleted/overwritten/updated. By default, these file(s) are placed at the client repository's root. Setting the input arg `use_directory: true` will place the generated file(s) in `dependencies-logs/`.
+
+#### Generating from Python Package
+
+By default, this action generates a `dependencies*.log` file using the environment built from the client's Python package. In addition, a dedicated `dependencies-EXTRA.log` file is generated for each package "extra".
+
+#### Generating from Docker Images
+
+If the user supplies Docker image(s) tagged with `"py-dep-this"` (configurable by the input arg, `docker_tag_to_dep`). `dependencies*.log` file(s), named `dependencies-docker-{IMAGE_NAME}.log`, are generated from within each container.
 
 ### Example File
+
         ```
         #
         # This file was autogenerated by WIPACrepo/wipac-dev-py-setup-action
@@ -43,4 +52,5 @@ _However,_ if there is a `Dockerfile` present at the root of the target repo, a
         ```
 
 ### Full CI-Workflow: Using Alongside Other GitHub Actions
+
 See https://github.com/WIPACrepo/wipac-dev-py-setup-action#full-ci-workflow-using-alongside-other-github-actions
diff --git a/action.sh b/action.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+sleep 0.1 && echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
+echo "$( basename "$0" )..." && echo
+set -ex
+
+ls $REPO_PATH
+
+# env vars
+export PACKAGE_NAME=$(python3 $GITHUB_ACTION_PATH/utils/get_package_name.py .)
+
+
+# grab local copy to avoid path mangling -- replace when https://github.com/WIPACrepo/wipac-dev-py-dependencies-action/issues/6
+pip install requests semantic-version
+temp_dir=$(mktemp -d) && cd $temp_dir && trap 'rm -rf $temp_dir' EXIT
+wget https://raw.githubusercontent.com/WIPACrepo/wipac-dev-tools/main/wipac_dev_tools/semver_parser_tools.py -O $temp_dir/semver_parser_tools_local.py
+
+# get python3 version (max) -- copied from https://github.com/WIPACrepo/wipac-dev-py-versions-action/blob/main/action.yml
+export PACKAGE_MAX_PYTHON_VERSION=$(python -c '
+import os, re
+import semver_parser_tools_local as semver_parser_tools
+
+repo_path = os.environ["REPO_PATH"]
+
+semver_range = ""
+if os.path.isfile(f"{repo_path}/pyproject.toml"):
+    # ex: requires-python = ">=3.8, <3.13"
+    pat = re.compile(r"requires-python = \"(?P<semver_range>[^\"]+)\"$")
+    with open(f"{repo_path}/pyproject.toml") as f:
+        for line in f:
+            if m := pat.match(line):
+                semver_range = m.group("semver_range")
+    if not semver_range:
+        raise Exception("could not find `requires-python` entry in pyproject.toml")
+elif os.path.isfile(f"{repo_path}/setup.cfg"):
+    # ex: python_requires = >=3.8, <3.13
+    pat = re.compile(r"python_requires = (?P<semver_range>.+)$")
+    with open(f"{repo_path}/setup.cfg") as f:
+        for line in f:
+            if m := pat.match(line):
+                semver_range = m.group("semver_range")
+    if not semver_range:
+        raise Exception("could not find `python_requires` entry in setup.cfg")
+else:
+    raise Exception("could not find pyproject.toml nor setup.cfg")
+
+top_python = semver_parser_tools.get_latest_py3_release()
+all_matches = semver_parser_tools.list_all_majmin_versions(
+  major=top_python[0],
+  semver_range=semver_range,
+  max_minor=top_python[1],
+)
+print(f"{max(all_matches)[0]}.{max(all_matches)[1]}")
+')
+
+echo $PACKAGE_MAX_PYTHON_VERSION
+
+
+# Build
+if [ -f $REPO_PATH/Dockerfile ]; then
+  # from Dockerfile(s)...
+  $GITHUB_ACTION_PATH/generate_dep_logs/gen-deps-from-user-docker-images.sh
+else
+  # from setup.cfg...
+  $GITHUB_ACTION_PATH/generate_dep_logs/gen-deps-from-repo-python-pkg.sh
+fi
+
+sleep 0.1 && echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
diff --git a/action.yml b/action.yml
@@ -10,10 +10,15 @@ inputs:
     description: 'The email used for "git config user.email"'
     required: false
     default: [email protected]
-  dockerfile_nametags:
-    description: 'A space-separated list of image names and tags to apply to each Dockerfile for locally dependent builds (these images are built first), ex: Dockerfile:icecube/skymap_scanner:latest Dockerfile_pulsar:icecube/skymap_scanner:latest_pulsar'
+  docker_tag_to_dep:
+    description: 'The email used for "git config user.email"'
+    required: false
+    default: py-dep-this
+  use_directory:
+    description: 'Whether to put all generated dep-log files, relative to the project/repo directory; (true/false)'
     required: false
-    default: ''
+    default: "false"
+
 
 # outputs:
 #   random-number:
@@ -29,6 +34,8 @@ runs:
 
     - name: Is this the most recent commit? It won't be if the action was reran
       run: |
+        # check git status
+        echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
         git fetch &> /dev/null
         if [[ $(git status -sb | grep behind) ]]; then
           echo "IS_GIT_BEHIND=true" >> $GITHUB_ENV
@@ -39,6 +46,8 @@ runs:
     - name: Git config
       if: env.IS_GIT_BEHIND != 'true'
       run: |
+        # git config
+        echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
         git config user.name ${{ inputs.git_committer_name }}
         git config user.email ${{ inputs.git_committer_email }}
       shell: bash
@@ -52,9 +61,15 @@ runs:
 
     - name: Build dependencies.log (and commit)
       if: env.IS_GIT_BEHIND != 'true'
+      env:
+        ACTION_REPOSITORY: ${{ github.action_repository }}  # https://github.com/github/docs/issues/25336#issuecomment-1736251764
+        GITHUB_ACTION_PATH: ${{ github.action_path }}
+        DOCKER_TAG_TO_DEP: ${{ inputs.docker_tag_to_dep }}
+        DEST_DIRECTORY_IF_WANTED: dependencies-logs/
       run: |
-        set -x  # turn on debugging
-
+        # build dependencies.log (and commit)
+        set +x; echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"; set -x
+        
         # append permissive line to .gitignore since *.log is commonly present
         line='!dependencies*.log'
         if [[ ! $(grep -F "$line" .gitignore) ]]; then
@@ -66,75 +81,33 @@ runs:
           git commit -m "<bot> update .gitignore" || true  # okay if no change
           tail .gitignore
         fi
-
+        
+        set +x; echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"; set -x
+        
         # remove any old ones, then regenerate only what's needed
-        rm dependencies*.log || true
-
-        export PACKAGE_NAME=$(python3 ${{ github.action_path }}/utils/get_package_name.py .)
-        export GITHUB_ACTION_PATH=${{ github.action_path }}
-        export ACTION_REPOSITORY="WIPACrepo/wipac-dev-py-setup-action"
-
-        # grab local copy to avoid path mangling -- replace when https://github.com/WIPACrepo/wipac-dev-py-dependencies-action/issues/6
-        pip install requests semantic-version
-        wget https://raw.githubusercontent.com/WIPACrepo/wipac-dev-tools/main/wipac_dev_tools/semver_parser_tools.py -O semver_parser_tools_local.py
+        (find . -name "dependencies*.log" -type f -print0 | xargs -0 rm) || true
         
-        # get python3 version (max) -- copied from https://github.com/WIPACrepo/wipac-dev-py-versions-action/blob/main/action.yml
-        export PACKAGE_MAX_PYTHON_VERSION=$(python -c '
-        import os, re
-        import semver_parser_tools_local as semver_parser_tools
+        # run script
+        export REPO_PATH=$(pwd)
+        /bin/bash $GITHUB_ACTION_PATH/action.sh
         
-        semver_range = ""
-        if os.path.isfile("pyproject.toml"):
-            # ex: requires-python = ">=3.8, <3.13"
-            pat = re.compile(r"requires-python = \"(?P<semver_range>[^\"]+)\"$")
-            with open("pyproject.toml") as f:
-                for line in f:
-                    if m := pat.match(line):
-                        semver_range = m.group("semver_range")
-            if not semver_range:
-                raise Exception("could not find `requires-python` entry in pyproject.toml")
-        elif os.path.isfile("setup.cfg"):
-            # ex: python_requires = >=3.8, <3.13
-            pat = re.compile(r"python_requires = (?P<semver_range>.+)$")
-            with open("setup.cfg") as f:
-                for line in f:
-                    if m := pat.match(line):
-                        semver_range = m.group("semver_range")
-            if not semver_range:
-                raise Exception("could not find `python_requires` entry in setup.cfg")
-        else:
-            raise Exception("could not find pyproject.toml nor setup.cfg")
+        set +x; echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"; set -x
         
-        top_python = semver_parser_tools.get_latest_py3_release()
-        all_matches = semver_parser_tools.list_all_majmin_versions(
-          major=top_python[0],
-          semver_range=semver_range,
-          max_minor=top_python[1],
-        )
-        print(f"{max(all_matches)[0]}.{max(all_matches)[1]}")
-        ')
-        echo $PACKAGE_MAX_PYTHON_VERSION
-
-        # Build
-        if [ -f ./Dockerfile ]; then
-          # from Dockerfile(s)...
-          export DOCKERFILE_NAMETAGS=${{ inputs.dockerfile_nametags }}
-          ${{ github.action_path }}/generate_dep_logs/gen-deps-from-repo-dockerfiles.sh
-        else
-          # from setup.cfg...
-          ${{ github.action_path }}/generate_dep_logs/gen-deps-from-repo-python-pkg.sh
+        if [[ $( echo "${{ inputs.use_directory }}" | awk '{print tolower($0)}' ) == "true" ]]; then
+            mkdir -p $DEST_DIRECTORY_IF_WANTED
+            find . -name "dependencies*.log" -type f -exec mv -i {} $DEST_DIRECTORY_IF_WANTED  \;
         fi
-
+        
         # Commit
-        rm semver_parser_tools_local.py
         git add .
         git commit -m "<bot> update dependencies*.log files(s)" || true  # okay if no change
-
       shell: bash
 
     - name: Push changes
       if: env.IS_GIT_BEHIND != 'true'
       run: |
+        # push changes (if needed)
+        echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
         status=`git status 2>&1 | tee`
         ahead=`echo -n "${status}" 2> /dev/null | grep "Your branch is ahead of" &> /dev/null; echo "$?"`
         if [ "$ahead" -eq "1" ]; then

diff --git a/generate_dep_logs/gen-deps-from-repo-dockerfiles.sh b/generate_dep_logs/gen-deps-from-repo-dockerfiles.sh