-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Generate and upload final files as artifacts. (#1)
* First upload. * Save the final files. * Enable all sizes. * use 2 cpus. * Split large primes. * Download only specific size. * Split after generate. * Generate twice similar to upstream OpenSSH. * Run 12 jobs for 7680. * Update after review. * More parallel jobs. * Give 'split: 24' a try for size 8192. * Split into 24 as suggested, Co-authored-by: Adi Roiban <[email protected]> * More jobs for sizes 6144 and 7680. * Fixed split for size 6144. * One last tweak for size 7680 to also finish under 2h. * Only run scheduled once a month. * Removed converting to Chevah Python format. * Updated time estimates. * Updated README.rst with download example. Co-authored-by: Adi Roiban <[email protected]> Co-authored-by: dumol <[email protected]>
- Loading branch information
Showing
3 changed files
with
272 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,36 +1,252 @@ | ||
| # | ||
| # Generated the prime numbers and generated used by SSH DH Group KEX. | ||
| # | ||
| name: GEN | ||
| # Generating is fast. Max 15 minutes. | ||
| # Validating is slow. 30 hours. | ||
| # | ||
| # Since the GitHub VM have 2 CPU each job will run 2 validating tasks. | ||
| # A job can run for 6 hours... so 12 hours of validation time per job. | ||
| # With 3 jobs for a certain size we should do it. | ||
| # | ||
| # It generates primes for 1024, 1536, 2048, 3072, 4096, 6144, 7680, 8192 | ||
| # | ||
| name: CD | ||
|
|
||
| # Only run on pull requests and only for Python files. | ||
| on: | ||
| push: | ||
| branches: [ master ] | ||
| # Pull requests are enabled for development. | ||
| pull_request: | ||
| branches: [ master ] | ||
| branches: [ main ] | ||
|
|
||
| # https://crontab.guru | ||
| schedule: | ||
| # Once a month | ||
| - cron: '0 2 1 * *' | ||
|
|
||
| jobs: | ||
|
|
||
| prevent_duplicates: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: chevah/auto-cancel-redundant-job@v1 | ||
| - uses: chevah/python-info-action@v1 | ||
|
|
||
|
|
||
| generate: | ||
| name: 'Size ${{ matrix.size }}' | ||
| name: 'generate ${{ matrix.size }}' | ||
| needs: prevent_duplicates | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| # size: [ 1024, 1536, 2048, 3072, 4096, 6144, 8192] | ||
| size: [ 1024, 1536, 2048] | ||
| size: [ 1024, 1536, 2048, 3072, 4096 ] | ||
| split: [2] | ||
| include: | ||
| - size: 6144 | ||
| split: 8 | ||
| - size: 7680 | ||
| split: 20 | ||
| - size: 8192 | ||
| split: 24 | ||
|
|
||
|
|
||
| runs-on: ubuntu-latest | ||
| steps: | ||
|
|
||
| # Make sure we don't have multiple jobs | ||
| - uses: chevah/auto-cancel-redundant-job@v1 | ||
|
|
||
| - name: Candidate | ||
| run: | | ||
| ssh-keygen -G candidate-${{ matrix.size }} -b ${{ matrix.size }} | ||
| ssh-keygen -M generate -O bits=${{ matrix.size }} full.1 & | ||
| PID1=$! | ||
| ssh-keygen -M generate -O bits=${{ matrix.size }} full.2 & | ||
| PID2=$! | ||
| # After getting the PID of each run wait for both | ||
| # and record any failure signaled via a non-zero exit code. | ||
| FAIL=0 | ||
| wait $PID1 || let "FAIL+=1" | ||
| wait $PID2 || let "FAIL+=1" | ||
| cat full.2 >> full.1 | ||
| # Split as screening a single file on GitHub can't take more than | ||
| # 6 hours. | ||
| # We split with 2 digits as 7680 requires 12 chunks. | ||
| # Split as round-robin as it looks like the first batch alwasy | ||
| # takes longer to validate. | ||
| split -d -a 2 -n r/${{ matrix.split }} full.1 g-${{ matrix.size }}. | ||
| exit $FAIL | ||
| - uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: candidate-${{ matrix.size }} | ||
| path: g-* | ||
| retention-days: 1 | ||
|
|
||
|
|
||
| screen: | ||
| needs: generate | ||
| name: 'screen ${{ matrix.size }}' | ||
| strategy: | ||
| matrix: | ||
| size: [ 1024, 1536, 2048, 3072, 4096 ] | ||
| split: [2] | ||
| start: [0] | ||
|
|
||
| # For big primes it can take 30 hours on a single CPU. | ||
| include: | ||
| - size: 6144 | ||
| split: 8 | ||
| start: 0 | ||
| - size: 6144 | ||
| split: 8 | ||
| start: 2 | ||
| - size: 6144 | ||
| split: 8 | ||
| start: 4 | ||
| - size: 6144 | ||
| split: 8 | ||
| start: 6 | ||
|
|
||
| - size: 7680 | ||
| split: 20 | ||
| start: 0 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 2 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 4 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 6 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 8 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 10 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 12 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 14 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 16 | ||
| - size: 7680 | ||
| split: 20 | ||
| start: 18 | ||
|
|
||
| - size: 8192 | ||
| split: 24 | ||
| start: 0 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 2 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 4 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 6 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 8 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 10 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 12 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 14 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 16 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 18 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 20 | ||
| - size: 8192 | ||
| split: 24 | ||
| start: 22 | ||
|
|
||
| runs-on: ubuntu-latest | ||
| steps: | ||
|
|
||
| - uses: actions/download-artifact@v2 | ||
| with: | ||
| name: candidate-${{ matrix.size }} | ||
|
|
||
| - name: Validate | ||
| run: | | ||
| ssh-keygen -f candidate-${{ matrix.size }} -T validated-${{ matrix.size }} | ||
| # Prepare all input as shell variables. | ||
| size=${{ matrix.size }} | ||
| first_index=${{ matrix.start }} | ||
| second_index=$(( $first_index + 1 )) | ||
| first_index=`printf %02d $first_index` | ||
| second_index=`printf %02d $second_index` | ||
| # Some debugging. | ||
| ls -al * | ||
| declare -p | ||
| # The action. | ||
| ssh-keygen -M screen -f g-$size.$first_index v-$size-$first_index & | ||
| PID1=$! | ||
| ssh-keygen -M screen -f g-$size.$second_index v-$size-$second_index & | ||
| PID2=$! | ||
| FAIL=0 | ||
| wait $PID1 || let "FAIL+=1" | ||
| wait $PID2 || let "FAIL+=1" | ||
| exit $FAIL | ||
| - uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: validated | ||
| path: v-* | ||
| retention-days: 1 | ||
|
|
||
|
|
||
| combine: | ||
| needs: screen | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
|
|
||
| - uses: actions/checkout@v2 | ||
| - uses: actions/download-artifact@v2 | ||
| with: | ||
| name: validated | ||
|
|
||
| - name: Debug | ||
| run: | | ||
| ls -al * | ||
| wc -l v-* | ||
| - name: OpenSSH /etc/ssh/moduli | ||
| run: | | ||
| now=`date -Iminutes` | ||
| echo "# https://github.com/chevah/ssh-moduli/ $now" > etc_ssh_moduli | ||
| echo "# Time Type Tests Tries Size Generator Modulus" >> etc_ssh_moduli | ||
| cat v-* | sort -u >> etc_ssh_moduli | ||
| # Clean all previous artifact to reduce the noise on the artifacts page. | ||
| - uses: chevah/delete-artifact@1-glob-support | ||
| with: | ||
| name: '**' | ||
| useGlob: true | ||
|
|
||
| - uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: etc_ssh_moduli | ||
| path: | | ||
| etc_ssh_moduli | ||
| - name: Artifacts link | ||
| run: | | ||
| echo ${{ github.event.repository.html_url }}/actions/runs/${{ github.run_id }}#artifacts |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| ssh-moduli | ||
| ========== | ||
|
|
||
| Helper to generate the prime numbers used in the SSH Diffie-Hellman Group Exchange key exchange | ||
|
|
||
| The resulting files in OpenSSH format are stored as artifacts. | ||
|
|
||
| See the artifacts for each scheduled run: | ||
| https://github.com/chevah/ssh-moduli/actions | ||
|
|
||
| You can retrieve the links for the already generated artifacts | ||
| via the GitHub API using this long command:: | ||
|
|
||
| curl -s https://api.github.com/repos/chevah/ssh-moduli/actions/artifacts\?per_page\=9 | jq '[.artifacts[] | {name : .name, archive_download_url : .archive_download_url}]' | jq -r '.[] | select (.name == "etc_ssh_moduli") | .archive_download_url' | ||
| OpenSSH upstream generation script which generates the candidates twice: | ||
| https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/moduli-gen/ | ||
|
|
||
| You can see the OpenSSH portable moduli file here. | ||
| This is pulled by Debian and Ubuntu...and I guess other distributions. | ||
| Updated 2 or 3 times per year. | ||
| https://github.com/openssh/openssh-portable/commits/master/moduli | ||
|
|
||
| It needs latest OpenSSH:: | ||
|
|
||
| ssh-keygen -M generate -O bits=2048 moduli-2048.candidates | ||
| ssh-keygen -M screen -f moduli-2048.candidates moduli-2048 | ||
|
|
||
|
|
||
| Limits for GitHub Hosted VM [1]: | ||
| * 6 hours per job | ||
| * 72 hours per workflow | ||
| * 20 concurrent jobs | ||
|
|
||
| Generating the candidates is fast. About 15 minutes for 8k. | ||
|
|
||
| Validating is slow. Some rough estimates: | ||
| * 4096 - 1 hour | ||
| * 6144 - 8 hours | ||
| * 7680 - 20 hours | ||
| * 8192 - 24 hours | ||
|
|
||
|
|
||
| [1] https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration#usage-limits |