Merge pull request #89 from praetorian-inc/dev

Gato Release 1.7

.github/workflows/integration_sh.yaml

@@ -6,11 +6,29 @@ env:
     SH_RUNNER_MANAGE_TOKEN: ${{secrets.SH_RUNNER_MANAGE_TOKEN }}
 
 jobs:
+
+    SH_Runner_Cleanup:
+    # First remove the SH runner in case it is still attached
+        name: Remove Self-Hosted Runner
+        runs-on: ubuntu-latest
+        steps:
+          - name: Checkout the repo
+            uses: actions/checkout@v3
+          - name: Setup python
+            uses: actions/setup-python@v4
+            with:
+              python-version: 3.11
+          - name: Force Remove SH Runner
+            run: |
+              pip install requests
+              python test/runner_helper.py force_remove
+
     # This job creates a self-hosted runner attached to the
     # test organization.
     Integration_SH_Runner:
         name: Ephemeral Self-Hosted Runner
         runs-on: ubuntu-latest
+        if: always()
         timeout-minutes: 16
         steps:
           - name: Checkout the repo
@@ -33,13 +51,12 @@ jobs:
             run: |
               cd actions-runner
               python ../test/runner_helper.py remove
-    SH_Runner_Cleanup:
-    # Removes the self-hosted runner forcibly in the event the cleanup operation
-    # did not work.
+
+    SH_Runner_Cleanup_2:
+    # remove the SH runner again
         name: Remove Self-Hosted Runner
         runs-on: ubuntu-latest
         needs: Integration_SH_Runner
-        if: failure()
         steps:
           - name: Checkout the repo
             uses: actions/checkout@v3
@@ -51,3 +68,4 @@ jobs:
             run: |
               pip install requests
               python test/runner_helper.py force_remove
+

.github/workflows/pytest.yaml

@@ -1,5 +1,5 @@
 name: Test and Lint
-on: 
+on:
     push:
     pull_request:
 
@@ -9,7 +9,8 @@ jobs:
         runs-on: ubuntu-latest
         strategy:
           matrix:
-            python-version: ["3.7", "3.8", "3.9", "3.10"]
+            python-version: ["3.9", "3.10", "3.11", "3.12"]
+
 
         timeout-minutes: 10
 
@@ -31,14 +32,14 @@ jobs:
               flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
           - name: Test with Pytest
             run: |
-              pytest --cov-fail-under=80
+              pytest --cov-fail-under=75
 
     OSX-test-and-lint:
         name: OS X Test and Lint
         runs-on: macos-latest
         strategy:
           matrix:
-            python-version: ["3.7", "3.8", "3.9", "3.10"]
+            python-version: ["3.11", "3.12"]
 
         timeout-minutes: 10
 
@@ -60,4 +61,4 @@ jobs:
               flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
           - name: Test with Pytest
             run: |
-              pytest --cov-fail-under=80
+              pytest --cov-fail-under=75

.gitignore

@@ -143,6 +143,10 @@ venv.bak/
 .dmypy.json
 dmypy.json
 
+# outputs
+*.json
+*.txt
+
 # Pyre type checker
 .pyre/
 

README.md

@@ -14,37 +14,48 @@ repositories.
 
 The tool has post-exploitation features to leverage a compromised personal
 access token in addition to enumeration features to identify poisoned pipeline
-execution vulnerabilities against public repositories that use self-hosted GitHub Actions 
-runners.
+execution vulnerabilities and actions artifacts secrets against public repositories.
 
 GitHub recommends that self-hosted runners only be utilized for private repositories, however, there are thousands of organizations that utilize self-hosted runners. Default configurations are often vulnerable, and Gato uses a mix of workflow file analysis and run-log analysis to identify potentially vulnerable repositories at scale.
 
-## Version 1.6
+## Version 1.7
 
-Gato version 1.6 improves the public repository enumeration feature set.
+Gato version 1.7 introduces the **Actions Artifacts Secrets Scanner**.
 
-Previously, Gato's code search functionality by default only looked for
-yaml files that explicitly had "self-hosted" in the name. Now, the
-code search functionality supports a SourceGraph query. This query has a 
-lower false negative rate and is not limited by GitHub's code search limit.
+The Actions Artifacts Secrets Scanner enumerates [GitHub Actions workflow artifacts](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/storing-and-sharing-data-from-a-workflow) for secrets. Praetorian researchers have leveraged the scanner to identify critical vulnerabilities in several prominent open-sourced projects. Details of these vulnerabilities will be released once the disclosure processes are complete. This work was initially inspired by research from [Palo Alto Networks](https://unit42.paloaltonetworks.com/github-repo-artifacts-leak-tokens/). 
 
-For example, the following query will identify public repositories that use 
-self-hosted runners:
+The Actions Artifacts Secrets Scanner performs the following actions:
+1. Downloads GitHub Actions workflow artifacts from the target repository
+2. Recursively extracts the downloaded artifacts
+3. Scans the artifacts for secrets with [NoseyParker](https://github.com/praetorian-inc/noseyparker)
+4. Reports the results
 
-`gato search --sourcegraph --output-text public_repos.txt`
+[NoseyParker](https://github.com/praetorian-inc/noseyparker/releases) must be installed in the $PATH of the system running the Actions Artifacts Secrets Scanner.
 
-This can be fed back into Gato's enumeration feature:
+Secrets in public workflow artifacts can contain many false positives. By default, the Actions Artifacts Secrets Scanner excludes rules and results associated with common false positives. To include all secrets scanning results, you can use the `--include_all_artifact_secrets` flag.
 
-`gato enumerate --repositories public_repos.txt --output-json enumeration_results.json`
+Here is an example command that runs the Actions Artifacts Secrets Scanner on a list of GitHub organizations, disables self-hosted runner enumeration, includes all artifact secret results, and outputs to a JSON file.
 
-Additionally the release contains several improvements under the hood to speed up the enumeration process. This includes changes to limit redundant run-log downloads (which are the slowest part of Gato's enumeration process) and using the GraphQL API to download workflow files when enumerating an entire organization. Finally, Gato will use a heuristic to detect if an attached runner is non-ephemeral. Most poisoned pipeline execution attacks require a non-ephemeral runner in order to exploit.
+```
+gato e --enum_wf_artifacts --include_all_artifact_secrets --skip_sh_runner_enum -O testorgs.txt  -oJ testorgoutput.json
+```
+
+By default, the Actions Artifacts Secrets Scanner imposes the following limitations to reduce the time spent on a single repository:
+- Only downloads one artifact per name
+- Downloads a maximum of 50 artifacts per repository
+- Downloads a maximum of 10 files greater than 536 MBs
+- Does not download any files greater than 2.68 GBs
+
+These constraints were optimized such that the Actions Artifacts Secrets Scanner can scan the top 200 GitHub organizations within 48 hours. If you want to modify these constraints, you can update them in the `scan_wf_artifacts()` function of `gato/enumerate/repository.py`.
 
-### New Features
+## New Features
 
-* SourceGraph Search Functionality
-* Improved Public Repository Enumeration Speed
-* Improved Workflow File Analysis
-* Non-ephemeral self-hosted runner detection
+- Added the Actions Artifacts Secrets Scanner
+- Support to run modules on a list of GitHub organizations
+- Ability to disable self-hosted runner enumeration
+- Option to disable sleep
+- Differentates between public and private repos in output
+- Several bug fixes
 
 ## Who is it for?
 
@@ -68,6 +79,7 @@ Additionally the release contains several improvements under the hood to speed u
 * Automated workflow secrets exfiltration
 * SOCKS5 Proxy Support
 * HTTPS Proxy Support
+* GitHub Actions Workflow Artifacts Secrets Scanning
 
 ## Getting Started
 
@@ -98,7 +110,7 @@ suite on the `dev` branch, so there should not be any _blatant_ bugs.
 
 If you want to use the `dev` branch, just check it out prior to running pip install - that's it!
 
-If you do run into any for your specific use case, by all means open an issue!
+If you do run into any bugs for your specific use case, by all means open an issue!
 
 
 ### Usage
@@ -128,7 +140,7 @@ mode or running unit tests, please see the [wiki](https://github.com/praetorian-
 
 ## Documentation
 
-Please see the [wiki](https://github.com/praetorian-inc/gato/wiki).
+Please see the [wiki](https://github.com/praetorian-inc/gato/wiki)
  for detailed documentation, as well as [OpSec](https://github.com/praetorian-inc/gato/wiki/opsec) considerations 
 for the tool's various modules!
 
@@ -146,8 +158,8 @@ If you are unsure if the behavior is a bug, use the discussions section instead!
 Contributions are welcome! Please [review](https://github.com/praetorian-inc/gato/wiki/Project-Design) our design methodology and coding 
 standards before working on a new feature!
 
-Additionally, if you are proposing significant changes to the tool, please open 
-an issue [open an issue](https://github.com/praetorian-inc/gato/issues/new) to 
+Additionally, if you are proposing significant changes to the tool, please
+[open an issue](https://github.com/praetorian-inc/gato/issues/new) to
 start a conversation about the motivation for the changes.
 
 ## License