scan-multiple-repositories always clones source repo on Azure DevOps #629

anael-l · 2024-01-27T16:20:55Z

Describe the bug

The scan-multiple-repositories command always clones the repo containing the frogbot-config.yml file instead of the repos defined in it
My pipeline + frogbot config are in a repo called CockpIT_frogbot
My frogbot config points to two repos CockpIT-front and CockpIT-back
As shown in the log, the config is read, but for both scans, the repo CockpIT_frogbot is cloned instead

Current behavior

##[section]Starting: Download and Run Frogbot
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.178.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
[command]/usr/bin/bash --noprofile --norc /opt/agt/_work/_temp/e71a6319-464e-4155-973a-d17de31b9031.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2395  100  2395    0     0  16292      0 --:--:-- --:--:-- --:--:-- 16292
Downloading the latest version of Frogbot...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 27.9M  100 27.9M    0     0  76.0M      0 --:--:-- --:--:-- --:--:-- 76.0M
Frogbot downloaded successfully!
17:08:38 [Info] Frogbot version: 2.19.8
17:08:38 [Debug] Reading config from file system. Looking for .frogbot/frogbot-config.yml
17:08:38 [Debug] frogbot-config.yml found in /opt/agt/_work/1/s/.frogbot/frogbot-config.yml
17:08:38 [Debug] The content of frogbot-config.yml that will be used is:
- params:
    git:
      repoName: CockpIT-front
      branches:
        - master

- params:
    git:
      repoName: CockpIT-back
      branches:
        - master
17:08:38 [Debug] Locking config file to run config AddOrEdit command.
17:08:38 [Debug] Sending HTTP HEAD request to: 'https://github.com/jfrog/frogbot'
17:08:38 [Debug] Creating lock in: /tmp/jfrog.cli.temp.-1706371718-4135036626/locks/config
17:08:38 [Warn] couldn't extract payload from Access Token.
The provided access token is not a valid JWT, probably a reference token.
Some package managers only support basic authentication which requires also a username.
If you plan to work with one of those package managers, please provide a username.
17:08:38 [Debug] Releasing lock: /tmp/jfrog.cli.temp.-1706371718-4135036626/locks/config/jfrog-cli.conf.lck.398.1706371718273425349
17:08:38 [Debug] Config AddOrEdit command completed successfully. config file is released.
17:08:38 [Debug] Usage Report: Sending info...
17:08:38 [Info] Running Frogbot "scan-multiple-repositories" command
17:08:38 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/system/version
17:08:38 [Debug] Sending HTTP POST request to: https://usage-ecosystem.jfrog.io/api/usage/report
17:08:38 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/artifactory/api/system/version
17:08:38 [Debug] Artifactory response: 200 
17:08:38 [Debug] JFrog Artifactory version is: 7.55.10
17:08:38 [Debug] Sending HTTP POST request to: https://<artifactory-server-url>/artifactory/api/system/usage
17:08:38 [Debug] JFrog Xray version is: 3.71.6
17:09:08 [Debug] Can't check access to 'https://github.com/jfrog/frogbot', error while sending request:
Head "https://github.com/jfrog/frogbot": dial tcp 140.82.121.4:443: i/o timeout
17:09:08 [Debug] Setting timeout for go-git to 120 seconds ...
17:09:08 [Debug] Created temp working directory: /tmp/jfrog.cli.temp.-1706371748-2178551145
17:09:08 [Debug] Running git clone https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot (master branch)...
17:09:08 [Debug] Project cloned from https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot to /tmp/jfrog.cli.temp.-1706371748-2178551145
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/system/version
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/entitlements/feature/contextual_analysis
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/HEAD' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/config' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/index' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/info' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.idx' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/heads' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/heads/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/remotes' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/remotes/origin' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/remotes/origin/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/tags' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/shallow' is excluded
17:09:08 [Info] Couldn't determine a package manager or build tool used by this project. Skipping the SCA scan...
17:09:08 [Info] Xray scan completed
17:09:08 [Info] Didn't find vulnerable dependencies with existing fix versions for CockpIT-front
17:09:08 [Debug] Setting timeout for go-git to 120 seconds ...
17:09:08 [Debug] Created temp working directory: /tmp/jfrog.cli.temp.-1706371748-1524852656
17:09:08 [Debug] Running git clone https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot (master branch)...
17:09:08 [Debug] Project cloned from https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot to /tmp/jfrog.cli.temp.-1706371748-1524852656
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/system/version
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/entitlements/feature/contextual_analysis
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/HEAD' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/config' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/index' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/info' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.idx' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/heads' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/heads/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/remotes' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/remotes/origin' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/remotes/origin/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/tags' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/shallow' is excluded
17:09:08 [Info] Couldn't determine a package manager or build tool used by this project. Skipping the SCA scan...
17:09:08 [Info] Xray scan completed
17:09:08 [Info] Didn't find vulnerable dependencies with existing fix versions for CockpIT-back
17:09:08 [Info] Frogbot "scan-multiple-repositories" command finished successfully
##[section]Finishing: Download and Run Frogbot

Reproduction steps

No response

Expected behavior

Each repo declared in the frogbot-config.yml file should be cloned and scan instead of the repo containing the pipeline and config.

JFrog Frogbot version

2.19.8

Package manager info

maven/npm but not relevant

Git provider

Azure DevOps

JFrog Frogbot configuration yaml file

frogbot.yml (pipeline)

pr: none
trigger: none

pool: Linux-Build

variables:
  JF_GIT_PROJECT: $(System.TeamProject)
  JF_GIT_REPO: $(Build.Repository.Name)
  JF_GIT_API_ENDPOINT: $(System.CollectionUri)
  JF_GIT_BASE_BRANCH: $(Build.SourceBranchName)
  JF_GIT_OWNER: $(System.TeamProject)
  JF_GIT_PROVIDER: 'azureRepos'

jobs:
  - job:
    displayName: "Frogbot Scan Repository and Fix"
    steps:
       - task: CmdLine@2
         displayName: 'Download and Run Frogbot'
         env:
            JF_URL: $(JF_URL)
            JF_ACCESS_TOKEN: $(JF_ACCESS_TOKEN)
            JF_GIT_TOKEN: $(System.AccessToken)
            JF_RELEASES_REPO: "frogbot-generic-external"
            JFROG_CLI_LOG_LEVEL: "DEBUG"
         inputs:
            script: |
               getFrogbotScriptPath=$(if [ -z "$JF_RELEASES_REPO" ]; then echo "https://releases.jfrog.io"; else echo "${JF_URL}/artifactory/${JF_RELEASES_REPO}"; fi)
               curl -fLg "$getFrogbotScriptPath/artifactory/frogbot/v2/[RELEASE]/getFrogbot.sh" --header "X-JFrog-Art-Api: $JF_ACCESS_TOKEN" | sh
               ./frogbot scan-multiple-repositories

frogbot-config.yml

- params:
    git:
      repoName: CockpIT-front
      branches:
        - master

- params:
    git:
      repoName: CockpIT-back
      branches:
        - master

Operating system type and version

RHEL 8

JFrog Xray version

3.71.6

The text was updated successfully, but these errors were encountered:

eranturgeman · 2024-01-28T08:20:41Z

Hello @anael-l, thank you for using Frogbot!
In your frogbot-config.yml, you've configured 'params' for both CockpIT-front and CockpIT-back. Before delving into the issue, please try using only one set of 'params' and specify the required working directories for scanning under params/projects/workingDirs. I want to eliminate the possibility that you initiated two separate scans. Please refer to our documentation and the frogbot-config.yml schema to configure it correctly.

anael-l · 2024-01-28T11:01:45Z

Hello @eranturgeman,
My goal IS to launch two scans of two different git repositories.
I've followed this doc: https://docs.jfrog-applications.jfrog.io/jfrog-applications/frogbot/setup-frogbot/frogbot-configuration#can-one-frogbot-config.yml-file-be-used-for-multiple-git-repositories
To setup one central frogbot config, to scans multiple other repositories that are in the same organization.
Isn't what the scan-multiple-repositories command is for ?

eranturgeman · 2024-01-29T08:56:19Z

@anael-l You are correct this is what it suppose to do.
Thank you for the reporting the issue.
Our team will look into it and we will keep you updated here

canberkkurtulmus · 2024-10-09T14:31:50Z

Hello @eranturgeman I guess the same issue continues on version 2.21.13, am I right or is there any solutions for that?

anael-l added the bug Something isn't working label Jan 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scan-multiple-repositories always clones source repo on Azure DevOps #629

scan-multiple-repositories always clones source repo on Azure DevOps #629

anael-l commented Jan 27, 2024

eranturgeman commented Jan 28, 2024

anael-l commented Jan 28, 2024

eranturgeman commented Jan 29, 2024

canberkkurtulmus commented Oct 9, 2024

scan-multiple-repositories always clones source repo on Azure DevOps #629

scan-multiple-repositories always clones source repo on Azure DevOps #629

Comments

anael-l commented Jan 27, 2024

Describe the bug

Current behavior

Reproduction steps

Expected behavior

JFrog Frogbot version

Package manager info

Git provider

JFrog Frogbot configuration yaml file

Operating system type and version

JFrog Xray version

eranturgeman commented Jan 28, 2024

anael-l commented Jan 28, 2024

eranturgeman commented Jan 29, 2024

canberkkurtulmus commented Oct 9, 2024