Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-64383] combined refrepo became our bottleneck, support a fanout location too #644

Open
wants to merge 147 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
147 commits
Select commit Hold shift + click to select a range
8f5ec33
Introduce support for reference repository paths ending with /${GIT_U…
jimklimov Dec 7, 2020
022bbc6
Refactor: move findParameterizedReferenceRepository() into LegacyComp…
jimklimov Dec 8, 2020
6210adb
Add isParameterizedReferenceRepository() and support for parameterize…
jimklimov Dec 8, 2020
06d4859
Handle the possibility that the URL for parameterized checkout is nul…
jimklimov Dec 8, 2020
b5a6a7b
LegacyCompatibleGitAPIImpl.java: document params for new routines
jimklimov Dec 8, 2020
f90bdca
LegacyCompatibleGitAPIImpl.java: refactor urlNormalized generation in…
jimklimov Dec 8, 2020
e09eff1
LegacyCompatibleGitAPIImpl.java: introduce support for .../${GIT_URL_…
jimklimov Dec 8, 2020
17010ca
WorkspaceWithRepo.java (test): extend localMirror() with customizable…
jimklimov Dec 8, 2020
72a4a05
GitClientCloneTest.java: add test_clone_reference_parameterized_sha25…
jimklimov Dec 8, 2020
c6c010f
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Dec 9, 2020
bbedefd
Fix referencePath getName() => getPath()
jimklimov Dec 9, 2020
5306163
Update tests for fRefrepoBase being under target/ dir from CWD of the…
jimklimov Dec 9, 2020
bc7e198
GitClientCloneTest: check if refrepo dir exists already
jimklimov Dec 9, 2020
3ff19df
WorkspaceWithRepo.java: localMirror(): normalize() the returned pathn…
jimklimov Dec 10, 2020
a509386
GitClientCloneTest.java: extend API choices for assertAlternateFilePo…
jimklimov Dec 10, 2020
603e390
GitClientCloneTest.java: fix test_clone_reference_parameterized_sha25…
jimklimov Dec 10, 2020
66e23a1
LegacyCompatibleGitAPIImpl.java CliGitAPIImpl.java JGitAPIImpl.java: …
jimklimov Dec 10, 2020
5af4400
LegacyCompatibleGitAPIImpl.java: add variants of findParameterizedRef…
jimklimov Dec 10, 2020
509c5ec
[TMP] LegacyCompatibleGitAPIImpl.java: trace findParameterizedReferen…
jimklimov Dec 10, 2020
0b0fcbf
[TMP] WorkspaceWithRepo.java (test): trace localMirror() processing t…
jimklimov Dec 10, 2020
6a9c79d
LegacyCompatibleGitAPIImpl.java: refactor to use referenceExpanded to…
jimklimov Dec 10, 2020
2641971
LegacyCompatibleGitAPIImpl.java: add GIT_URL_FALLBACK and GIT_URL_SHA…
jimklimov Dec 10, 2020
c10231c
GitClientCloneTest.java: add cases for parameterized reference repo w…
jimklimov Dec 10, 2020
2016719
LegacyCompatibleGitAPIImpl.java: refactor normalizeGitUrl()
jimklimov Dec 10, 2020
71c0850
[WIP] LegacyCompatibleGitAPIImpl.java: skeleton for GIT_SUBMODULES an…
jimklimov Dec 10, 2020
ab04ae7
Introduce getRemoteUrls() and getRemotePushUrls()
jimklimov Dec 11, 2020
179f5c1
LegacyCompatibleGitAPIImpl.java: normalizeGitUrl(): revise for local …
jimklimov Dec 11, 2020
7c0ad4f
[WIP] LegacyCompatibleGitAPIImpl.java: skeleton for GIT_SUBMODULES an…
jimklimov Dec 11, 2020
ead03a6
CliGitAPIImpl.java: extend getRemote{,Push}Urls() with filters for AC…
jimklimov Dec 11, 2020
5fde3c9
GitClientCloneTest.java: if we expect particular wsRefrepo spelling, …
jimklimov Dec 11, 2020
6c7374c
GitClientCloneTest.java: for fallback case, expect wsRefrepoBase itself
jimklimov Dec 11, 2020
a2cf84f
GitClientCloneTest.java: use the LegacyCompatibleGitAPIImpl.normalize…
jimklimov Dec 11, 2020
823bd0f
GitClientCloneTest.java: for fallback case, expect wsRefrepoBase itse…
jimklimov Dec 11, 2020
0e5e0cd
LegacyCompatibleGitAPIImpl.java: skeleton for getSubmodulesUrls() as …
jimklimov Dec 12, 2020
172e40e
LegacyCompatibleGitAPIImpl.java: update comments for ideas about subm…
jimklimov Dec 14, 2020
d6cebe8
LegacyCompatibleGitAPIImpl.java: getSubmodulesUrls(): implement the s…
jimklimov Dec 14, 2020
864e34f
LegacyCompatibleGitAPIImpl.java: move arrDirNames declaration higher …
jimklimov Jan 10, 2021
55fda14
LegacyCompatibleGitAPIImpl.java: update commented thoughts
jimklimov Jan 10, 2021
3e81723
LegacyCompatibleGitAPIImpl.java: prepare to parse submodule workspaces
jimklimov Jan 10, 2021
ff65341
LegacyCompatibleGitAPIImpl.java: uncomment needle tracking in a loop …
jimklimov Jan 10, 2021
122746f
LegacyCompatibleGitAPIImpl.java: fix @param usage
jimklimov Jan 10, 2021
bf978ee
LegacyCompatibleGitAPIImpl.java: implement handling output of getSubm…
jimklimov Jan 10, 2021
aacc0cd
LegacyCompatibleGitAPIImpl.java: getSubmodulesUrls() / findParameteri…
jimklimov Jan 10, 2021
1a8306a
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Jan 10, 2021
769a2d4
LegacyCompatibleGitAPIImpl.java: log-tracing for getSubmodulesUrls()
jimklimov Jan 11, 2021
654362a
LegacyCompatibleGitAPIImpl.java: for now look in subdirs regardless o…
jimklimov Jan 11, 2021
5b6ff6d
LegacyCompatibleGitAPIImpl.java: comment updated
jimklimov Jan 11, 2021
8d8514c
LegacyCompatibleGitAPIImpl.java: rename getObjectPath() to less misle…
jimklimov Jan 11, 2021
925eaba
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
a103bdd
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
aaaef9c
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
a77dab9
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
d664eb3
LegacyCompatibleGitAPIImpl.java: pass not-normalized URL from findPar…
jimklimov Jan 11, 2021
e29d947
LegacyCompatibleGitAPIImpl.java: check also not-normalized URL in fin…
jimklimov Jan 11, 2021
1816632
LegacyCompatibleGitAPIImpl.java: only add ".git" to reference if it i…
jimklimov Jan 11, 2021
8b693fd
LegacyCompatibleGitAPIImpl.java: fix tracing into private LOGGER inst…
jimklimov Jan 11, 2021
e49b985
LegacyCompatibleGitAPIImpl.java: trace looking for the needle in arrD…
jimklimov Jan 11, 2021
ab29c68
LegacyCompatibleGitAPIImpl.java: comment reshuffle
jimklimov Jan 11, 2021
66798fd
LegacyCompatibleGitAPIImpl.java: refactor findParameterizedReferenceR…
jimklimov Jan 11, 2021
776a249
LegacyCompatibleGitAPIImpl.java: trace which sub-git workspaces we do…
jimklimov Jan 11, 2021
094de4d
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to explicitl…
jimklimov Jan 11, 2021
d30de21
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to look in "…
jimklimov Jan 11, 2021
7ed0f0f
GitClient and its implementations: extend API with a newGit(somedir) …
jimklimov Jan 12, 2021
14cf345
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to look in r…
jimklimov Jan 12, 2021
2b61767
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to return fA…
jimklimov Jan 12, 2021
1c9a8f5
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to track rel…
jimklimov Jan 12, 2021
a29b00c
LegacyCompatibleGitAPIImpl.java: clearer trace messages in getSubmodu…
jimklimov Jan 12, 2021
25bef67
LegacyCompatibleGitAPIImpl.java: track referenceBaseDirAbs to compare…
jimklimov Jan 12, 2021
39dd9b4
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to return fA…
jimklimov Jan 12, 2021
c349b25
LegacyCompatibleGitAPIImpl.java: comment the ends of long logical blocks
jimklimov Jan 12, 2021
6a7bb18
LegacyCompatibleGitAPIImpl.java: use absolute pathnames "natively" in…
jimklimov Jan 12, 2021
f475606
LegacyCompatibleGitAPIImpl.java: do not need to check bare-mode dirs …
jimklimov Jan 12, 2021
2ddd7df
LegacyCompatibleGitAPIImpl.java: check basename-like dirs for being g…
jimklimov Jan 12, 2021
721d791
LegacyCompatibleGitAPIImpl.java: check the parent refrepo dir for hos…
jimklimov Jan 12, 2021
69f5b59
LegacyCompatibleGitAPIImpl.java: trace what getRemoteUrls() gave us
jimklimov Jan 12, 2021
9c48cb6
LegacyCompatibleGitAPIImpl.java: leave a TODO block for checking subm…
jimklimov Jan 12, 2021
f465258
LegacyCompatibleGitAPIImpl.java: trace mis-processing an assumed git …
jimklimov Jan 12, 2021
f6e756a
LegacyCompatibleGitAPIImpl.java: comment about abs dirnames in second…
jimklimov Jan 12, 2021
ce1827e
LegacyCompatibleGitAPIImpl.java: fix GitClient for abs dirnames in se…
jimklimov Jan 12, 2021
c5e863f
LegacyCompatibleGitAPIImpl.java: Log into build console the start and…
jimklimov Jan 12, 2021
d0bf23c
LegacyCompatibleGitAPIImpl.java: Refactor a useless needle to be null…
jimklimov Jan 12, 2021
0b761da
LegacyCompatibleGitAPIImpl.java: Refactor getSubmodulesUrls() to unif…
jimklimov Jan 12, 2021
6046f4b
LegacyCompatibleGitAPIImpl.java: Refactor getSubmodulesUrls() to name…
jimklimov Jan 12, 2021
3f7b4e4
LegacyCompatibleGitAPIImpl.java: re-use arrDirnames in getSubmodulesU…
jimklimov Jan 12, 2021
61a7d0b
LegacyCompatibleGitAPIImpl.java: comment the plan about submodule sea…
jimklimov Jan 12, 2021
b91e00b
LegacyCompatibleGitAPIImpl.java: update comments around shorter getSu…
jimklimov Jan 13, 2021
65684f3
LegacyCompatibleGitAPIImpl.java: only track absolute dirname values i…
jimklimov Jan 13, 2021
0c99c95
LegacyCompatibleGitAPIImpl.java: comment the plan for submodule handling
jimklimov Jan 13, 2021
327cb48
LegacyCompatibleGitAPIImpl.java: finally, do recursion in getSubmodul…
jimklimov Jan 13, 2021
45d268c
LegacyCompatibleGitAPIImpl.java: check not only for ".git" but also "…
jimklimov Jan 16, 2021
0413d95
LegacyCompatibleGitAPIImpl.java: for GIT_SUBMODULES mode search, if t…
jimklimov Jan 16, 2021
fad3727
LegacyCompatibleGitAPIImpl.java: implement a use-case of ".git" file …
jimklimov Jan 16, 2021
d37fe49
LegacyCompatibleGitAPIImpl.java: refactor getSubmodulesUrls() to use …
jimklimov Jan 16, 2021
6afacda
LegacyCompatibleGitAPIImpl.java: refactor getSubmodulesUrls() to retu…
jimklimov Jan 16, 2021
b278523
LegacyCompatibleGitAPIImpl.java: add logging for getObjectsFile()
jimklimov Jan 17, 2021
4468443
LegacyCompatibleGitAPIImpl.java: in getObjectsFile(), comment why it …
jimklimov Jan 17, 2021
0edcc0c
JGitAPIImpl.java / CliGitAPIImpl.java: inform the build console log r…
jimklimov Jan 17, 2021
0767c2a
LegacyCompatibleGitAPIImpl.java: in getObjectsFile() method comment, …
jimklimov Jan 17, 2021
03b3056
LegacyCompatibleGitAPIImpl.java: little clean-up in getObjectsFile()
jimklimov Jan 17, 2021
fd88900
LegacyCompatibleGitAPIImpl.java: referenceBaseDirAbs in getSubmodules…
jimklimov Jan 17, 2021
d72999b
LegacyCompatibleGitAPIImpl.java: suggest a selection of results[] *si…
jimklimov Jan 17, 2021
f1f478a
LegacyCompatibleGitAPIImpl.java: mark helpers as public static
jimklimov Jan 18, 2021
91ed630
LegacyCompatibleGitAPIImpl.java: check if referenceBaseDir is usable …
jimklimov Jan 18, 2021
de98cd1
GitClientCloneTest.java: reword original test for GIT_URL into checki…
jimklimov Jan 31, 2021
aa9eaad
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Jan 31, 2021
82a43eb
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Jan 31, 2021
23745a8
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: listFiles()…
jimklimov Jan 31, 2021
f51e60f
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: getSubmodul…
jimklimov Jan 31, 2021
a5fdbd3
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: have non-ze…
jimklimov Jan 31, 2021
4341cd2
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: "new File()…
jimklimov Jan 31, 2021
4cd45c8
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: use a file …
jimklimov Jan 31, 2021
fd293d7
LegacyCompatibleGitAPIImpl.java: normalizeUrl() failed with full path…
jimklimov Feb 22, 2021
c45cedd
LegacyCompatibleGitAPIImpl.java: normalizeUrl(): @SuppressFBWarnings …
jimklimov Feb 22, 2021
be57864
CliGitAPIImpl.java: launchCommandIn(): report workDir when throwing e…
jimklimov Jan 26, 2022
8f843d4
LegacyCompatibleGitAPIImpl.java: getSubmodulesUrls(): do not fail che…
jimklimov Jan 26, 2022
4ee6935
JGitAPIImpl.java + CliGitAPIImpl.java: report "reference" pathname th…
jimklimov Jan 26, 2022
e726d40
Merge branch 'master' as of 2022-08-01 into refrepo-args
jimklimov Aug 1, 2022
558d076
[JENKINS-69193] GitAPITestCase: test_submodule_update_with_error(): r…
jimklimov Aug 1, 2022
4402820
[JENKINS-69193] GitAPITestUpdateCliGit: rely less on strict wording o…
jimklimov Aug 10, 2022
5d97abe
Merge branch 'JENKINS-69193-bis' into refrepo-args
jimklimov Aug 10, 2022
4222f4e
Merge branch 'master' into refrepo-args
MarkEWaite Sep 5, 2022
63ca351
Merge branch 'master' into refrepo-args
MarkEWaite Sep 14, 2022
2544b99
Merge branch 'master' into refrepo-args
jimklimov Dec 21, 2022
059f7a2
Merge branch 'master' into refrepo-args
MarkEWaite Apr 11, 2023
7c28d4d
Merge branch 'spotless' into refrepo-args
MarkEWaite Apr 11, 2023
c882050
Format source changes
MarkEWaite Apr 11, 2023
9863085
Fix merge mistake
MarkEWaite Apr 11, 2023
8334390
Merge remote-tracking branch 'upstream/master' into refrepo-args
jimklimov Jun 14, 2023
eeb47b4
GitClient.java: minor fix to javadoc for subGit()
jimklimov Jun 14, 2023
9f162aa
Reconcile code style with changes in the PR #644 (added by @markewait…
jimklimov Jun 14, 2023
d3a6d56
Mark @Override methods like in master branch
jimklimov Jun 14, 2023
6840c75
Change PrintWriter use of UTF8 like in master branch
jimklimov Jun 14, 2023
d8ddcdb
GitClient.java: minor fix to javadoc for newGit()
jimklimov Jun 14, 2023
896c2c6
LegacyCompatibleGitAPIImpl: import DigestUtils to use by short reference
jimklimov Jun 14, 2023
62355d5
GitClientCloneTest: drop use of FileUtils.readFileToString() like in …
jimklimov Jun 14, 2023
2c10b59
spotless is clueless
jimklimov Jun 14, 2023
59e7474
Merge branch 'master' into refrepo-args
MarkEWaite Sep 13, 2023
4e1f3ab
Remove redundant null check
MarkEWaite Sep 13, 2023
f844f23
Resolve spotbugs warnings for toString() and unread vars
MarkEWaite Sep 13, 2023
43c38ba
Resolve spotbugs warnings on toLowerCase
MarkEWaite Sep 13, 2023
65b55c5
Assure stream is closed on exception
MarkEWaite Sep 13, 2023
9da15fa
Merge branch 'master' into refrepo-args
MarkEWaite Jan 17, 2024
1f379a7
Merge branch 'master' into refrepo-args
MarkEWaite Mar 9, 2024
cedfd7f
Merge remote-tracking branch 'upstream/master' into refrepo-args
jimklimov Jun 14, 2024
b2d7fef
Merge remote-tracking branch 'upstream/master' as of 2024-10-31 into …
jimklimov Oct 31, 2024
9dc373e
Merge remote-tracking branch 'upstream/master' as of 2025-01-16 into …
jimklimov Jan 16, 2025
d2fd099
Merge remote-tracking branch 'upstream/master' into refrepo-args
jimklimov Jan 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/main/java/hudson/plugins/git/GitAPI.java
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,12 @@
return Git.USE_CLI ? super.subGit(subdir) : jgit.subGit(subdir);
}

/** {@inheritDoc} */
@Override
public GitClient newGit(String somedir) {
return Git.USE_CLI ? super.newGit(somedir) : jgit.newGit(somedir);

Check warning on line 181 in src/main/java/hudson/plugins/git/GitAPI.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Not covered line

Line 181 is not covered by tests
}

/** {@inheritDoc} */
@Override
public void setRemoteUrl(String name, String url) throws GitException, InterruptedException {
Expand Down
192 changes: 157 additions & 35 deletions src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,12 @@
return new CliGitAPIImpl(gitExe, new File(workspace, subdir), listener, environment);
}

/** {@inheritDoc} */
@Override
public GitClient newGit(String somedir) {
return new CliGitAPIImpl(gitExe, new File(somedir), listener, environment);

Check warning on line 357 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Not covered line

Line 357 is not covered by tests
}

/**
* Initialize an empty repository for further git operations.
*
Expand Down Expand Up @@ -837,670 +843,697 @@
}

if (reference != null && !reference.isEmpty()) {
File referencePath = new File(reference);
if (!referencePath.exists()) {
listener.getLogger().println("[WARNING] Reference path does not exist: " + reference);
} else if (!referencePath.isDirectory()) {
listener.getLogger().println("[WARNING] Reference path is not a directory: " + reference);
if (isParameterizedReferenceRepository(reference)) {
// LegacyCompatibleGitAPIImpl.java has a logging trace, but not into build console via listener
listener.getLogger()
.println("[INFO] The git reference repository path '" + reference + "' "
+ "is parameterized, it may take a few git queries logged "
+ "below to resolve it into a particular directory name");
}
File referencePath = findParameterizedReferenceRepository(reference, url);
if (referencePath == null) {

Check warning on line 854 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Partially covered line

Line 854 is only partially covered, one branch is missing
listener.getLogger()
.println("[ERROR] Could not make File object from reference path, "

Check warning on line 856 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Not covered lines

Lines 855-856 are not covered by tests
+ "skipping its use: " + reference);
} else {
// reference path can either be a normal or a base repository
File objectsPath = new File(referencePath, ".git/objects");
if (!objectsPath.isDirectory()) {
// reference path is bare repo
objectsPath = new File(referencePath, "objects");
if (!referencePath.getPath().equals(reference)) {
// Note: both these logs are needed, they are used in selftest
String msg = "Parameterized reference path ";
msg += "'" + reference + "'";
msg += " replaced with: ";
msg += "'" + referencePath.getPath() + "'";
if (referencePath.exists()) {
listener.getLogger().println("[WARNING] " + msg);
} else {
listener.getLogger().println("[WARNING] " + msg + " does not exist");
}
reference = referencePath.getPath();
}
if (!objectsPath.isDirectory()) {
listener.getLogger()
.println(
"[WARNING] Reference path does not contain an objects directory (not a git repo?): "
+ objectsPath);

if (!referencePath.exists()) {
listener.getLogger().println("[WARNING] Reference path does not exist: " + reference);
} else if (!referencePath.isDirectory()) {
listener.getLogger().println("[WARNING] Reference path is not a directory: " + reference);
} else {
File alternates = new File(workspace, ".git/objects/info/alternates");
try (PrintWriter w = new PrintWriter(alternates, Charset.defaultCharset())) {
String absoluteReference =
objectsPath.getAbsolutePath().replace('\\', '/');
listener.getLogger().println("Using reference repository: " + reference);
// git implementations on windows also use
w.print(absoluteReference);
} catch (IOException e) {
listener.error("Failed to setup reference");
File objectsPath = getObjectsFile(referencePath);
if (objectsPath == null || !objectsPath.isDirectory()) {

Check warning on line 879 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Partially covered line

Line 879 is only partially covered, 2 branches are missing
listener.getLogger()
.println("[WARNING] Reference path does not contain an objects directory "

Check warning on line 881 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Not covered line

Line 881 is not covered by tests
+ "(not a git repo?): " + objectsPath);
} else {
// Go behind git's back to write a meta file in new workspace
File alternates = new File(workspace, ".git/objects/info/alternates");
try (PrintWriter w = new PrintWriter(
alternates, Charset.defaultCharset().toString())) {
String absoluteReference =
objectsPath.getAbsolutePath().replace('\\', '/');
listener.getLogger().println("Using reference repository: " + reference);
// git implementations on windows also use
w.print(absoluteReference);
} catch (UnsupportedEncodingException ex) {
listener.error("Default character set is an unsupported encoding");
} catch (FileNotFoundException e) {
listener.error("Failed to setup reference");
}
}
}
}
}

if (refspecs == null) {
refspecs = Collections.singletonList(new RefSpec("+refs/heads/*:refs/remotes/" + origin + "/*"));
}
fetch_().from(urIish, refspecs)
.shallow(shallow)
.depth(depth)
.timeout(timeout)
.tags(tags)
.execute();
setRemoteUrl(origin, url);
for (RefSpec refSpec : refspecs) {
launchCommand("config", "--add", "remote." + origin + ".fetch", refSpec.toString());
}
}
};
}

/**
* merge.
*
* @return a {@link org.jenkinsci.plugins.gitclient.MergeCommand} object.
*/
@Override
public MergeCommand merge() {
return new MergeCommand() {
private ObjectId rev;
private String comment;
private String strategy;
private String fastForwardMode;
private boolean squash;
private boolean commit = true;

@Override
public MergeCommand setRevisionToMerge(ObjectId rev) {
this.rev = rev;
return this;
}

@Override
public MergeCommand setStrategy(MergeCommand.Strategy strategy) {
this.strategy = strategy.toString();
return this;
}

@Override
public MergeCommand setGitPluginFastForwardMode(MergeCommand.GitPluginFastForwardMode fastForwardMode) {
this.fastForwardMode = fastForwardMode.toString();
return this;
}

@Override
public MergeCommand setSquash(boolean squash) {
this.squash = squash;
return this;
}

@Override
public MergeCommand setMessage(String comment) {
this.comment = comment;
return this;
}

@Override
public MergeCommand setCommit(boolean commit) {
this.commit = commit;
return this;
}

@Override
public void execute() throws GitException, InterruptedException {
ArgumentListBuilder args = new ArgumentListBuilder();
args.add("merge");
if (squash) {
args.add("--squash");
}

if (!commit) {
args.add("--no-commit");
}

if (comment != null && !comment.isEmpty()) {
args.add("-m");
args.add(comment);
}

if (strategy != null
&& !strategy.isEmpty()
&& !strategy.equals(MergeCommand.Strategy.DEFAULT.toString())) {
args.add("-s");
if (strategy.equals(MergeCommand.Strategy.RECURSIVE_THEIRS.toString())) {
args.add("recursive");
args.add("--strategy-option");
args.add("theirs");
} else {
args.add(strategy);
}
}

args.add(fastForwardMode);
if (rev == null) {
throw new GitException("MergeCommand requires a revision to merge");
}
args.add(rev.name());

/* See JENKINS-45228 */
/* Git merge requires authentication in LFS merges, plugin does not authenticate the git merge command */
String repoUrl = null;
try {
String defaultRemote = getDefaultRemote();
if (defaultRemote != null && !defaultRemote.isEmpty()) {
repoUrl = getRemoteUrl(defaultRemote);
}
} catch (GitException e) {
/* Nothing to do, just keeping repoUrl = null */
}

if (repoUrl != null) {
StandardCredentials cred = credentials.get(repoUrl);
if (cred == null) {
cred = defaultCredentials;
}
launchCommandWithCredentials(args, workspace, cred, repoUrl);
} else {
/* Merge is allowed even if a remote URL is not defined. */
/* If there is no remote URL, there is no need to use credentials in the merge. */
launchCommand(args);
}
}
};
}

/**
* rebase.
*
* @return a {@link org.jenkinsci.plugins.gitclient.RebaseCommand} object.
*/
@Override
public RebaseCommand rebase() {
return new RebaseCommand() {
private String upstream;

@Override
public RebaseCommand setUpstream(String upstream) {
this.upstream = upstream;
return this;
}

@Override
public void execute() throws GitException, InterruptedException {
try {
ArgumentListBuilder args = new ArgumentListBuilder();
args.add("rebase");
args.add(upstream);
launchCommand(args);
} catch (GitException e) {
launchCommand("rebase", "--abort");
throw new GitException("Could not rebase " + upstream, e);
}
}
};
}

/**
* init_.
*
* @return a {@link org.jenkinsci.plugins.gitclient.InitCommand} object.
*/
@Override
public InitCommand init_() {
return new InitCommand() {

private String workspace;
private boolean bare;

@Override
public InitCommand workspace(String workspace) {
this.workspace = workspace;
return this;
}

@Override
public InitCommand bare(boolean bare) {
this.bare = bare;
return this;
}

@Override
public void execute() throws GitException, InterruptedException {
/* Match JGit - create directory if it does not exist */
/* Multi-branch pipeline assumes init() creates directory */
File workspaceDir = new File(workspace);
if (!workspaceDir.exists()) {
boolean ok = workspaceDir.mkdirs();
if (!ok && !workspaceDir.exists()) {
throw new GitException("Could not create directory '" + workspaceDir.getAbsolutePath() + "'");
}
}

ArgumentListBuilder args = new ArgumentListBuilder();
args.add("init", workspace);

if (bare) {
args.add("--bare");
}

warnIfWindowsTemporaryDirNameHasSpaces();

try {
launchCommand(args);
} catch (GitException e) {
throw new GitException("Could not init " + workspace, e);
}
}
};
}

/**
* Remove untracked files and directories, including files listed
* in the ignore rules.
*
* @param cleanSubmodule flag to add extra -f
* @throws hudson.plugins.git.GitException if underlying git operation fails.
* @throws java.lang.InterruptedException if interrupted.
*/
@Override
public void clean(boolean cleanSubmodule) throws GitException, InterruptedException {
reset(true);
String cmd = "-fdx";
if (cleanSubmodule) {
cmd = "-ffdx";
}

launchCommand("clean", cmd);
}

/**
* Remove untracked files and directories, including files listed
* in the ignore rules.
*
* @throws hudson.plugins.git.GitException if underlying git operation fails.
* @throws java.lang.InterruptedException if interrupted.
*/
@Override
public void clean() throws GitException, InterruptedException {
this.clean(false);
}

/** {@inheritDoc} */
@Override
public ObjectId revParse(String revName) throws GitException, InterruptedException {

String arg = sanitize(revName + "^{commit}");
String result = launchCommand("rev-parse", arg);
String line = StringUtils.trimToNull(result);
if (line == null) {
throw new GitException("rev-parse no content returned for " + revName);
}
return ObjectId.fromString(line);
}

/**
* On Windows command prompt, '^' is an escape character (https://en.wikipedia.org/wiki/Escape_character#Windows_Command_Prompt)
* This isn't a problem if 'git' we are executing is git.exe, because '^' is a special character only for the command processor,
* but if 'git' we are executing is git.cmd (which is the case of msysgit), then the arguments we pass in here ends up getting
* processed by the command processor, and so 'xyz^{commit}' becomes 'xyz{commit}' and fails.
* <p>
* We work around this problem by surrounding this with double-quote on Windows.
* Unlike POSIX, where the arguments of a process is modeled as String[], Win32 API models the
* arguments of a process as a single string (see CreateProcess). When we surround one argument with a quote,
* java.lang.ProcessImpl on Windows preserve as-is and generate a single string like the following to pass to CreateProcess:
* <pre>
* git rev-parse "tag^{commit}"
* </pre>
* If we invoke git.exe, MSVCRT startup code in git.exe will handle escape and executes it as we expect.
* If we invoke git.cmd, cmd.exe will not eats this ^ that's in double-quote. So it works on both cases.
* <p>
* Note that this is a borderline-buggy behaviour arguably. If I were implementing ProcessImpl for Windows
* in JDK, My passing a string with double-quotes around it to be expanded to the following:
* <pre>
* git rev-parse "\"tag^{commit}\""
* </pre>
* So this work around that we are doing for Windows relies on the assumption that Java runtime will not
* change this behaviour.
* <p>
* Also note that on Unix we cannot do this. Similarly, other ways of quoting (like using '^^' instead of '^'
* that you do on interactive command prompt) do not work either, because MSVCRT startup won't handle
* those in the same way cmd.exe does.
*
* See JENKINS-13007 where this blew up on Windows users.
* See https://github.com/msysgit/msysgit/issues/36 where I filed this as a bug to msysgit.
**/
private String sanitize(String arg) {
if (isWindows()) {
arg = '"' + arg + '"';
}
return arg;
}

/**
* validateRevision.
*
* @param revName a {@link java.lang.String} object.
* @return a {@link org.eclipse.jgit.lib.ObjectId} object.
* @throws hudson.plugins.git.GitException if underlying git operation fails.
* @throws java.lang.InterruptedException if interrupted.
*/
public ObjectId validateRevision(String revName) throws GitException, InterruptedException {
String result = launchCommand("rev-parse", "--verify", revName);
String line = StringUtils.trimToNull(result);
if (line == null) {
throw new GitException("null result from rev-parse(" + revName + ")");
}
return ObjectId.fromString(line);
}

/** {@inheritDoc} */
@Override
public String describe(String commitIsh) throws GitException, InterruptedException {
String result = launchCommand("describe", "--tags", commitIsh);
String line = firstLine(result);
if (line == null) {
throw new GitException("null first line from describe(" + commitIsh + ")");
}
return line.trim();
}

/** {@inheritDoc} */
@Override
public void prune(RemoteConfig repository) throws GitException, InterruptedException {
String repoName = repository.getName();
String repoUrl = getRemoteUrl(repoName);
if (repoUrl != null && !repoUrl.isEmpty()) {
ArgumentListBuilder args = new ArgumentListBuilder();
args.add("remote", "prune", repoName);

StandardCredentials cred = credentials.get(repoUrl);
if (cred == null) {
cred = defaultCredentials;
}

try {
launchCommandWithCredentials(args, workspace, cred, new URIish(repoUrl));
} catch (URISyntaxException ex) {
throw new GitException("Invalid URL " + repoUrl, ex);
}
}
}

@SuppressFBWarnings(
value = "RV_DONT_JUST_NULL_CHECK_READLINE",
justification = "Only needs first line, exception if multiple detected")
private @CheckForNull String firstLine(String result) throws GitException {
BufferedReader reader = new BufferedReader(new StringReader(result));
String line;
try {
line = reader.readLine();
if (line == null) {
return null;
}
if (reader.readLine() != null) {
throw new GitException("Result has multiple lines");
}
} catch (IOException e) {
throw new GitException("Error parsing result", e);
}

return line;
}

/**
* changelog.
*
* @return a {@link org.jenkinsci.plugins.gitclient.ChangelogCommand} object.
*/
@Override
public ChangelogCommand changelog() {
return new ChangelogCommand() {

/** Equivalent to the git-log raw format but using ISO 8601 date format - also prevent to depend on git CLI future changes */
public static final String RAW =
"commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %ai%ncommitter %cN <%cE> %ci%n%n%w(0,4,4)%B";

private final List<String> revs = new ArrayList<>();

private Integer n = null;
private Writer out = null;

@Override
public ChangelogCommand excludes(String rev) {
revs.add(sanitize('^' + rev));
return this;
}

@Override
public ChangelogCommand excludes(ObjectId rev) {
return excludes(rev.name());
}

@Override
public ChangelogCommand includes(String rev) {
revs.add(rev);
return this;
}

@Override
public ChangelogCommand includes(ObjectId rev) {
return includes(rev.name());
}

@Override
public ChangelogCommand to(Writer w) {
this.out = w;
return this;
}

@Override
public ChangelogCommand max(int n) {
this.n = n;
return this;
}

@Override
public void abort() {
/* No cleanup needed to abort the CliGitAPIImpl ChangelogCommand */
}

@Override
public void execute() throws GitException, InterruptedException {
ArgumentListBuilder args = new ArgumentListBuilder(gitExe, "whatchanged", "--no-abbrev", "-M");
if (isAtLeastVersion(1, 8, 3, 0)) {
args.add("--format=" + RAW);
} else {
/* Ancient git versions don't support the required format string, use 'raw' and hope it works */
args.add("--format=raw");
}
if (n != null) {
args.add("-n").add(n);
}
for (String rev : this.revs) {
args.add(rev);
}

if (out == null) {
throw new IllegalStateException();
}

// "git whatchanged" std output gives us byte stream of data
// Commit messages in that byte stream are UTF-8 encoded.
// We want to decode bytestream to strings using UTF-8 encoding.
try (WriterOutputStream w = new WriterOutputStream(out, StandardCharsets.UTF_8)) {
if (launcher.launch()
.cmds(args)
.envs(environment)
.stdout(w)
.stderr(listener.getLogger())
.pwd(workspace)
.join()
!= 0) {
throw new GitException("Error: " + args + " in " + workspace);
}
} catch (IOException e) {
throw new GitException("Error: " + args + " in " + workspace, e);
}
}
};
}

/** {@inheritDoc} */
@Override
public List<String> showRevision(ObjectId from, ObjectId to) throws GitException, InterruptedException {
return showRevision(from, to, true);
}

/** {@inheritDoc} */
@Override
public List<String> showRevision(ObjectId from, ObjectId to, Boolean useRawOutput)
throws GitException, InterruptedException {
ArgumentListBuilder args =
new ArgumentListBuilder("log", "--full-history", "--no-abbrev", "--format=raw", "-M", "-m");
if (useRawOutput) {
args.add("--raw");
}

if (from != null) {
args.add(from.name() + ".." + to.name());
} else {
args.add("-1", to.name());
}

StringWriter writer = new StringWriter();
writer.write(launchCommand(args));
return new ArrayList<>(Arrays.asList(writer.toString().split("\\n")));
}

/**
* submoduleInit.
*
* @throws hudson.plugins.git.GitException if underlying git operation fails.
* @throws java.lang.InterruptedException if interrupted.
*/
@Override
public void submoduleInit() throws GitException, InterruptedException {
launchCommand("submodule", "init");
}

/** {@inheritDoc} */
@Override
public void addSubmodule(String remoteURL, String subdir) throws GitException, InterruptedException {
launchCommand("submodule", "add", remoteURL, subdir);
}

/**
* Sync submodule URLs
*
* @throws hudson.plugins.git.GitException if underlying git operation fails.
* @throws java.lang.InterruptedException if interrupted.
*/
@Override
public void submoduleSync() throws GitException, InterruptedException {
// Check if git submodule has sync support.
// Only available in git 1.6.1 and above
launchCommand("submodule", "sync");
}

/**
* Update submodules.
*
* @return a {@link org.jenkinsci.plugins.gitclient.SubmoduleUpdateCommand} object.
*/
@Override
public SubmoduleUpdateCommand submoduleUpdate() {
return new SubmoduleUpdateCommand() {
private boolean recursive = false;
private boolean remoteTracking = false;
private boolean parentCredentials = false;
private boolean shallow = false;
private String ref = null;
private Map<String, String> submodBranch = new HashMap<>();
private Integer timeout;
private Integer depth = 1;
private int threads = 1;

@Override
public SubmoduleUpdateCommand recursive(boolean recursive) {
this.recursive = recursive;
return this;
}

@Override
public SubmoduleUpdateCommand remoteTracking(boolean remoteTracking) {
this.remoteTracking = remoteTracking;
return this;
}

@Override
public SubmoduleUpdateCommand parentCredentials(boolean parentCredentials) {
this.parentCredentials = parentCredentials;
return this;
}

@Override
public SubmoduleUpdateCommand ref(String ref) {
this.ref = ref;
return this;
}

@Override
public SubmoduleUpdateCommand useBranch(String submodule, String branchname) {
this.submodBranch.put(submodule, branchname);
return this;
}

@Override
public SubmoduleUpdateCommand timeout(Integer timeout) {
this.timeout = timeout;
return this;
}

@Override
public SubmoduleUpdateCommand shallow(boolean shallow) {
this.shallow = shallow;
return this;
}

@Override
public SubmoduleUpdateCommand depth(Integer depth) {
this.depth = depth;
return this;
}

@Override
public SubmoduleUpdateCommand threads(int threads) {
this.threads = threads;
return this;
}

/**
* @throws GitException if executing the Git command fails
* @throws InterruptedException if called methods throw same exception
*/
@Override
public void execute() throws GitException, InterruptedException {
// Initialize the submodules to ensure that the git config
// contains the URLs from .gitmodules.
submoduleInit();

ArgumentListBuilder args = new ArgumentListBuilder();
args.add("submodule", "update");
if (recursive) {
args.add("--init", "--recursive");
}
if (remoteTracking && isAtLeastVersion(1, 8, 2, 0)) {
args.add("--remote");

for (Map.Entry<String, String> entry : submodBranch.entrySet()) {
launchCommand(
"config",
"-f",
".gitmodules",
"submodule." + entry.getKey() + ".branch",
entry.getValue());
}
}
if ((ref != null) && !ref.isEmpty()) {
File referencePath = new File(ref);
if (!referencePath.exists()) {
listener.getLogger().println("[WARNING] Reference path does not exist: " + ref);
} else if (!referencePath.isDirectory()) {
listener.getLogger().println("[WARNING] Reference path is not a directory: " + ref);
} else {
args.add("--reference", ref);
}
if (!isParameterizedReferenceRepository(ref)) {

Check warning on line 1527 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Not covered lines

Lines 893-1527 are not covered by tests
File referencePath = new File(ref);
if (!referencePath.exists()) {
listener.getLogger().println("[WARNING] Reference path does not exist: " + ref);
} else if (!referencePath.isDirectory()) {
listener.getLogger().println("[WARNING] Reference path is not a directory: " + ref);
} else {
args.add("--reference", ref);
}
} // else handled below in per-module loop
}
if (shallow) {
if (depth == null) {
Expand Down Expand Up @@ -1549,9 +1582,34 @@
listener.error("Invalid repository for " + sModuleName);
throw new GitException("Invalid repository for " + sModuleName);
}
String strURIish = urIish.toPrivateString();

if (isParameterizedReferenceRepository(ref)) {

Check warning on line 1587 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Partially covered line

Line 1587 is only partially covered, one branch is missing
File referencePath = findParameterizedReferenceRepository(ref, strURIish);
if (referencePath == null) {
listener.getLogger()
.println("[ERROR] Could not make File object from reference path, "
+ "skipping its use: " + ref);
} else {
String expRef = null;
if (referencePath.getPath().equals(ref)) {
expRef = ref;
} else {
expRef = referencePath.getPath();
expRef += " (expanded from " + ref + ")";
}
if (!referencePath.exists()) {
listener.getLogger().println("[WARNING] Reference path does not exist: " + expRef);
} else if (!referencePath.isDirectory()) {
listener.getLogger().println("[WARNING] Reference path is not a directory: " + expRef);
} else {
args.add("--reference", referencePath.getPath());

Check warning on line 1606 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Not covered lines

Lines 1588-1606 are not covered by tests
}
}
}

// Find credentials for this URL
StandardCredentials cred = credentials.get(urIish.toPrivateString());
StandardCredentials cred = credentials.get(strURIish);
if (parentCredentials) {
String parentUrl = getRemoteUrl(getDefaultRemote());
URIish parentUri = null;
Expand Down Expand Up @@ -1661,6 +1719,62 @@
return StringUtils.trim(firstLine(result));
}

/** {@inheritDoc} */
@Override
public @CheckForNull Map<String, String> getRemoteUrls() throws GitException, InterruptedException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There don't appear to be any tests for this new method. The method was not called from the modified tests in GitClientCloneTest.

I'd like tests that confirm the method is well-behaved. There need to be tests of the method before it is merged.

The tests could be created in GitClientTest or in a new test class created based on GitClientTest. Because GitClientTest is parameterized and iterates over git, jgit, and jgitapache, it increases the coverage without dramatically increasing the amount of test code.

String result = launchCommand("config", "--local", "--list");
Map<String, String> uriNames = new HashMap<>();
for (String line : result.split("\\R+")) {
line = StringUtils.trim(line);
if (!line.startsWith("remote.") || !line.contains(".url=")) {
continue;
}

String remoteName = StringUtils.substringBetween(line, "remote.", ".url=");
String remoteUri = StringUtils.substringAfter(line, ".url=");

// If uri String values end up identical, Map only stores one entry
uriNames.put(remoteUri, remoteName);

try {
URI u = new URI(remoteUri);
uriNames.put(u.toASCIIString(), remoteName);
URI uSafe = new URI(u.getScheme(), u.getHost(), u.getPath(), u.getFragment());
uriNames.put(uSafe.toString(), remoteName);
uriNames.put(uSafe.toASCIIString(), remoteName);
} catch (URISyntaxException ue) {
} // ignore, move along
}
return uriNames;
}

/** {@inheritDoc} */
@Override
public @CheckForNull Map<String, String> getRemotePushUrls() throws GitException, InterruptedException {
String result = launchCommand("config", "--local", "--list");
Map<String, String> uriNames = new HashMap<>();
for (String line : result.split("\\R+")) {
line = StringUtils.trim(line);
if (!line.startsWith("remote.") || !line.contains(".pushurl=")) {
continue;
}

String remoteName = StringUtils.substringBetween(line, "remote.", ".pushurl=");
String remoteUri = StringUtils.substringAfter(line, ".pushurl=");
uriNames.put(remoteUri, remoteName);

try {
URI u = new URI(remoteUri);
uriNames.put(u.toASCIIString(), remoteName);
URI uSafe = new URI(u.getScheme(), u.getHost(), u.getPath(), u.getFragment());
uriNames.put(uSafe.toString(), remoteName);
uriNames.put(uSafe.toASCIIString(), remoteName);
} catch (URISyntaxException ue) {
} // ignore, move along
}
return uriNames;

Check warning on line 1775 in src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Not covered lines

Lines 1725-1775 are not covered by tests
}

/** {@inheritDoc} */
@Override
public void setRemoteUrl(String name, String url) throws GitException, InterruptedException {
Expand Down Expand Up @@ -2849,8 +2963,16 @@
}

if (status != 0) {
throw new GitException("Command \"" + command + "\" returned status code " + status + ":\nstdout: "
+ stdout + "\nstderr: " + stderr);
if (workDir == null)
workDir = java.nio.file.Paths.get(".")
.toAbsolutePath()
.normalize()
.toFile();
throw new GitException("Command \"" + command
+ "\" executed in workdir \"" + workDir.toString()
+ "\" returned status code " + status
+ ":\nstdout: " + stdout
+ "\nstderr: " + stderr);
}

return stdout;
Expand Down
35 changes: 34 additions & 1 deletion src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,28 @@ public interface GitClient {
*/
String getRemoteUrl(String name) throws GitException, InterruptedException;

/**
* getRemoteUrls.
*
* @return a Map where String keys represent URIs (with and
* without passwords, if any; ASCII or not, if
* applicable) for all remotes configured in this
* repository/workspace, and values represent names.
* There may be several URIs corresponding to same name.
*/
public Map<String, String> getRemoteUrls() throws GitException, InterruptedException;

/**
* getRemotePushUrls.
*
* @return a Map where String keys represent push-only URIs
* (with and without passwords, if any; ASCII or not,
* if applicable) for all remotes configured in this
* repository/workspace, and values represent names.
* There may be several URIs corresponding to same name.
*/
public Map<String, String> getRemotePushUrls() throws GitException, InterruptedException;

/**
* For a given repository, set a remote's URL
*
Expand Down Expand Up @@ -696,12 +718,23 @@ Map<String, String> getRemoteSymbolicReferences(String remoteRepoUrl, String pat
*/
List<ObjectId> revList(String ref) throws GitException, InterruptedException;

// --- new instance of same applied class

/**
* newGit.
*
* @return an {@link IGitAPI} implementation to manage another git repository
* with same general settings and implementation as the current one.
* @param somedir a {@link java.lang.String} object.
*/
GitClient newGit(String somedir);

// --- submodules

/**
* subGit.
*
* @return a IGitAPI implementation to manage git submodule repository
* @return an {@link IGitAPI} implementation to manage git submodule repository
* @param subdir a {@link java.lang.String} object.
*/
GitClient subGit(String subdir);
Expand Down
Loading
Loading