Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-64383] combined refrepo became our bottleneck, support a fanout location too #644

Open
wants to merge 147 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
147 commits
Select commit Hold shift + click to select a range
8f5ec33
Introduce support for reference repository paths ending with /${GIT_U…
jimklimov Dec 7, 2020
022bbc6
Refactor: move findParameterizedReferenceRepository() into LegacyComp…
jimklimov Dec 8, 2020
6210adb
Add isParameterizedReferenceRepository() and support for parameterize…
jimklimov Dec 8, 2020
06d4859
Handle the possibility that the URL for parameterized checkout is nul…
jimklimov Dec 8, 2020
b5a6a7b
LegacyCompatibleGitAPIImpl.java: document params for new routines
jimklimov Dec 8, 2020
f90bdca
LegacyCompatibleGitAPIImpl.java: refactor urlNormalized generation in…
jimklimov Dec 8, 2020
e09eff1
LegacyCompatibleGitAPIImpl.java: introduce support for .../${GIT_URL_…
jimklimov Dec 8, 2020
17010ca
WorkspaceWithRepo.java (test): extend localMirror() with customizable…
jimklimov Dec 8, 2020
72a4a05
GitClientCloneTest.java: add test_clone_reference_parameterized_sha25…
jimklimov Dec 8, 2020
c6c010f
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Dec 9, 2020
bbedefd
Fix referencePath getName() => getPath()
jimklimov Dec 9, 2020
5306163
Update tests for fRefrepoBase being under target/ dir from CWD of the…
jimklimov Dec 9, 2020
bc7e198
GitClientCloneTest: check if refrepo dir exists already
jimklimov Dec 9, 2020
3ff19df
WorkspaceWithRepo.java: localMirror(): normalize() the returned pathn…
jimklimov Dec 10, 2020
a509386
GitClientCloneTest.java: extend API choices for assertAlternateFilePo…
jimklimov Dec 10, 2020
603e390
GitClientCloneTest.java: fix test_clone_reference_parameterized_sha25…
jimklimov Dec 10, 2020
66e23a1
LegacyCompatibleGitAPIImpl.java CliGitAPIImpl.java JGitAPIImpl.java: …
jimklimov Dec 10, 2020
5af4400
LegacyCompatibleGitAPIImpl.java: add variants of findParameterizedRef…
jimklimov Dec 10, 2020
509c5ec
[TMP] LegacyCompatibleGitAPIImpl.java: trace findParameterizedReferen…
jimklimov Dec 10, 2020
0b0fcbf
[TMP] WorkspaceWithRepo.java (test): trace localMirror() processing t…
jimklimov Dec 10, 2020
6a9c79d
LegacyCompatibleGitAPIImpl.java: refactor to use referenceExpanded to…
jimklimov Dec 10, 2020
2641971
LegacyCompatibleGitAPIImpl.java: add GIT_URL_FALLBACK and GIT_URL_SHA…
jimklimov Dec 10, 2020
c10231c
GitClientCloneTest.java: add cases for parameterized reference repo w…
jimklimov Dec 10, 2020
2016719
LegacyCompatibleGitAPIImpl.java: refactor normalizeGitUrl()
jimklimov Dec 10, 2020
71c0850
[WIP] LegacyCompatibleGitAPIImpl.java: skeleton for GIT_SUBMODULES an…
jimklimov Dec 10, 2020
ab04ae7
Introduce getRemoteUrls() and getRemotePushUrls()
jimklimov Dec 11, 2020
179f5c1
LegacyCompatibleGitAPIImpl.java: normalizeGitUrl(): revise for local …
jimklimov Dec 11, 2020
7c0ad4f
[WIP] LegacyCompatibleGitAPIImpl.java: skeleton for GIT_SUBMODULES an…
jimklimov Dec 11, 2020
ead03a6
CliGitAPIImpl.java: extend getRemote{,Push}Urls() with filters for AC…
jimklimov Dec 11, 2020
5fde3c9
GitClientCloneTest.java: if we expect particular wsRefrepo spelling, …
jimklimov Dec 11, 2020
6c7374c
GitClientCloneTest.java: for fallback case, expect wsRefrepoBase itself
jimklimov Dec 11, 2020
a2cf84f
GitClientCloneTest.java: use the LegacyCompatibleGitAPIImpl.normalize…
jimklimov Dec 11, 2020
823bd0f
GitClientCloneTest.java: for fallback case, expect wsRefrepoBase itse…
jimklimov Dec 11, 2020
0e5e0cd
LegacyCompatibleGitAPIImpl.java: skeleton for getSubmodulesUrls() as …
jimklimov Dec 12, 2020
172e40e
LegacyCompatibleGitAPIImpl.java: update comments for ideas about subm…
jimklimov Dec 14, 2020
d6cebe8
LegacyCompatibleGitAPIImpl.java: getSubmodulesUrls(): implement the s…
jimklimov Dec 14, 2020
864e34f
LegacyCompatibleGitAPIImpl.java: move arrDirNames declaration higher …
jimklimov Jan 10, 2021
55fda14
LegacyCompatibleGitAPIImpl.java: update commented thoughts
jimklimov Jan 10, 2021
3e81723
LegacyCompatibleGitAPIImpl.java: prepare to parse submodule workspaces
jimklimov Jan 10, 2021
ff65341
LegacyCompatibleGitAPIImpl.java: uncomment needle tracking in a loop …
jimklimov Jan 10, 2021
122746f
LegacyCompatibleGitAPIImpl.java: fix @param usage
jimklimov Jan 10, 2021
bf978ee
LegacyCompatibleGitAPIImpl.java: implement handling output of getSubm…
jimklimov Jan 10, 2021
aacc0cd
LegacyCompatibleGitAPIImpl.java: getSubmodulesUrls() / findParameteri…
jimklimov Jan 10, 2021
1a8306a
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Jan 10, 2021
769a2d4
LegacyCompatibleGitAPIImpl.java: log-tracing for getSubmodulesUrls()
jimklimov Jan 11, 2021
654362a
LegacyCompatibleGitAPIImpl.java: for now look in subdirs regardless o…
jimklimov Jan 11, 2021
5b6ff6d
LegacyCompatibleGitAPIImpl.java: comment updated
jimklimov Jan 11, 2021
8d8514c
LegacyCompatibleGitAPIImpl.java: rename getObjectPath() to less misle…
jimklimov Jan 11, 2021
925eaba
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
a103bdd
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
aaaef9c
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
a77dab9
LegacyCompatibleGitAPIImpl.java: in findParameterizedReferenceReposit…
jimklimov Jan 11, 2021
d664eb3
LegacyCompatibleGitAPIImpl.java: pass not-normalized URL from findPar…
jimklimov Jan 11, 2021
e29d947
LegacyCompatibleGitAPIImpl.java: check also not-normalized URL in fin…
jimklimov Jan 11, 2021
1816632
LegacyCompatibleGitAPIImpl.java: only add ".git" to reference if it i…
jimklimov Jan 11, 2021
8b693fd
LegacyCompatibleGitAPIImpl.java: fix tracing into private LOGGER inst…
jimklimov Jan 11, 2021
e49b985
LegacyCompatibleGitAPIImpl.java: trace looking for the needle in arrD…
jimklimov Jan 11, 2021
ab29c68
LegacyCompatibleGitAPIImpl.java: comment reshuffle
jimklimov Jan 11, 2021
66798fd
LegacyCompatibleGitAPIImpl.java: refactor findParameterizedReferenceR…
jimklimov Jan 11, 2021
776a249
LegacyCompatibleGitAPIImpl.java: trace which sub-git workspaces we do…
jimklimov Jan 11, 2021
094de4d
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to explicitl…
jimklimov Jan 11, 2021
d30de21
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to look in "…
jimklimov Jan 11, 2021
7ed0f0f
GitClient and its implementations: extend API with a newGit(somedir) …
jimklimov Jan 12, 2021
14cf345
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to look in r…
jimklimov Jan 12, 2021
2b61767
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to return fA…
jimklimov Jan 12, 2021
1c9a8f5
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to track rel…
jimklimov Jan 12, 2021
a29b00c
LegacyCompatibleGitAPIImpl.java: clearer trace messages in getSubmodu…
jimklimov Jan 12, 2021
25bef67
LegacyCompatibleGitAPIImpl.java: track referenceBaseDirAbs to compare…
jimklimov Jan 12, 2021
39dd9b4
LegacyCompatibleGitAPIImpl.java: fix getSubmodulesUrls() to return fA…
jimklimov Jan 12, 2021
c349b25
LegacyCompatibleGitAPIImpl.java: comment the ends of long logical blocks
jimklimov Jan 12, 2021
6a7bb18
LegacyCompatibleGitAPIImpl.java: use absolute pathnames "natively" in…
jimklimov Jan 12, 2021
f475606
LegacyCompatibleGitAPIImpl.java: do not need to check bare-mode dirs …
jimklimov Jan 12, 2021
2ddd7df
LegacyCompatibleGitAPIImpl.java: check basename-like dirs for being g…
jimklimov Jan 12, 2021
721d791
LegacyCompatibleGitAPIImpl.java: check the parent refrepo dir for hos…
jimklimov Jan 12, 2021
69f5b59
LegacyCompatibleGitAPIImpl.java: trace what getRemoteUrls() gave us
jimklimov Jan 12, 2021
9c48cb6
LegacyCompatibleGitAPIImpl.java: leave a TODO block for checking subm…
jimklimov Jan 12, 2021
f465258
LegacyCompatibleGitAPIImpl.java: trace mis-processing an assumed git …
jimklimov Jan 12, 2021
f6e756a
LegacyCompatibleGitAPIImpl.java: comment about abs dirnames in second…
jimklimov Jan 12, 2021
ce1827e
LegacyCompatibleGitAPIImpl.java: fix GitClient for abs dirnames in se…
jimklimov Jan 12, 2021
c5e863f
LegacyCompatibleGitAPIImpl.java: Log into build console the start and…
jimklimov Jan 12, 2021
d0bf23c
LegacyCompatibleGitAPIImpl.java: Refactor a useless needle to be null…
jimklimov Jan 12, 2021
0b761da
LegacyCompatibleGitAPIImpl.java: Refactor getSubmodulesUrls() to unif…
jimklimov Jan 12, 2021
6046f4b
LegacyCompatibleGitAPIImpl.java: Refactor getSubmodulesUrls() to name…
jimklimov Jan 12, 2021
3f7b4e4
LegacyCompatibleGitAPIImpl.java: re-use arrDirnames in getSubmodulesU…
jimklimov Jan 12, 2021
61a7d0b
LegacyCompatibleGitAPIImpl.java: comment the plan about submodule sea…
jimklimov Jan 12, 2021
b91e00b
LegacyCompatibleGitAPIImpl.java: update comments around shorter getSu…
jimklimov Jan 13, 2021
65684f3
LegacyCompatibleGitAPIImpl.java: only track absolute dirname values i…
jimklimov Jan 13, 2021
0c99c95
LegacyCompatibleGitAPIImpl.java: comment the plan for submodule handling
jimklimov Jan 13, 2021
327cb48
LegacyCompatibleGitAPIImpl.java: finally, do recursion in getSubmodul…
jimklimov Jan 13, 2021
45d268c
LegacyCompatibleGitAPIImpl.java: check not only for ".git" but also "…
jimklimov Jan 16, 2021
0413d95
LegacyCompatibleGitAPIImpl.java: for GIT_SUBMODULES mode search, if t…
jimklimov Jan 16, 2021
fad3727
LegacyCompatibleGitAPIImpl.java: implement a use-case of ".git" file …
jimklimov Jan 16, 2021
d37fe49
LegacyCompatibleGitAPIImpl.java: refactor getSubmodulesUrls() to use …
jimklimov Jan 16, 2021
6afacda
LegacyCompatibleGitAPIImpl.java: refactor getSubmodulesUrls() to retu…
jimklimov Jan 16, 2021
b278523
LegacyCompatibleGitAPIImpl.java: add logging for getObjectsFile()
jimklimov Jan 17, 2021
4468443
LegacyCompatibleGitAPIImpl.java: in getObjectsFile(), comment why it …
jimklimov Jan 17, 2021
0edcc0c
JGitAPIImpl.java / CliGitAPIImpl.java: inform the build console log r…
jimklimov Jan 17, 2021
0767c2a
LegacyCompatibleGitAPIImpl.java: in getObjectsFile() method comment, …
jimklimov Jan 17, 2021
03b3056
LegacyCompatibleGitAPIImpl.java: little clean-up in getObjectsFile()
jimklimov Jan 17, 2021
fd88900
LegacyCompatibleGitAPIImpl.java: referenceBaseDirAbs in getSubmodules…
jimklimov Jan 17, 2021
d72999b
LegacyCompatibleGitAPIImpl.java: suggest a selection of results[] *si…
jimklimov Jan 17, 2021
f1f478a
LegacyCompatibleGitAPIImpl.java: mark helpers as public static
jimklimov Jan 18, 2021
91ed630
LegacyCompatibleGitAPIImpl.java: check if referenceBaseDir is usable …
jimklimov Jan 18, 2021
de98cd1
GitClientCloneTest.java: reword original test for GIT_URL into checki…
jimklimov Jan 31, 2021
aa9eaad
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Jan 31, 2021
82a43eb
LegacyCompatibleGitAPIImpl.java: findParameterizedReferenceRepository…
jimklimov Jan 31, 2021
23745a8
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: listFiles()…
jimklimov Jan 31, 2021
f51e60f
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: getSubmodul…
jimklimov Jan 31, 2021
a5fdbd3
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: have non-ze…
jimklimov Jan 31, 2021
4341cd2
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: "new File()…
jimklimov Jan 31, 2021
4cd45c8
LegacyCompatibleGitAPIImpl.java: fix spotbugs complaints: use a file …
jimklimov Jan 31, 2021
fd293d7
LegacyCompatibleGitAPIImpl.java: normalizeUrl() failed with full path…
jimklimov Feb 22, 2021
c45cedd
LegacyCompatibleGitAPIImpl.java: normalizeUrl(): @SuppressFBWarnings …
jimklimov Feb 22, 2021
be57864
CliGitAPIImpl.java: launchCommandIn(): report workDir when throwing e…
jimklimov Jan 26, 2022
8f843d4
LegacyCompatibleGitAPIImpl.java: getSubmodulesUrls(): do not fail che…
jimklimov Jan 26, 2022
4ee6935
JGitAPIImpl.java + CliGitAPIImpl.java: report "reference" pathname th…
jimklimov Jan 26, 2022
e726d40
Merge branch 'master' as of 2022-08-01 into refrepo-args
jimklimov Aug 1, 2022
558d076
[JENKINS-69193] GitAPITestCase: test_submodule_update_with_error(): r…
jimklimov Aug 1, 2022
4402820
[JENKINS-69193] GitAPITestUpdateCliGit: rely less on strict wording o…
jimklimov Aug 10, 2022
5d97abe
Merge branch 'JENKINS-69193-bis' into refrepo-args
jimklimov Aug 10, 2022
4222f4e
Merge branch 'master' into refrepo-args
MarkEWaite Sep 5, 2022
63ca351
Merge branch 'master' into refrepo-args
MarkEWaite Sep 14, 2022
2544b99
Merge branch 'master' into refrepo-args
jimklimov Dec 21, 2022
059f7a2
Merge branch 'master' into refrepo-args
MarkEWaite Apr 11, 2023
7c28d4d
Merge branch 'spotless' into refrepo-args
MarkEWaite Apr 11, 2023
c882050
Format source changes
MarkEWaite Apr 11, 2023
9863085
Fix merge mistake
MarkEWaite Apr 11, 2023
8334390
Merge remote-tracking branch 'upstream/master' into refrepo-args
jimklimov Jun 14, 2023
eeb47b4
GitClient.java: minor fix to javadoc for subGit()
jimklimov Jun 14, 2023
9f162aa
Reconcile code style with changes in the PR #644 (added by @markewait…
jimklimov Jun 14, 2023
d3a6d56
Mark @Override methods like in master branch
jimklimov Jun 14, 2023
6840c75
Change PrintWriter use of UTF8 like in master branch
jimklimov Jun 14, 2023
d8ddcdb
GitClient.java: minor fix to javadoc for newGit()
jimklimov Jun 14, 2023
896c2c6
LegacyCompatibleGitAPIImpl: import DigestUtils to use by short reference
jimklimov Jun 14, 2023
62355d5
GitClientCloneTest: drop use of FileUtils.readFileToString() like in …
jimklimov Jun 14, 2023
2c10b59
spotless is clueless
jimklimov Jun 14, 2023
59e7474
Merge branch 'master' into refrepo-args
MarkEWaite Sep 13, 2023
4e1f3ab
Remove redundant null check
MarkEWaite Sep 13, 2023
f844f23
Resolve spotbugs warnings for toString() and unread vars
MarkEWaite Sep 13, 2023
43c38ba
Resolve spotbugs warnings on toLowerCase
MarkEWaite Sep 13, 2023
65b55c5
Assure stream is closed on exception
MarkEWaite Sep 13, 2023
9da15fa
Merge branch 'master' into refrepo-args
MarkEWaite Jan 17, 2024
1f379a7
Merge branch 'master' into refrepo-args
MarkEWaite Mar 9, 2024
cedfd7f
Merge remote-tracking branch 'upstream/master' into refrepo-args
jimklimov Jun 14, 2024
b2d7fef
Merge remote-tracking branch 'upstream/master' as of 2024-10-31 into …
jimklimov Oct 31, 2024
9dc373e
Merge remote-tracking branch 'upstream/master' as of 2025-01-16 into …
jimklimov Jan 16, 2025
d2fd099
Merge remote-tracking branch 'upstream/master' into refrepo-args
jimklimov Jan 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
GitClientCloneTest.java: use the LegacyCompatibleGitAPIImpl.normalize…
…GitUrl() directly
jimklimov committed Jan 10, 2021
commit a2cf84f23d9cdcf9b2c65223ef2fd6534b6e08f8
Original file line number Diff line number Diff line change
@@ -257,7 +257,7 @@ public Boolean isParameterizedReferenceRepository(String reference) {
* using the same access protocol. This routine converts the "url" string
* in a way that helps us confirm whether two spellings mean same thing.
*/
public String normalizeGitUrl(String url, Boolean checkLocalPaths) {
public static String normalizeGitUrl(String url, Boolean checkLocalPaths) {
String urlNormalized = url.replaceAll("/*$", "").replaceAll(".git$", "").toLowerCase();
if (!url.contains("://")) {
if (!url.startsWith("/") &&
Original file line number Diff line number Diff line change
@@ -293,8 +293,10 @@ public void test_clone_reference_parameterized_fallback() throws Exception, IOEx
@Test
public void test_clone_reference_parameterized_sha256() throws Exception, IOException, InterruptedException {
String wsMirror = workspace.localMirror();
/* Same rules of URL normalization as in LegacyCompatibleGitAPIImpl.java */
String wsMirrorNormalized = wsMirror.replaceAll("/*$", "").replaceAll(".git$", "").toLowerCase();
/* Same rules of URL normalization as in LegacyCompatibleGitAPIImpl.java
* should be okay for network URLs but are too complex for local pathnames */
//String wsMirrorNormalized = wsMirror.replaceAll("/*$", "").replaceAll(".git$", "").toLowerCase();
String wsMirrorNormalized = LegacyCompatibleGitAPIImpl.normalizeGitUrl(wsMirror, true);
String wsMirrorHash = org.apache.commons.codec.digest.DigestUtils.sha256Hex(wsMirrorNormalized);

/* Make a new repo replica to use as refrepo, in specified location */
@@ -350,8 +352,10 @@ public void test_clone_reference_parameterized_sha256() throws Exception, IOExce
@Test
public void test_clone_reference_parameterized_sha256_fallback() throws Exception, IOException, InterruptedException {
String wsMirror = workspace.localMirror();
/* Same rules of URL normalization as in LegacyCompatibleGitAPIImpl.java */
String wsMirrorNormalized = wsMirror.replaceAll("/*$", "").replaceAll(".git$", "").toLowerCase();
/* Same rules of URL normalization as in LegacyCompatibleGitAPIImpl.java
* should be okay for network URLs but are too complex for local pathnames */
//String wsMirrorNormalized = wsMirror.replaceAll("/*$", "").replaceAll(".git$", "").toLowerCase();
String wsMirrorNormalized = LegacyCompatibleGitAPIImpl.normalizeGitUrl(wsMirror, true);
String wsMirrorHash = org.apache.commons.codec.digest.DigestUtils.sha256Hex(wsMirrorNormalized);

/* Make a new repo replica to use as refrepo, in specified location */