SECURITY-3: Switch from Ant -> Gradle #419

MrCreosote · 2024-02-10T01:55:02Z

This PR is a bit large but for me it was easier to understand the changes with the full ant -> gradle conversion. Instead, the PR is split up mostly by commit - the first commit gets the basics (compile & test) working, and then each new commit adds a new feature (or in the case of the last commit, improves an old one).

I found that Eclipse Buildship really doesn't like our project layout, and so the next PR will switch the project to a standard Gradle layout, but that involves moving every single source file, so... different PR.

Tested by replacing the old auth jar in workspace deluxe with the fat jar

Service responds to testmode endpoints that access the db, namely GET customroles Also split the build into dependencies and compile so the dependencies are cached

MrCreosote · 2024-02-10T01:56:34Z

build.gradle

+	// For some reason the test directories are included in the war file
+	// although the files are gone. Not sure why


This gets fixed in the next PR

codecov · 2024-02-10T02:02:34Z

Codecov Report

Merging #419 (8fce168) into develop (09b2011) will not change coverage.
Report is 1 commits behind head on develop.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             develop     #419   +/-   ##
==========================================
  Coverage      93.39%   93.39%           
  Complexity      2142     2142           
==========================================
  Files            126      126           
  Lines           7520     7520           
  Branches        1178     1178           
==========================================
  Hits            7023     7023           
  Misses           454      454           
  Partials          43       43

ialarmedalien · 2024-02-12T19:08:15Z

build.gradle

+task buildGitCommitFile {
+	doLast {
+		def commitId = grgit.head().id
+		// is there a varible for builddir/classe/java/main?


I have these fixed locally, they'll show up in a future PR if that's ok

ialarmedalien · 2024-02-12T19:09:57Z

build.gradle

+	 * TODO TEST Figure out why tests fail without this and remove. Might have something to do
+	 * with the stfuLoggers() call in many of the tests, might kill logging for tests that
+	 * require it
+	 * Although it seems to make Mongo start up correctly as well which is odd


same issue in the workspace -- there seems to be some sort of interaction between gradle and the loggers that means tests only pass with forkEvery on

ialarmedalien · 2024-02-12T19:15:14Z

build.gradle

+	// anyway.
+	// https://mvnrepository.com/artifact/org.jetbrains/syslog4j/0.9.46
+	// Need to rework the java common logger to not use syslog4j at all since it's abandonware
+	// and has a ton of CVEs, even in the newer versions.


you can get it easily by adding clojars to the repo list:

repositories { mavenCentral() maven { name = "Clojars" url = "https://repo.clojars.org/" } }

see https://github.com/kbase/workspace_deluxe/pull/676/files#diff-49a96e7eea8a94af862798a45174e6ac43eb4f8b4bd40759b5da63ba31ec3ef7

Ok, cool. That being said, I think there's a reason it's not on maven - it hasn't been maintained for 10 years and is riddled with CVEs. At this point I'd prefer this hacky garbage since it gives us even more impetus to get rid of syslog4j altogether

Presumably the same logic also applies to the kbase jars?

I don't think so - we can get rid of syslog4j because we don't want the services logging to syslog anymore anyway. We still need the code in the kbase jars like common / auth

I have thought about trying to get GHA to build jars though, I think they have a maven set up

Yes, I looked into it when I was doing the workspace conversion. It would certainly be neater than having a repo of jars...

.github/workflows/test.yml

ialarmedalien · 2024-02-13T13:22:50Z

build.gradle

+ *
+ * This generated file contains a sample Java application project to get you started.
+ * For more details take a look at the 'Building Java & JVM projects' chapter in the Gradle
+ * User Manual available at https://docs.gradle.org/7.4.2/userguide/building_java_projects.html


Not sure if all this blurb is really needed...

Nuked locally, will show up in a future PR

Xiangs18 · 2024-02-14T00:33:44Z

@MrCreosote
Do you think we are ready to remove .classpath as well?

Xiangs18 · 2024-02-14T01:26:47Z

Dockerfile

+COPY templates /tmp/auth2/templates/
+COPY war /tmp/auth2/war/
+# for the git commit
+COPY .git /tmp/auth2/.git/


What's the purpose of moving .git inside the container? To track/verify version?

This step needs access to git: https://github.com/kbase/auth2/pull/419/files#diff-49a96e7eea8a94af862798a45174e6ac43eb4f8b4bd40759b5da63ba31ec3ef7R23

It doesn't factor in the final container size due to the 2 stage build

MrCreosote · 2024-02-14T01:29:30Z

Do you think we are ready to remove .classpath as well?

It's removed in the next commit after I get Eclipse builds working (with a minor annoyance, but working)

Xiangs18 · 2024-02-14T01:51:27Z

@MrCreosote
I am a little bit confused about 3 tasks: generateTemplateFileList, fatTestJar, and generateManageAuthScript. Could you please explain more why we need each of them?

MrCreosote · 2024-02-14T01:57:53Z

I am a little bit confused about 3 tasks: generateTemplateFileList, fatTestJar, and generateManageAuthScript. Could you please explain more why we need each of them?

Easy one first: generateManageAuthScript generates this: https://github.com/kbase/auth2/blob/develop/README.md?plain=1#L260

generateTemplateFileList creates a list of all the templates (a manifest) in the templates directory. This is solely to make it easier for the test rig to pull the templates out of the fat test jar, as listing directories inside a jar is unnecessarily difficult.

fatTestJar generates a fat test jar, including all dependencies, the templates, and the templates manifest. The fat jar will replace the current auth jar + the jars repo in workspace and other repos for running an auth2 server in their test rigs. The fat jar is completely self contained, so the jars repo is no longer needed

Xiangs18 · 2024-02-14T02:10:53Z

@MrCreosote
And where will this fat jar locate in the repo? I cannot find it anywhere after running fatTestJar task.

MrCreosote · 2024-02-14T02:21:11Z

And where will this fat jar locate in the repo? I cannot find it anywhere after running fatTestJar task.

It's in build/libs

Xiangs18

LGTM!

MrCreosote added 12 commits February 8, 2024 12:21

Tests pass

f9c71c3

Try and fix coverage

f140d30

Handle creating git commit file

9df0d13

Build server with gradle w/ ant

8f8527a

Build a fat test jar with dependencies and templates

f983686

Get fat test jar working for test support

87244ab

Tested by replacing the old auth jar in workspace deluxe with the fat jar

Get javadoc working

7b38eb0

Make docker-compose start the service & test

a02ab2f

Service responds to testmode endpoints that access the db, namely GET customroles Also split the build into dependencies and compile so the dependencies are cached

Create the manage_auth script

3a15bfc

Delete build.xml

d890287

Update release notes

2d1f2f9

Only store the git commit file in the build

8fce168

MrCreosote requested review from ialarmedalien and Xiangs18 February 10, 2024 01:55

MrCreosote commented Feb 10, 2024

View reviewed changes

ialarmedalien reviewed Feb 12, 2024

View reviewed changes

.github/workflows/test.yml Show resolved Hide resolved

ialarmedalien approved these changes Feb 12, 2024

View reviewed changes

ialarmedalien reviewed Feb 13, 2024

View reviewed changes

Xiangs18 reviewed Feb 14, 2024

View reviewed changes

Xiangs18 approved these changes Feb 14, 2024

View reviewed changes

MrCreosote merged commit 37bff04 into kbase:develop Feb 14, 2024
9 checks passed

MrCreosote deleted the dev-gradle branch February 14, 2024 02:30

MrCreosote changed the title ~~Switch from Ant -> Gradle~~ SECURITY-3: Switch from Ant -> Gradle Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SECURITY-3: Switch from Ant -> Gradle #419

SECURITY-3: Switch from Ant -> Gradle #419

MrCreosote commented Feb 10, 2024

MrCreosote Feb 10, 2024

codecov bot commented Feb 10, 2024 •

edited

Loading

ialarmedalien Feb 12, 2024 •

edited

Loading

MrCreosote Feb 12, 2024

ialarmedalien Feb 12, 2024

ialarmedalien Feb 12, 2024 •

edited

Loading

MrCreosote Feb 12, 2024

ialarmedalien Feb 12, 2024

MrCreosote Feb 12, 2024 •

edited

Loading

MrCreosote Feb 12, 2024

ialarmedalien Feb 13, 2024

ialarmedalien Feb 13, 2024

MrCreosote Feb 13, 2024

Xiangs18 commented Feb 14, 2024 •

edited

Loading

Xiangs18 Feb 14, 2024 •

edited

Loading

MrCreosote Feb 14, 2024

MrCreosote commented Feb 14, 2024

Xiangs18 commented Feb 14, 2024 •

edited

Loading

MrCreosote commented Feb 14, 2024

Xiangs18 commented Feb 14, 2024 •

edited

Loading

MrCreosote commented Feb 14, 2024

Xiangs18 left a comment

		// For some reason the test directories are included in the war file
		// although the files are gone. Not sure why

SECURITY-3: Switch from Ant -> Gradle #419

SECURITY-3: Switch from Ant -> Gradle #419

Conversation

MrCreosote commented Feb 10, 2024

Choose a reason for hiding this comment

codecov bot commented Feb 10, 2024 • edited Loading

Codecov Report

ialarmedalien Feb 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ialarmedalien Feb 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrCreosote Feb 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xiangs18 commented Feb 14, 2024 • edited Loading

Xiangs18 Feb 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrCreosote commented Feb 14, 2024

Xiangs18 commented Feb 14, 2024 • edited Loading

MrCreosote commented Feb 14, 2024

Xiangs18 commented Feb 14, 2024 • edited Loading

MrCreosote commented Feb 14, 2024

Xiangs18 left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 10, 2024 •

edited

Loading

ialarmedalien Feb 12, 2024 •

edited

Loading

ialarmedalien Feb 12, 2024 •

edited

Loading

MrCreosote Feb 12, 2024 •

edited

Loading

Xiangs18 commented Feb 14, 2024 •

edited

Loading

Xiangs18 Feb 14, 2024 •

edited

Loading

Xiangs18 commented Feb 14, 2024 •

edited

Loading

Xiangs18 commented Feb 14, 2024 •

edited

Loading