Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#59: create adjust endpoint for reading checkpoint defintion of partioning #96

Closed
wants to merge 58 commits into from
Closed

#59: create adjust endpoint for reading checkpoint defintion of partioning #96

wants to merge 58 commits into from

Conversation

TebaleloS
Copy link
Collaborator

@TebaleloS TebaleloS commented Oct 25, 2023 •
edited
Loading

Retrieve checkpoint endpoint

  • Defined endpoint for creating partitioning.
  • Prepared the service method that will be used to create parting if it doesn't exist.
  • Added a case class that defines the fields that will be used to retrieve the checkpoint definition when creating the partitioning.

Closes #59

TebaleloS and others added 23 commits September 14, 2023 10:06
@TebaleloS
Fixes #50
6b94980
@TebaleloS
Fixes #50
3c2ef9c
@TebaleloS
Fixes #50
6453d64
@TebaleloS
Fixes #50
2ddebaf
@TebaleloS
Fixed #50 - Added liscense
7316188
@lsulak
Merge remote-tracking branch 'origin/feature/37-dao-between-agent-ser…
6d6c71a 
…ver-adjustments' into feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data

# Conflicts:
#	server/src/main/scala/za/co/absa/atum/web/model/Checkpoint.scala
@lsulak
#50: post-merge fixes
55fe01f
@TebaleloS
Fixed #50 - added db connection class
2314e82
@TebaleloS
Merge remote-tracking branch 'origin/feature/37-dao-between-agent-ser…
ec9832a 
…ver-adjustments' into feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data

# Conflicts:
#	project/plugins.sbt
@TebaleloS
Fixed #50 - refactored the endpoint to use JDBC drivers
798e54f
@TebaleloS
Fixed #50 - refactored
4aaba46
@TebaleloS
Fixed #50 - refactored
b5a8677
@TebaleloS
Fixed #50 - Loaded db configs
0ca557d
@TebaleloS
Fixes #50 - fixed conflicts
12b514f
@TebaleloS
Fixed #50 - merged master and fixed the conflicts
1238951
@TebaleloS
Fixed #50 - merged master and fixed the conflict
c418ba2
@TebaleloS
Fixed #50 - Added slickpg dependancy
e19061b
@TebaleloS
Fixed #50 - Changed slickpg dependancy version.
f702170
@TebaleloS
Closes #50
565f98b
@TebaleloS
Closes #50 - Changed repositories package to database and checkpoint …
3241f97 
…repository to Runs
@TebaleloS
Closes #50 - annotated service for resolving the bean issue
0e84b16
@TebaleloS
Closes #50 - Added some scala doc
8534a8c
@TebaleloS
Merge branch 'master' into feature/#50-Create-or-adjust-endpoint-for-…
563d569 
…accepting-and-saving-checkpoint-data
@TebaleloS TebaleloS changed the title create adjust endpoint for reading checkpoint defintion of partioning: 59 #59: create adjust endpoint for reading checkpoint defintion of partioning Oct 25, 2023
@TebaleloS TebaleloS force-pushed the feature/#59-Create-adjust-endpoint-for-reading-checkpoint-defintion-of-partioning branch from 7bda244 to 2db07ba Compare November 3, 2023 13:17
@TebaleloS TebaleloS force-pushed the feature/#59-Create-adjust-endpoint-for-reading-checkpoint-defintion-of-partioning branch from e16e9d0 to ed5a6bf Compare November 8, 2023 10:33
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 8, 2023

JaCoCo agent module code coverage report - spark:2 - scala 2.12.12

There is no coverage information present for the Files changed

Total Project Coverage 83.26% 🍏

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 8, 2023 •
edited
Loading

JaCoCo server module code coverage report - scala 2.12.12

File Coverage [70%] ❌
AtumConfig.scala 100% 🍏
TestService.scala 100% 🍏
PostgresAccessProvider.scala 95.45% 🍏
BaseConfig.scala 66.67% ❌
DatabaseService.scala 25.81% ❌
Partition.scala 0% ❌
Total Project Coverage 17.16% ❌

@TebaleloS TebaleloS force-pushed the feature/#59-Create-adjust-endpoint-for-reading-checkpoint-defintion-of-partioning branch from 4bd6d03 to a92e1ba Compare November 9, 2023 12:31
@lsulak lsulak added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Nov 10, 2023
lsulak
lsulak reviewed Nov 10, 2023
View reviewed changes
server/src/main/scala/za/co/absa/atum/server/api/service/DatabaseService.scala Outdated
/**
* Function to retrieve checkpoint based on the provided fields
*
* @param filterCriteria JSON object containing the fields for filtering the checkpoint
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this filterCriteria about? I saw it elsewhere as well

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has being changed to partitioningInfo.

lsulak
lsulak reviewed Nov 10, 2023
View reviewed changes
server/src/test/scala/za/co/absa/atum/server/api/service/DatabaseServiceTest.scala Outdated

test("testReadCheckpoint") {}

test("testPostgresAccessProvider") {}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't really test anything, is it still in progress?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it was just a preparation and I forgot to remove them because the implementation was totally wrong.

@lsulak
Copy link
Collaborator

lsulak commented Nov 10, 2023

I really need this so I fixed the merge conflicts and did some refactoring. Apologies @TebaleloS, I know that it's not a good practice to do this, but I want to work on the #23 during the weekend a bit.

I also removed the Partitioning Created DTO you created, because PartitioningDTO is perfectly suitable for the job now I think. If something is missing, we can add it there, not creating a new DTO that is almost the same as the old one.

@lsulak
Copy link
Collaborator

lsulak commented Nov 10, 2023

The build is failing because of the unit tests in DatabaseServiceTest - when I commented it out, it was fully working

@lsulak lsulak closed this Nov 14, 2023
@lsulak
Copy link
Collaborator

lsulak commented Nov 14, 2023

Closing, replaced by #121 and future PR related to the DB tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lsulak lsulak lsulak left review comments

@benedeki benedeki Awaiting requested review from benedeki benedeki is a code owner

@Zejnilovic Zejnilovic Awaiting requested review from Zejnilovic Zejnilovic is a code owner

@dk1844 dk1844 Awaiting requested review from dk1844 dk1844 is a code owner

@salamonpavel salamonpavel Awaiting requested review from salamonpavel salamonpavel is a code owner

Assignees

@TebaleloS TebaleloS

Labels
work in progress Work on this item is not yet finished (mainly intended for PRs)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create (adjust) endpoint for reading checkpoint defintion of partioning
2 participants
@TebaleloS @lsulak