Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#59: create adjust endpoint for reading checkpoint defintion of partioning #96

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
6b94980
Fixes #50
TebaleloS Sep 14, 2023
3c2ef9c
Fixes #50
TebaleloS Sep 14, 2023
6453d64
Fixes #50
TebaleloS Sep 14, 2023
2ddebaf
Fixes #50
TebaleloS Sep 14, 2023
7316188
Fixed #50 - Added liscense
TebaleloS Sep 19, 2023
6d6c71a
Merge remote-tracking branch 'origin/feature/37-dao-between-agent-ser…
lsulak Sep 21, 2023
55fe01f
#50: post-merge fixes
lsulak Sep 21, 2023
2314e82
Fixed #50 - added db connection class
TebaleloS Sep 26, 2023
ec9832a
Merge remote-tracking branch 'origin/feature/37-dao-between-agent-ser…
TebaleloS Sep 28, 2023
798e54f
Fixed #50 - refactored the endpoint to use JDBC drivers
TebaleloS Sep 28, 2023
4aaba46
Fixed #50 - refactored
TebaleloS Oct 7, 2023
b5a8677
Fixed #50 - refactored
TebaleloS Oct 9, 2023
0ca557d
Fixed #50 - Loaded db configs
TebaleloS Oct 9, 2023
12b514f
Fixes #50 - fixed conflicts
TebaleloS Oct 10, 2023
1238951
Fixed #50 - merged master and fixed the conflicts
TebaleloS Oct 10, 2023
c418ba2
Fixed #50 - merged master and fixed the conflict
TebaleloS Oct 13, 2023
e19061b
Fixed #50 - Added slickpg dependancy
TebaleloS Oct 16, 2023
f702170
Fixed #50 - Changed slickpg dependancy version.
TebaleloS Oct 16, 2023
565f98b
Closes #50
TebaleloS Oct 16, 2023
3241f97
Closes #50 - Changed repositories package to database and checkpoint …
TebaleloS Oct 16, 2023
0e84b16
Closes #50 - annotated service for resolving the bean issue
TebaleloS Oct 23, 2023
8534a8c
Closes #50 - Added some scala doc
TebaleloS Oct 23, 2023
563d569
Merge branch 'master' into feature/#50-Create-or-adjust-endpoint-for-…
TebaleloS Oct 23, 2023
83d8b74
Merge branch 'master' into feature/#50-Create-or-adjust-endpoint-for-…
TebaleloS Oct 25, 2023
eef07f4
Closes #50
TebaleloS Oct 26, 2023
f5b7591
Merge remote-tracking branch 'origin/feature/#50-Create-or-adjust-end…
TebaleloS Oct 26, 2023
d33f683
Closes #50 - removed test class for the endpoint
TebaleloS Oct 26, 2023
21befee
Merged master
TebaleloS Oct 27, 2023
9c9cf61
Merged master and resolve confflicts
TebaleloS Oct 31, 2023
e710558
Resolved comments
TebaleloS Oct 31, 2023
f2fb62b
removed out directory
TebaleloS Oct 31, 2023
4147167
Merge branch 'master' into feature/#50-Create-or-adjust-endpoint-for-…
TebaleloS Oct 31, 2023
6eb5eb0
#50: fixing the build, correcting dependencies and using the same Spr…
lsulak Oct 31, 2023
248a0a5
#50: adding handy util for options
lsulak Oct 31, 2023
8be7ba6
#50: fixing the build and simplifying the server code
lsulak Oct 31, 2023
c5a269f
#50: adding missing licence header
lsulak Oct 31, 2023
d1163ff
Fixes #59 - Merged #50 and added get checkpoint controller
TebaleloS Oct 25, 2023
8520a34
Fixes #59 - Redefined the endpoint
TebaleloS Nov 2, 2023
a314d61
Fixes #59 - Redefined the endpoint
TebaleloS Nov 2, 2023
27ee7d6
Fixes #59 - Redefined the endpoint and added the Service
TebaleloS Nov 2, 2023
029ea05
Fixes #59 - Redefined the endpoint and added the Service
TebaleloS Nov 2, 2023
0900264
Fixes #59 - Fixed licences
TebaleloS Nov 2, 2023
f2d1e1f
Fixes #59
TebaleloS Nov 2, 2023
4f9a7aa
Fixes #59
TebaleloS Nov 2, 2023
2db07ba
Fixes #59 - Modified return type on the read_checkpoint endpoint
TebaleloS Nov 3, 2023
7776fb2
Fixes #59
TebaleloS Nov 8, 2023
ed5a6bf
Fixing merge conflicts from master branch
TebaleloS Nov 8, 2023
c6bd426
Fixes #59 - adding license headers
TebaleloS Nov 8, 2023
d4d89c3
Fixes #59 - Refactored the endpoint
TebaleloS Nov 9, 2023
a92e1ba
Fixes #59 - Refactored the code
TebaleloS Nov 9, 2023
b98d7c4
Fixes #59 - Removed unused code
TebaleloS Nov 9, 2023
43e8ebe
Merge branch 'master' into feature/#59-Create-adjust-endpoint-for-rea…
lsulak Nov 10, 2023
92a70a8
moving back irrelevant changes
lsulak Nov 10, 2023
050641a
moving back irrelevant changes and refactoring
lsulak Nov 10, 2023
deea142
Merge branch 'master' into feature/#59-Create-adjust-endpoint-for-rea…
lsulak Nov 12, 2023
d9fcb0d
Merge branch 'master' into feature/#59-Create-adjust-endpoint-for-rea…
lsulak Nov 12, 2023
741f312
Fixes #59 - Adding db service test cases
TebaleloS Nov 14, 2023
ae39775
Fixes #59 - Adding db service test cases
TebaleloS Nov 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,6 @@ package za.co.absa.atum.model.dto

case class PartitioningDTO (
partitioning: Seq[PartitionDTO],
parentPartitioning: Option[Seq[PartitionDTO]]
parentPartitioning: Option[Seq[PartitionDTO]],
author: String
)
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ package za.co.absa.atum.server.api.controller
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.http.HttpStatus
import org.springframework.web.bind.annotation._
import za.co.absa.atum.model.dto.CheckpointDTO
import za.co.absa.atum.model.dto.{AtumContextDTO, CheckpointDTO}
import za.co.absa.atum.model.dto.{AdditionalDataDTO, AtumContextDTO, CheckpointDTO, MeasureDTO, PartitioningDTO}
import za.co.absa.atum.server.api.service.DatabaseService

import java.util.concurrent.CompletableFuture


Expand All @@ -40,4 +40,17 @@ class PrototypeController @Autowired()(databaseService: DatabaseService){
def createCheckpoint(@RequestBody checkpoint: CheckpointDTO): CompletableFuture[CheckpointDTO] = {
databaseService.saveCheckpoint(checkpoint)
}

/**
* Creates a partitioning in a DB and returns an Atum Context out of it.
*
* @param partitioningInfo DTO (JSON-like) object containing fields that will be used for creating a partitioning.
* @return A new AtumContext object that uses newly obtained partitioning.
*/
@PostMapping(Array("/createPartitioning"))
@ResponseStatus(HttpStatus.OK)
def createPartitioningIfNotExists(@RequestBody partitioningInfo: PartitioningDTO): AtumContextDTO = {
databaseService.createPartitioningIfNotExists(partitioningInfo)
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,22 @@

package za.co.absa.atum.server.api.service

import org.springframework.beans.factory.annotation.Autowired
import org.springframework.stereotype.Service
import za.co.absa.atum.model.dto.CheckpointDTO
<<<<<<< HEAD
import za.co.absa.atum.model.dto.{AdditionalDataDTO, AtumContextDTO, CheckpointDTO, MeasureDTO, PartitionDTO, PartitioningCreationDTO}
=======
import za.co.absa.atum.model.dto.{CheckpointDTO, PartitionDTO, PartitioningDTO}
>>>>>>> d9fcb0d2c85e2893d1ce37926f08fc99881d0d06
import za.co.absa.atum.server.api.implicits.scalaToJavaFuture
import za.co.absa.atum.server.api.provider.PostgresAccessProvider

import java.util.concurrent.CompletableFuture
import scala.concurrent.ExecutionContext

@Service
class DatabaseService @Autowired()() {
class DatabaseService (@Autowired postgresAccessProvider: PostgresAccessProvider = new PostgresAccessProvider){

val postgresAccessProvider: PostgresAccessProvider = new PostgresAccessProvider
// private val postgresAccessProvider: PostgresAccessProvider = new PostgresAccessProvider

/** This service function saves the checkpoint into the database. */
def saveCheckpoint(checkpoint: CheckpointDTO): CompletableFuture[CheckpointDTO] = {
Expand All @@ -38,4 +41,17 @@ class DatabaseService @Autowired()() {
} yield checkpoint
}

/**
* Function to create partitioning based on the provided fields
*
* @param partitioningInfo json information to be added while creating a partition
*/
def createPartitioningIfNotExists(partitioningInfo: PartitioningDTO): AtumContextDTO = {
// Todo - db function call to be implemented in #23
val partitioning = Seq()
val measures = Set.empty[MeasureDTO]
val additionalData = AdditionalDataDTO(additionalData = Map.empty)

AtumContextDTO(partitioning, measures, additionalData)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
package za.co.absa.atum.server.api.service

import org.junit.jupiter.api.BeforeEach
import org.mockito.{Mockito, MockitoSugar}
import org.scalatest.{BeforeAndAfter, BeforeAndAfterEach}
import za.co.absa.atum.model.dto._
import za.co.absa.atum.server.api.provider.PostgresAccessProvider
import org.scalatest.concurrent.ScalaFutures

import java.time.OffsetDateTime
import java.util.UUID
import za.co.absa.atum.model.dto.MeasureResultDTO.{ResultValueType, TypedValue}
import org.scalatest.flatspec.AsyncFlatSpec
import org.scalatest.matchers.should.Matchers

import scala.concurrent.Future

class DatabaseServiceTest extends AsyncFlatSpec with Matchers with MockitoSugar with ScalaFutures with BeforeAndAfterEach {

// Mock dependencies
val postgresAccessProvider: PostgresAccessProvider = mock[PostgresAccessProvider]
val databaseService = new DatabaseService(postgresAccessProvider)


"saveCheckpoint" should "return a CompletableFuture containing the saved CheckpointDTO" in {
// Construct an instance of CheckpointDTOC
val id = UUID.randomUUID()
val measure = MeasureDTO("measurementName", Seq("controlColumn1", "controlColumn2"))
val mainValue = TypedValue("Value1", ResultValueType.String)
val supportValue = Map("supportKey" -> TypedValue("supportValue", ResultValueType.String))
val measureResult = MeasureResultDTO(mainValue, supportValue)
val partition = PartitionDTO("key", "value")
val partitioning = PartitioningDTO(Seq(partition), Some(Seq(partition)))

val measurement = MeasurementDTO(measure, measureResult)

val checkpoint = CheckpointDTO(
id = id,
name = "myCheckpoint",
author = "Author1",
measuredByAtumAgent = true,
partitioning = Seq(partition),
processStartTime = OffsetDateTime.now(),
processEndTime = Some(OffsetDateTime.now().plusHours(1)),
measurements = Seq(measurement)
)

// Mock behavior of postgresAccessProvider.runs.writeCheckpoint
when(postgresAccessProvider.runs.writeCheckpoint(checkpoint))
.thenReturn(Future.successful(Unit))

// Act & Assert
val result = databaseService.saveCheckpoint(checkpoint)
print(result)
assert(result.get() == checkpoint)
}

// Test case 2: SaveCheckpoint Function with Null Input
"SaveCheckpoint" should "handle null or invalid input gracefully" in {

val invalidCheckpoint = null

// Act & Assert
assertThrows[Exception] {
databaseService.saveCheckpoint(invalidCheckpoint)
}
}

// test("testReadCheckpoint") {}

}
Loading