RDS instance in modify loop due to Crossplane checking AWS initialized fields. (version 0.45)

[WolfGanGeRTech]

What happened?

When a new RDS instance is created using Crossplane with "BackupTarget", "monitoringInterval" or "NetworkType" empty, the AWS will initialize those fields. This will cause Crossplane to see a difference between the values in the manifest "nil" and the default values in AWS. So it will keep applying changes to AWS to force the RDS to remove those fields.
Since this is not possible, Crossplane will keep in loop trying to apply the diffs.

I think this is related to the issue identified here #1949 , but instead of IAM Role it's a RDS instance.

Sample of the logs seen:

  &v1alpha1.DBInstanceParameters{
        ... // 2 ignored and 2 identical fields
        AvailabilityZone:        nil,
        BackupRetentionPeriod:   nil,
-       BackupTarget:            nil,
+       BackupTarget:            &"region",
        CACertificateIdentifier: nil,
        CharacterSetName:        nil,
        ... // 1 ignored and 24 identical fields
        MasterUsername:        nil,
        MaxAllocatedStorage:   nil,
-       MonitoringInterval:    nil,
+       MonitoringInterval:    &0,
        MonitoringRoleARN:     nil,
        MultiAZ:               nil,
        NcharCharacterSetName: nil,
-       NetworkType:           nil,
+       NetworkType:           &"IPV4",
        ... // 4 ignored and 13 identical fields
  }

How can we reproduce it?

Launch a new RDS with "monitoringInterval" and "NetworkType" empty and check AWS Cloudtrail requests to see the "modifyDB" events.

What environment did it happen in?

Crossplane version: 0.14.0
Crossplane aws-contrib: 0.45

[wotolom]

Hi. Thanks for opening this issue.

I also came across this bug yesterday.

For DBs with oracle engine the parameter NcharCharacterSetName also needs to be lateInitialized.

My guess is that the upgrade of github.com/aws/aws-sdk-go-v2/service/rds from 1.14 to 1.39 brought in these api changes.
(see: https://github.com/crossplane-contrib/provider-aws/blame/78c8c20903796a65ffa95a8ea6c75d9304c6ab48/go.mod#L23)

I will try to open a PR later today to address this. (want to check DBCluster also)

[wotolom]

short update:

while lateInitializing these fields did fix the problem, it seems the root cause may be lying somewhere different.
When creating a DBInstance that is part of a DBCluster, there were some more parameters that are triggering an update-loop:

Found observed difference in dbinstance
  &v1alpha1.DBInstanceParameters{
  	... // 1 ignored field
- 	AllocatedStorage: nil,
+ 	AllocatedStorage: &1,
  	... // 1 ignored field
  	AutoMinorVersionUpgrade: nil,
  	AvailabilityZone:        nil,
- 	BackupRetentionPeriod:   nil,
+ 	BackupRetentionPeriod:   &1,
  	... // 2 ignored and 1 identical fields
- 	CopyTagsToSnapshot:       nil,
+ 	CopyTagsToSnapshot:       &false,
  	CustomIAMInstanceProfile: nil,
  	DBClusterIdentifier:      nil,
  	... // 1 ignored and 2 identical fields
  	DBSubnetGroupName:   nil,
  	DBSystemID:          nil,
- 	DeletionProtection:  nil,
+ 	DeletionProtection:  &false,
  	Domain:              nil,
  	DomainAuthSecretARN: nil,
  	... // 4 ignored and 24 identical fields
  	PromotionTier:      nil,
  	PubliclyAccessible: nil,
- 	StorageEncrypted:   nil,
+ 	StorageEncrypted:   &false,
  	StorageThroughput:  nil,
- 	StorageType:        nil,
+ 	StorageType:        &"aurora",
  	... // 1 ignored and 4 identical fields
  }

As far as I can currently tell, they all seem to be one-time "init" after creation.

While I try to further analyse why we are now seeing this behaviour,
I would kindly ask people who face this issue to look into the MR in the cluster and check if these fields - that they did not set/apply - are present under spec.forProvider.
If this is the case, please try manually removing the concerning parameters (with e.g. kubectl edit) and see if this fixes the update-loop. Confirming this manual fix/workaround would be appreciated.

[z0rc]

I would kindly ask people who face this issue to look into the MR in the cluster and check if these fields - that they did not set/apply - are present under spec.forProvider.
If this is the case, please try manually removing the concerning parameters (with e.g. kubectl edit) and see if this fixes the update-loop. Confirming this manual fix/workaround would be appreciated.

I'm my case I use Composition to create DBInstance resource (among others). My composition resource don't have any of initialized fields under spec.forProvider, but created dbinstance resource has them. Manually editing out fields in question allows to stop reconciliation loop.

[WolfGanGeRTech]

I created a new RDS instance and confirmed that removing the fields from the mr stops the reconciliation loop.