Seemingly impossible JSON serialization -- NUL-corrupted keys, values. Worse on JDK21?

[jaredstehler]

I'm posting this as a long-shot attempt to see if anyone recognizes this issue or what might be causing it; this is definitely a confusing one!

tl;dr -- we have observed messages in our kafka topics which have strings where some portion of them have been written as unicode NUL bytes. we are using jackson to serialize objects into json as byte[]. this occurs for keys as well as values, which means it's not user input. it's also quite intermittent, though it seems to occur much more frequently on Java 21 than on Java 17.

We noticed this issue a few months back in perhaps one or two messages, and it went away. However, when we started running our producer apps on Java 21, this became a more widespread issue, though still highly intermittent (i.e. ~5 out of millions of messages would exhibit this behavior).

The issue manifests as JSON with corrupted keys or values; a corrupted string has the correct length, but with up to 100% if its suffix chars replaced by NUL (0) bytes. Since every case I've observed shows the corruption confined to within "valid" json structure, it would appear that this is occuring in the jackson layer, and not somewhere beyond.

Looking at SerializedString, there is an old comment about volatile fields; I have yet to reproduce this in a way to capture that level of the serializer state to (dis)prove relevance here: https://github.com/FasterXML/jackson-core/blob/93e18c6e2fc06ebd0bb9a74c4db462662f09db20/src/main/java/com/fasterxml/jackson/core/io/SerializedString.java#L26-L43

The messages occur after an app has been running for ~ 1 to 2 hours, but again, quite rare and intermittent.

This is with jackson 2.12.6, and we are not using afterburner or blackbird. Producers are uniformly running on arm64 hosts, with G1GC.

Here are some representative examples of the issue.. the invalid keys are all bean properties, there are no generic maps being serialized here.
(note that the actual binary data has NUL bytes, not encoded \u0000 text, which we see on the deser side as)

Illegal unquoted character ((CTRL-CHAR, code 0))

should be newState.amendmentDetails:

{
  "amendmentId": 1234,
  "auditAction": "CREATE",
  "newState": {
    "effectiveAt": 1709347167435,
    "id": 2345,
    "subscriptionId": 6789,
    "amendmen\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000": [
      {
        "@type": "AmendmentDetail",
        "amendmentId": 4321,
        "type": "THE_TYPE",
        "value": [
          2025,
          2,
          28
        ]
      }
    ]
  }
}

should be oldPropertyDefinition.propertyRequirements:

{
  "propertyName": "prop-name",
  "changeType": "UPDATE",
  "timestamp": 1709222691402,
  "newProperty": {
    "id": 1234,
    "name": "prop-name",
    "type": "number"
  },
  "oldProperty": {
    "id": 5678,
    "name": "prop-name",
    "type": "number"
  },
  "newPropertyDefinition": {
    "property": {
      "id": 1234,
      "name": "prop-name",
      "type": "number"
    },
    "typeId": "1234"
  },
  "oldPropertyDefinition": {
    "property": {
      "id": 1234,
      "name": "prop_name",
      "label": "Prop label \"details (example)\"",
      "type": "number",
    },
    "property\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000": {
      "gates": [],
      "names": [
        "name1"
      ],
      "settings": [],
      "operator": "AND"
    },
    "group": "platform",
    "typeId": "1234"
  },
  "id": 1234,
  "typeId": "1234",
  "partition": "987_foo-bar"
}

should be newPropertyDefinition.property:

{
  "newPropertyDefinition": {
    "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000": {
      "id": 866081123,
      "name": "prop-def"
    },
    "typeId": "123"
  },
  "id": 3245,
  "typeId": "789"
}

in this case, the value is a java enum, should be "OR":

{
  "propertyRequirements": {
    "gates": [],
    "names": [
      "name1",
      "name2"
    ],
    "settings": [],
    "operator": "\u0000\u0000"
  },
}

[cowtowncoder]

Very interesting (and bit alarming). I do not remember hearing about this issue before; your guess wrt SerializedString sounds plausible.

[Venorcis]

Just want to throw in we also sometimes see this issue for half a year or so now; just some random characters in the payload get replaced with \u0000 (always exactly 1 nullbyte for 1 character)

[JooHyukKim]

Which JDK & Jackson version you are using?

[cowtowncoder]

Quick note: anyone experiencing this should try our jackson-core 2.18.0, since as per FasterXML/jackson-core#1274 (created based on this discussion) changes SerializedString in a way that might resolve the issue.

[Venorcis]

We are on Jackson 2.17.2, I'll see if we can move to 2.18 @cowtowncoder, thank you

[cowtowncoder]

@Venorcis great! Let us know how things go.