ConfigResponse Protobuf Broken

[cranst0n]

There is an issue parsing ConfigResponse from the AuxBrain API as noted here.

In addition to failures using your attempts in Javascript, I've tried using Scala and Dart libraries unsuccessfully.

I believe I've tracked down the underlying cause using protoscope which can parse raw protocol buffer data and show the underlying structure in terms of tag numbers and values.

Here's an excert of the protoscope output of a ShellSpec taken from the dlcCatalog section of a ConfigResponse:

...
  2: {
      1: {"ei_depot_1_new_order"}
      4: 20
      5: 0
      6: 0.0  # 0x0i64
      12: {
        1: 100
        2: {
          1: {"ei_depot_1_new_order"}
          2: {
            14:SGROUP
            13: 101
            13:EGROUP
            13:EGROUP
            14:SGROUP
          }
          3: {"rpo"}
          4: {
            "https://www.auxbrain.com/dlc/shells/ei_depot_1_new_order_36e225eeb89397d3698046a"
            "14321db3b.rpoz"
          }
          5: {"d385b916e34fcc09fde5dbc6baf33101928f47a3b9ee479f9688cb2eb3d2e745"}
          6: 1
          7: 241532
        }
      }
      13: {"new_order"}
      14: 0
      17: -1.691003821822219e09   # 0xc1d932aaeb749f3ci64
      21: 651
    }
...

Most of the ShellSpec messages have an identical structure as this one, except this particular one:

...
  2: {
    1: {"ei_depot_1_pink"}
    3: {"Pink"}
    4: 16
    5: 0
    6: 0.0  # 0x0i64
    12: {
      1: 100   # AssetType given via the enums value defined in the definition file
      2: {
        1: {"ei_depot_1_pink"}
        2: {
          14:SGROUP
          13: 101
          13:EGROUP
          13:EGROUP
          14:SGROUP
        }
        3: {"rpo"}
        4: {
          "https://www.auxbrain.com/dlc/shells/ei_depot_1_pink_17632cbe8c95299bfe7101c1df22"
          "f2af.rpoz"
        }
        5: {"1429dd3fd2f18d66e891aeb9b5dacabac600fc5147b21f362fa51e0ecd8cf2ee"}
        6: 1
        7: 257372
      }
      1: {"ei_depot_1"}    # <-- This looks to be the culprit
    }
    13: {"pink"}
    14: 0
    15: 0
    17: -1.691003821822258e09   # 0xc1d932aaeb749fe0i64
    21: 46
  }
...

Notice that the ShellSpec.ShellPiece denoted by the 12 tag number has 2 child elements with the same tag number of 1. The first child has a value of 100 and the second has a value of {"ei_depot_1"}. Theses both signify that the ShellSpec.ShellPiece has an asset type of Depot_1 but the representation is complement different. I'm not sure if this is the only example of this, but it's the first case that blows up my protobuf parser.

I suspect that the Python parsing code doesn't care about this while other languages do (the proper behavior IMO).

If this is all true, then I think the only fixes are to either modify the parser for whatever language you're using (yuck), or better (but probably less likely), is to have AuxBrain fix their definition of this shell in particular on the server side.

[cranst0n]

Following up, I can confirm that after removing the 'culprit' field from my initial analysis and then reencoding with protoscope, I was able to successfully parse the edited message using the standard Dart protobuf library. Gaining more confidence this is the real issue.

It's still confusing to me how the Python protobuf library could emit this encoding in the first place though.

[cranst0n]

Seems like the additional mystery field is an unknown field which is preserved on the Python side. If I call DiscardUnknownFields on the Python object before saving the ShellSpec in question, I'm able to successfully parse the message in Dart. Reelevant Documentation.

Seems like unknown fields shouldn't be so exotic that a popular Javascript, Dart, etc. protobuf library wouldn't be able to handle it.

[carpetsage]

Nice find. I'll update my python config fetcher to discard that field as well.

[cranst0n]

I don't think that will have an effect in your script since you're saving the json representation, but don't think it will hurt either. Or maybe I'm missing a piece of what your python stuff is doing.

I've developed a workaround where I search for the unknown field byte sequence and modify the tag number which works fine, but has the potential (however unlikely) to break in the future if aux decides to add additional fields to the shellspec struct that will use my modified tag number. I can share if you're interested, but it's probably not something you'd want to use in something as widely used as your site.