Avrora schema registry registration fails due to Invalid payload format

[EM51641]

Hi !

Amazing library but I just encountered an issue detailed underneath:

Description

When attempting to register an Avro schema with Schema Registry using Avrora, the registration fails with a FunctionClauseError. The error occurs because the payload being sent to the Schema Registry is a map when it should be a binary string.

Current Behavior

The encode/decode test fails when trying to register the schema with the following error:

** (FunctionClauseError) no function clause matching in Avrora.HTTPClient.post/3

     The following arguments were given to Avrora.HTTPClient.post/3:

         # 1
         "http://localhost:8081/subjects/io.confluent.Payment/versions"

         # 2
         %{schema: "{\"namespace\":\"io.confluent\",\"name\":\"Payment\",\"type\":\"record\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"}

         # 3
         [content_type: "application/vnd.schemaregistry.v1+json", ssl_options: [verify: :verify_none], user_agent: "Avrora/0.25.0 Elixir", authorization: "Basic XXXX"]

     Attempted function clauses (showing 1 out of 1):

         def post(url, payload, options) when is_binary(payload)

     code: decoded = message |> AvroTest.encode() |> AvroTest.decode()
     stacktrace:
       (avrora 0.29.0) lib/avrora/http_client.ex:22: Avrora.HTTPClient.post/3
       (avrora 0.29.0) lib/avrora/storage/registry.ex:128: Avrora.Storage.Registry.http_client_post/2
       (avrora 0.29.0) lib/avrora/storage/registry.ex:74: Avrora.Storage.Registry.put/2
       (avrora 0.29.0) lib/avrora/resolver.ex:102: Avrora.Resolver.resolve_with_registry/1
       (avrora 0.29.0) lib/avrora/resolver.ex:77: Avrora.Resolver.resolve/1
       (avrora 0.29.0) lib/avrora/codec/schema_registry.ex:70: Avrora.Codec.SchemaRegistry.encode/2
       (avrora 0.29.0) lib/avrora/encoder.ex:153: Avrora.Encoder.encode/2
       (avro_test 0.1.0) lib/avro_test.ex:20: AvroTest.encode/1
       test/avro_test_test.exs:19: (test)

The payload is being sent as a map:

%{schema: "{\"namespace\":\"io.confluent\"..."}

when it should be a binary string.

Configuration

config :avrora,
  schemas_path: "priv/schemas",
  # registry_type: :file,
  registry_schemas_autoreg: true,
  registry_url: "http://localhost:8081",
  registry_auth: {:basic, ["username", "password"]},  # optional
  registry_user_agent: "Avrora/0.25.0 Elixir"       # optional

Schema Definition

Located at priv/schemas/payment.avsc:

{
  "type": "record",
  "name": "Payment",
  "namespace": "io.confluent",
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "amount",
      "type": "double"
    }
  ]
}

Confluent registry and kafka brokers config on a docker compose file.

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  schema-registry:
    image: confluentinc/cp-schema-registry:latest
    depends_on:
      - kafka
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'kafka:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

The behavior tested is here:

defmodule AvroTest do
  @moduledoc """
  Documentation for `AvroTest`.
  """

  def encode(message) do
    {:ok, encoded} = Avrora.encode(message, schema_name: "io.confluent.Payment")
    encoded
  end

  def decode(encoded) do
    {:ok, decoded} = Avrora.decode(encoded, schema_name: "io.confluent.Payment")
    decoded
  end
end

This is a simple test case:

defmodule AvroTestTest do
  use ExUnit.Case, async: false

  setup do
    Application.ensure_all_started(:avro_test)
    :ok
  end

  test "encode and decode" do
    message = %{"id" => "tx-1", "amount" => 15.99}
    decoded = message |> AvroTest.encode() |> AvroTest.decode()
    assert decoded == message
  end
end

[Strech]

Hey @EM51641 thanks for the kind words! Could you also tell me which version you use? I see the agent header as Avrora/0.25.0 Elixir.

If it's the 0.25 I would suggest to upgrade to 0.29.1

[lucas-nelson]

This is working in 0.28.0, broken in 0.29.1 (and 0.29.0)

It's this change: b11472d#diff-bf53ad6d056234d501d148f29e0f44f1b9884363abf9b4ec4dd25d640ebc8f67R74

Old code: value which is a binary, new code: %{schema: value} (ie. a Map). The http_client_post function passes this as payload untouched into the second param in the post function call. Which fails because when is_binary(payload).

PS: in my error report, I see this in the options param:

       [
         content_type: "application/vnd.schemaregistry.v1+json",
         ssl_options: [verify: :verify_none],
         user_agent: "Avrora/0.29.1 Elixir"
       ]

[azeemchauhan]

Yes, this is a bug, we are passing a map which is not matching with Avrora.HTTPClient's function definition

 def post(url, payload, options \\ []) when is_binary(payload) do 
 end

Since we are mocking this function in tests that why were't able to catch it. Ideally we should mock the external call (:httpc.request(:post, {}, [], [])) instead.

[Strech]

Fix was merged, but I would like to keep the issue to not forget and cover it with test

[Strech]

The v0.29.2 was released, thanks for the head up @EM51641 !