Socket Data Raw Encoding (Vector 0.20.0)

[musenotdead]

Hi everyone :)

Our company just recently started using Vector and we love it so far and it vast functionality. One of our main use cases is to have Vector put multicast UDP data onto a Kafka topic (see below for the basic configuration). The issue I am running into is that the message put onto Kafka isn't the raw packet data and looks to be UTF8 encoded. See below for the expected data vs received data. Is there any way to fix this? Is there a configuration parameter I am missing?

vector 0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)

Vector Configuration File

[sources.in]
type = "socket" # required
address = "224.3.3.100:61000" # required, required when mode = "tcp" or mode = "udp"
mode = "udp" # required
decoding.codec = "bytes"
# framing = { method = "bytes" }

[sinks.out]
# General
type = "kafka" # required
inputs = ["in"] # required
bootstrap_servers = "kafka:9094" # required
key_field = "user_id" # required
topic = "live_data" # required
encoding.codec = "json" # required

Expected Data

00 17 0e 02 55 55 aa aa  00 ff 00 ac 03 a3 49 c9
00 00 00 00 0b 00 48 df  df 01 01 02 00 06 01 0b
eb a4 0f db 1f 74 50 fe  6c ea f9 22 fb 12 00 00
00 3f fc 00 00 0b ce 41  01 00 0a e9 78 0a 58 1d
01 20 00 01 41 00 00 00  00 00 00 02 00 00 00 00 
00 00 d8 b7 7c 00 00 00  00 00 50 ca     

Received Data
Here is the data on Kafka:

{"host":"192.148.154.5:53016","message":"\u0000\u0017\u000e\u0002UU��\u0000�\u0000�\u0003�I�\u0000\u0000\u0000\u0000\u000b\u0000H��\u0001\u0001\u0002\u0000\u0006\u0001\u000b�\u000f�\u001ftP�l��\"�\u0012\u0000\u0000\u0000?�\u0000\u0000\u000b�A\u0001\u0000\n�x\nX\u001d\u0001 \u0000\u0001A\u0000\u0000\u0000\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0000\u0000\u0000ط|\u0000\u0000\u0000\u0000\u0000P�","source_type":"socket","timestamp":"2022-04-05T12:00:16.843574010Z"}

Which translates to:

00 17 0E 02 55 55 EF BF BD EF BF BD 00 EF BF BD 00 EF BF BD 03 EF BF BD 49 EF BF BD 00 00 00 00 0B 00 48 EF BF BD EF BF BD 01 01 02 00 06 01 0B EF BF BD 0F EF BF BD 1F 74 50 EF BF BD 6C EF BF BD EF BF BD 22 EF BF BD 12 00 00 00 3F EF BF BD 00 00 0B EF BF BD 41 01 00 0A EF BF BD 78 0A 58 1D 01 20 00 01 41 00 00 00 00 00 00 02 00 00 00 00 00 00 D8 B7 7C 00 00 00 00 00 50 EF BF BD

Thanks for any help in advance! We are looking forward to deploying Vector across our products :)

[musenotdead]

This may be related #10571

[musenotdead]

We solved this by having a transform with base64 encoding :)

[transforms.do_something]
type = "remap"
inputs = [ "in" ]
source = """
. = encode_base64!(.message)
"""