VRL parse_json depth

[nabokihms]

Hello, community! I have a question regarding some VRL features.

There is a problem with complicated nested JSON objects using the Elasticsearch sink. I use a simple remap rule to parse messages into objects - .parsed_message = parse_json!(.message); del(.message). However, users tend to put anything in their JSON logs. Vector then parses these nested documents, and Elasticsearch indexes ALL the fields.

There is no way to change the users' behavior because log collecting is provided as a service. I'd like to protect a message parsing depth somehow to prevent Vector/Elasticsearch overloading.

The example:

{
  "1": {
    "2": {
      "3": {
        "4": {
          "5": {
            "6": "finish"
          }
        }
      }
    }
  }
}

I want have a message converted into something like this (docs no more profound than five levels deep are allowed):

{
  "1": {
    "2": {
      "3": {
        "4": {
          "5": "{\"6\": \"finish\"}"
        }
      }
    }
  }
}

1. My initial idea was to implement a function for VRL to control the depth like parse_json_shallow(5, .message).
2. I thought about writing this remap in Lua, but Lua transforms are ~60% slower and seem to utilize only a single core.

Are there any workarounds for that case? What would you suggest?

[jszwedko]

Hi @nabokihms !

I wonder if flatten could help here? https://vector.dev/docs/reference/vrl/functions/#flatten . I'm not thinking of an easy way to do that only after a certain depth in VRL though. I think we could enhance flatten with that functionality though. Would that work for you? Otherwise, we could definitely add another function or option to parse_json to enable this.

lua is much slower, but is an option. To workaround the single thread issue, you could partition the data and run it through N lua transforms, joining it back together afterwards. We do have some plans to allow lua and other "stateful" transforms run in parallel in the near-ish future.

cc/ @JeanMertz this could be another iteration use-case?

[nabokihms]

@jszwedko, thank you for the quick answer! I will try to play with both Lua and flatten and then come back to this discussion.

[JeanMertz]

We have a somewhat similar example in our test-suite (currently disabled, because iteration isn't released yet, but will be in the next release):

https://github.com/vectordotdev/vector/blob/656316f1adf604f8a852da8c9a8c1eab5ef3adc0/lib/vrl/tests/tests/rfcs/8381/run_encode_json_on_all_top_level_object_fields.vrl

In this case, you'd want to iterate 5 levels deep, and then use encode_json on that final level, something like this would work in the next release:

# first level
map_values(.) -> |value| {
  if value.is_object() {
	# second level
	map_values(value) -> |value| {
		if value.is_object() {
			# third level
			map_values(value) -> |value| {
				if value.is_object() {
					# fourth level
					map_values(value) -> |value| {
						if value.is_object() || value.is_array() {
							# string-encode collection types as JSON
				   		 encode_json(value)
						} else {
							value
						}
					}
				} else {
					value
				}
			}
		} else {
			value
		}
	}
  } else {
    value
  }
}

It's not pretty, but it does get the job done.

I do think there's value in limiting the level of recursion in parse_json, we might want to add a max_depth option to that function as an easy solution to this problem.

[nabokihms]

@JeanMertz, this example of VRL iteration is excellent! Thank you. It is good to hear that the iteration support is nearly finished.

We are developing a kind of a kubernetes operator on top of vector, so users have no right to write their own VRL rules. Our operator generates rules from templates and structured configs. It is not crucial for VRL rules to look cute or something :)

@jszwedko, thank you for the hint with Lua transforms! It works like a charm. I didn't manage to invent a solution using the flatten function, though.

Folks, if you don't mind, I would like to open an issue for the max_depth option support (and maybe work on implementation someday).