The difference between logs and metric events

[candlerb]

Something I'm confused about, which relates to issue #9306.

I am led to understand that logs and metric events are treated differently. I see examples with {"log": {...}} and {"metric": {...}} [^1]. The data model documentation makes this explicit:

A metric event in Vector represents a numerical operation performed on a time series. In Vector, unlike in other tools, metrics are first-class citizens. They are not represented as logs.

However, by the time I get to a sink - let's say the console sink for simplicity - the "log" or "metric" wrapper has been removed. All I see is a JSON object:

{"name":"process_cpu_seconds_total","timestamp":"2021-09-24T08:26:55.450211329Z","kind":"absolute","counter":{"value":4853.94}}

{"appname":"non","exampleSDID@32473":{"eventID":"1011","eventSource":"Application","iut":"3"},"facility":"user","host":"dynamicwireless.name","hostname":"dynamicwireless.name","message":"Try to override the THX port, maybe it will reboot the neural interface!","msgid":"ID931","procid":2426,"severity":"notice","source_ip":"127.0.0.1","source_type":"syslog","version":1}

Even in an intermediate transform I don't see the outer wrapper. For example, I would write

  . = parse_json!(.message)

and not

  . = parse_json!(.log.message)

So my question is: how does the distinction between logs and metric events make a difference in practice to the end user? Is it possible to distinguish between them in a transform or sink? If not, what purpose does the separation serve?

EDIT: if I receive data from a general-purpose source - kafka say - what determines if this is a stream of logs or metrics?

---

[^1] #9306 says these are from "the cue data". In the repo I found website/cue/reference.cue although I can't see where it's linked in the website.

[jszwedko]

Hey @candlerb !

Apologies for the delay. These are good questions.

However, by the time I get to a sink - let's say the console sink for simplicity - the "log" or "metric" wrapper has been removed. All I see is a JSON object:

This is an error in our documentation (as you noted with link to #9306). It shouldn't be rendering the wrapping object with log and metric keys. This is an artifact of how we represent events as an enum in the cue configuration.

When Vector is dealing with events they will either be logs (which are essentially just key/value maps) or metrics (which are structs with a specific set of fields as described on https://vector.dev/docs/about/under-the-hood/architecture/data-model/metric/).

EDIT: if I receive data from a general-purpose source - kafka say - what determines if this is a stream of logs or metrics?

Currently the kafka source only accepts logs but we are hoping to change that in the future to allow metrics to be easily sent over Kafka as well. We haven't yet figured out what the external data model for this will be (i.e. how Vector will encode / decode generic events from Kafka). One option for doing this would be to model it as you had seen it in the docs, with a wrapping object that indicates the type of the data.

The one example we do currently have of sending logs and metrics over the same transport is the vector source and sink which encode the events as protobuf; which is capable of modeling the enum nature of events.

vector/lib/vector-core/proto/event.proto

Lines 6 to 11 in 74c3f06

	message EventWrapper {
	oneof event {
	Log log = 1;
	Metric metric = 2;
	}
	}

Does that make things clearer? Let me know if you have questions! I bumped #9306 again to see if we can resolve that to reduce confusion.

[candlerb]

Many thanks for your reply.

I have an upcoming use case to consume metrics which have being posted inside JSON documents to a Kafka queue. I need to pick them up, extract the metrics (one JSON document can contain multiple metrics), and post them to prometheus remote-write.

It seems like Vector might be a good fit for this, as long as one incoming message (log) can be parsed in VML and expanded to multiple outgoing messages (metrics)

Can VML tell the difference between logs and metrics?

Can VML consume a log and emit a metric, or multiple metrics?

Is each sink specific to "metrics" or "logs"? What happens if a sink which is designed for "metrics" gets passed a "log", or vice versa?

The list of sinks doesn't categorise them as processing "logs" or "metrics". And the prometheus remote write documentation doesn't explicitly say what happens if you pass it a log rather than a metric.

There are some cases, like Elasticsearch in particular, where I can imagine it needing to take either logs or metrics.

[candlerb]

When Vector is dealing with events they will either be logs (which are essentially just key/value maps)

Although would it be fair to say that at some parts of a log record are expected by convention - in particular that there are normally keys "message" and "timestamp"?

[jszwedko]

Yeah, there is some convention for log messages: message should essentially always be there unless removed by a transform; host and timestamp are typically added by sources.

[sushovan23]

@candlerb I'm in process on configuring same flow of json metrics -> kafka -> vector -> prometheus.
what transformation did you end up using?

[candlerb]

Haven't yet - the kafka source I was expecting to consume metrics from has not yet been made available :-(